1 1 General Introduction 1 ******************** 1 1 This file documents 'awk', a program that you can use to select 1 particular records in a file and perform operations upon them. 1 1 Copyright (C) 1989, 1991, 1992, 1993, 1996-2005, 2007, 2009-2018 1 Free Software Foundation, Inc. 1 1 1 This is Edition 4.2 of 'GAWK: Effective AWK Programming: A User's 1 Guide for GNU Awk', for the 4.2.1 (or later) version of the GNU 1 implementation of AWK. 1 1 Permission is granted to copy, distribute and/or modify this document 1 under the terms of the GNU Free Documentation License, Version 1.3 or 1 any later version published by the Free Software Foundation; with the 1 Invariant Sections being "GNU General Public License", with the 1 Front-Cover Texts being "A GNU Manual", and with the Back-Cover Texts as 1 in (a) below. A copy of the license is included in the section entitled 1 "GNU Free Documentation License". 1 1 a. The FSF's Back-Cover Text is: "You have the freedom to copy and 1 modify this GNU manual." 1
1
· Foreword3 Some nice words about this
1 Info file.
· Foreword4 More nice words.
· Preface What this Info file is about; brief
1 history and acknowledgments.
· Getting Started A basic introduction to using
1 'awk'. How to run an 'awk'
1 program. Command-line syntax.
· Invoking Gawk How to run 'gawk'.
· Regexp All about matching things using regular
1 expressions.
· Reading Files How to read files and manipulate fields.
· Printing How to print using 'awk'. Describes
1 the 'print' and 'printf'
1 statements. Also describes redirection of
1 output.
· Expressions Expressions are the basic building blocks
1 of statements.
· Patterns and Actions Overviews of patterns and actions.
· Arrays The description and use of arrays. Also
1 includes array-oriented control statements.
· Functions Built-in and user-defined functions.
· Library Functions A Library of 'awk' Functions.
· Sample Programs Many 'awk' programs with complete
1 explanations.
· Advanced Features Stuff for advanced users, specific to
1 'gawk'.
· Internationalization Getting 'gawk' to speak your
1 language.
· Debugger The 'gawk' debugger.
· Arbitrary Precision Arithmetic Arbitrary precision arithmetic with
1 'gawk'.
· Dynamic Extensions Adding new built-in functions to
1 'gawk'.
· Language History The evolution of the 'awk'
1 language.
· Installation Installing 'gawk' under various
1 operating systems.
· Notes Notes about adding things to 'gawk'
1 and possible future work.
· Basic Concepts A very quick introduction to programming
1 concepts.
· Glossary An explanation of some unfamiliar terms.
· Copying Your right to copy and distribute
1 'gawk'.
· GNU Free Documentation License The license for this Info file.
· Index Concept and Variable Index.
1
· History The history of 'gawk' and
1 'awk'.
· Names What name to use to find
1 'awk'.
· This Manual Using this Info file. Includes
1 sample input files that you can use.
· Conventions Typographical Conventions.
· Manual History Brief history of the GNU project and
1 this Info file.
· How To Contribute Helping to save the world.
· Acknowledgments Acknowledgments.
· Running gawk How to run 'gawk' programs;
1 includes command-line syntax.
· One-shot Running a short throwaway
1 'awk' program.
· Read Terminal Using no input files (input from the
1 keyboard instead).
· Long Putting permanent 'awk'
1 programs in files.
· Executable Scripts Making self-contained 'awk'
1 programs.
· Comments Adding documentation to 'gawk'
1 programs.
· Quoting More discussion of shell quoting
1 issues.
· DOS Quoting Quoting in Windows Batch Files.
· Sample Data Files Sample data files for use in the
1 'awk' programs illustrated in
1 this Info file.
· Very Simple A very simple example.
· Two Rules A less simple one-line example using
1 two rules.
· More Complex A more complex example.
· Statements/Lines Subdividing or combining statements
1 into lines.
· Other Features Other Features of 'awk'.
· When When to use 'gawk' and when to
1 use other things.
· Intro Summary Summary of the introduction.
· Command Line How to run 'awk'.
· Options Command-line options and their
1 meanings.
· Other Arguments Input file names and variable
1 assignments.
· Naming Standard Input How to specify standard input with
1 other files.
· Environment Variables The environment variables
1 'gawk' uses.
· AWKPATH Variable Searching directories for
1 'awk' programs.
· AWKLIBPATH Variable Searching directories for
1 'awk' shared libraries.
· Other Environment Variables The environment variables.
· Exit Status 'gawk''s exit status.
· Include Files Including other files into your
1 program.
· Loading Shared Libraries Loading shared libraries into your
1 program.
· Obsolete Obsolete Options and/or features.
· Undocumented Undocumented Options and Features.
· Invoking Summary Invocation summary.
· Regexp Usage How to Use Regular Expressions.
· Escape Sequences How to write nonprinting characters.
· Regexp Operators Regular Expression Operators.
· Bracket Expressions What can go between '[...]'.
· Leftmost Longest How much text matches.
· Computed Regexps Using Dynamic Regexps.
· GNU Regexp Operators Operators specific to GNU software.
· Case-sensitivity How to do case-insensitive matching.
· Regexp Summary Regular expressions summary.
· Records Controlling how data is split into
1 records.
· awk split records How standard 'awk' splits
1 records.
· gawk split records How 'gawk' splits records.
· Fields An introduction to fields.
· Nonconstant Fields Nonconstant Field Numbers.
· Changing Fields Changing the Contents of a Field.
· Field Separators The field separator and how to change
1 it.
· Default Field Splitting How fields are normally separated.
· Regexp Field Splitting Using regexps as the field separator.
· Single Character Fields Making each character a separate
1 field.
· Command Line Field Separator Setting 'FS' from the command
1 line.
· Full Line Fields Making the full line be a single
1 field.
· Field Splitting Summary Some final points and a summary table.
· Constant Size Reading constant width data.
· Fixed width data Processing fixed-width data.
· Skipping intervening Skipping intervening fields.
· Allowing trailing data Capturing optional trailing data.
· Fields with fixed data Field values with fixed-width data.
· Splitting By Content Defining Fields By Content
· Testing field creation Checking how 'gawk' is
1 splitting records.
· Multiple Line Reading multiline records.
· Getline Reading files under explicit program
1 control using the 'getline'
1 function.
· Plain Getline Using 'getline' with no
1 arguments.
· Getline/Variable Using 'getline' into a variable.
· Getline/File Using 'getline' from a file.
· Getline/Variable/File Using 'getline' into a variable
1 from a file.
· Getline/Pipe Using 'getline' from a pipe.
· Getline/Variable/Pipe Using 'getline' into a variable
1 from a pipe.
· Getline/Coprocess Using 'getline' from a coprocess.
· Getline/Variable/Coprocess Using 'getline' into a variable
1 from a coprocess.
· Getline Notes Important things to know about
1 'getline'.
· Getline Summary Summary of 'getline' Variants.
· Read Timeout Reading input with a timeout.
· Retrying Input Retrying input after certain errors.
· Command-line directories What happens if you put a directory on
1 the command line.
· Input Summary Input summary.
· Input Exercises Exercises.
· Print The 'print' statement.
· Print Examples Simple examples of 'print'
1 statements.
· Output Separators The output separators and how to
1 change them.
· OFMT Controlling Numeric Output With
1 'print'.
· Printf The 'printf' statement.
· Basic Printf Syntax of the 'printf' statement.
· Control Letters Format-control letters.
· Format Modifiers Format-specification modifiers.
· Printf Examples Several examples.
· Redirection How to redirect output to multiple
1 files and pipes.
· Special FD Special files for I/O.
· Special Files File name interpretation in
1 'gawk'. 'gawk' allows
1 access to inherited file descriptors.
· Other Inherited Files Accessing other open files with
1 'gawk'.
· Special Network Special files for network
1 communications.
· Special Caveats Things to watch out for.
· Close Files And Pipes Closing Input and Output Files and
1 Pipes.
· Nonfatal Enabling Nonfatal Output.
· Output Summary Output summary.
· Output Exercises Exercises.
· Values Constants, Variables, and Regular
1 Expressions.
· Constants String, numeric and regexp constants.
· Scalar Constants Numeric and string constants.
· Nondecimal-numbers What are octal and hex numbers.
· Regexp Constants Regular Expression constants.
· Using Constant Regexps When and how to use a regexp constant.
· Standard Regexp Constants Regexp constants in standard
1 'awk'.
· Strong Regexp Constants Strongly typed regexp constants.
· Variables Variables give names to values for
1 later use.
· Using Variables Using variables in your programs.
· Assignment Options Setting variables on the command line
1 and a summary of command-line syntax.
1 This is an advanced method of input.
· Conversion The conversion of strings to numbers
1 and vice versa.
· Strings And Numbers How 'awk' Converts Between
1 Strings And Numbers.
· Locale influences conversions How the locale may affect conversions.
· All Operators 'gawk''s operators.
· Arithmetic Ops Arithmetic operations ('+',
1 '-', etc.)
· Concatenation Concatenating strings.
· Assignment Ops Changing the value of a variable or a
1 field.
· Increment Ops Incrementing the numeric value of a
1 variable.
· Truth Values and Conditions Testing for true and false.
· Truth Values What is "true" and what is
1 "false".
· Typing and Comparison How variables acquire types and how
1 this affects comparison of numbers and
1 strings with '<', etc.
· Variable Typing String type versus numeric type.
· Comparison Operators The comparison operators.
· POSIX String Comparison String comparison with POSIX rules.
· Boolean Ops Combining comparison expressions using
1 boolean operators '||' ("or"),
1 '&&' ("and") and '!'
1 ("not").
· Conditional Exp Conditional expressions select between
1 two subexpressions under control of a
1 third subexpression.
· Function Calls A function call is an expression.
· Precedence How various operators nest.
· Locales How the locale affects things.
· Expressions Summary Expressions summary.
· Pattern Overview What goes into a pattern.
· Regexp Patterns Using regexps as patterns.
· Expression Patterns Any expression can be used as a
1 pattern.
· Ranges Pairs of patterns specify record
1 ranges.
· BEGIN/END Specifying initialization and cleanup
1 rules.
· Using BEGIN/END How and why to use BEGIN/END rules.
· I/O And BEGIN/END I/O issues in BEGIN/END rules.
· BEGINFILE/ENDFILE Two special patterns for advanced
1 control.
· Empty The empty pattern, which matches every
1 record.
· Using Shell Variables How to use shell variables with
1 'awk'.
· Action Overview What goes into an action.
· Statements Describes the various control
1 statements in detail.
· If Statement Conditionally execute some
1 'awk' statements.
· While Statement Loop until some condition is
1 satisfied.
· Do Statement Do specified action while looping
1 until some condition is satisfied.
· For Statement Another looping statement, that
1 provides initialization and increment
1 clauses.
· Switch Statement Switch/case evaluation for conditional
1 execution of statements based on a
1 value.
· Break Statement Immediately exit the innermost
1 enclosing loop.
· Continue Statement Skip to the end of the innermost
1 enclosing loop.
· Next Statement Stop processing the current input
1 record.
· Nextfile Statement Stop processing the current file.
· Exit Statement Stop execution of 'awk'.
· Built-in Variables Summarizes the predefined variables.
· User-modified Built-in variables that you change to
1 control 'awk'.
· Auto-set Built-in variables where 'awk'
1 gives you information.
· ARGC and ARGV Ways to use 'ARGC' and
1 'ARGV'.
· Pattern Action Summary Patterns and Actions summary.
· Array Basics The basics of arrays.
· Array Intro Introduction to Arrays
· Reference to Elements How to examine one element of an
1 array.
· Assigning Elements How to change an element of an array.
· Array Example Basic Example of an Array
· Scanning an Array A variation of the 'for'
1 statement. It loops through the
1 indices of an array's existing
1 elements.
· Controlling Scanning Controlling the order in which arrays
1 are scanned.
· Numeric Array Subscripts How to use numbers as subscripts in
1 'awk'.
· Uninitialized Subscripts Using Uninitialized variables as
1 subscripts.
· Delete The 'delete' statement removes an
1 element from an array.
· Multidimensional Emulating multidimensional arrays in
1 'awk'.
· Multiscanning Scanning multidimensional arrays.
· Arrays of Arrays True multidimensional arrays.
· Arrays Summary Summary of arrays.
· Built-in Summarizes the built-in functions.
· Calling Built-in How to call built-in functions.
· Numeric Functions Functions that work with numbers,
1 including 'int()', 'sin()'
1 and 'rand()'.
· String Functions Functions for string manipulation,
1 such as 'split()', 'match()'
1 and 'sprintf()'.
· Gory Details More than you want to know about
1 '\' and '&' with
1 'sub()', 'gsub()', and
1 'gensub()'.
· I/O Functions Functions for files and shell
1 commands.
· Time Functions Functions for dealing with timestamps.
· Bitwise Functions Functions for bitwise operations.
· Type Functions Functions for type information.
· I18N Functions Functions for string translation.
· User-defined Describes User-defined functions in
1 detail.
· Definition Syntax How to write definitions and what they
1 mean.
· Function Example An example function definition and
1 what it does.
· Function Caveats Things to watch out for.
· Calling A Function Don't use spaces.
· Variable Scope Controlling variable scope.
· Pass By Value/Reference Passing parameters.
· Return Statement Specifying the value a function
1 returns.
· Dynamic Typing How variable types can change at
1 runtime.
· Indirect Calls Choosing the function to call at
1 runtime.
· Functions Summary Summary of functions.
· Library Names How to best name private global
1 variables in library functions.
· General Functions Functions that are of general use.
· Strtonum Function A replacement for the built-in
1 'strtonum()' function.
· Assert Function A function for assertions in
1 'awk' programs.
· Round Function A function for rounding if
1 'sprintf()' does not do it
1 correctly.
· Cliff Random Function The Cliff Random Number Generator.
· Ordinal Functions Functions for using characters as
1 numbers and vice versa.
· Join Function A function to join an array into a
1 string.
· Getlocaltime Function A function to get formatted times.
· Readfile Function A function to read an entire file at
1 once.
· Shell Quoting A function to quote strings for the
1 shell.
· Data File Management Functions for managing command-line
1 data files.
· Filetrans Function A function for handling data file
1 transitions.
· Rewind Function A function for rereading the current
1 file.
· File Checking Checking that data files are readable.
· Empty Files Checking for zero-length files.
· Ignoring Assigns Treating assignments as file names.
· Getopt Function A function for processing command-line
1 arguments.
· Passwd Functions Functions for getting user
1 information.
· Group Functions Functions for getting group
1 information.
· Walking Arrays A function to walk arrays of arrays.
· Library Functions Summary Summary of library functions.
· Library Exercises Exercises.
· Running Examples How to run these examples.
· Clones Clones of common utilities.
· Cut Program The 'cut' utility.
· Egrep Program The 'egrep' utility.
· Id Program The 'id' utility.
· Split Program The 'split' utility.
· Tee Program The 'tee' utility.
· Uniq Program The 'uniq' utility.
· Wc Program The 'wc' utility.
· Miscellaneous Programs Some interesting 'awk'
1 programs.
· Dupword Program Finding duplicated words in a
1 document.
· Alarm Program An alarm clock.
· Translate Program A program similar to the 'tr'
1 utility.
· Labels Program Printing mailing labels.
· Word Sorting A program to produce a word usage
1 count.
· History Sorting Eliminating duplicate entries from a
1 history file.
· Extract Program Pulling out programs from Texinfo
1 source files.
· Simple Sed A Simple Stream Editor.
· Igawk Program A wrapper for 'awk' that
1 includes files.
· Anagram Program Finding anagrams from a dictionary.
· Signature Program People do amazing things with too much
1 time on their hands.
· Programs Summary Summary of programs.
· Programs Exercises Exercises.
· Nondecimal Data Allowing nondecimal input data.
· Array Sorting Facilities for controlling array
1 traversal and sorting arrays.
· Controlling Array Traversal How to use PROCINFO["sorted_in"].
· Array Sorting Functions How to use 'asort()' and
1 'asorti()'.
· Two-way I/O Two-way communications with another
1 process.
· TCP/IP Networking Using 'gawk' for network
1 programming.
· Profiling Profiling your 'awk' programs.
· Advanced Features Summary Summary of advanced features.
· I18N and L10N Internationalization and Localization.
· Explaining gettext How GNU 'gettext' works.
· Programmer i18n Features for the programmer.
· Translator i18n Features for the translator.
· String Extraction Extracting marked strings.
· Printf Ordering Rearranging 'printf' arguments.
· I18N Portability 'awk'-level portability
1 issues.
· I18N Example A simple i18n example.
· Gawk I18N 'gawk' is also
1 internationalized.
· I18N Summary Summary of I18N stuff.
· Debugging Introduction to 'gawk'
1 debugger.
· Debugging Concepts Debugging in General.
· Debugging Terms Additional Debugging Concepts.
· Awk Debugging Awk Debugging.
· Sample Debugging Session Sample debugging session.
· Debugger Invocation How to Start the Debugger.
· Finding The Bug Finding the Bug.
· List of Debugger Commands Main debugger commands.
· Breakpoint Control Control of Breakpoints.
· Debugger Execution Control Control of Execution.
· Viewing And Changing Data Viewing and Changing Data.
· Execution Stack Dealing with the Stack.
· Debugger Info Obtaining Information about the
1 Program and the Debugger State.
· Miscellaneous Debugger Commands Miscellaneous Commands.
· Readline Support Readline support.
· Limitations Limitations and future plans.
· Debugging Summary Debugging summary.
· Computer Arithmetic A quick intro to computer math.
· Math Definitions Defining terms used.
· MPFR features The MPFR features in 'gawk'.
· FP Math Caution Things to know.
· Inexactness of computations Floating point math is not exact.
· Inexact representation Numbers are not exactly represented.
· Comparing FP Values How to compare floating point values.
· Errors accumulate Errors get bigger as they go.
· Getting Accuracy Getting more accuracy takes some work.
· Try To Round Add digits and round.
· Setting precision How to set the precision.
· Setting the rounding mode How to set the rounding mode.
· Arbitrary Precision Integers Arbitrary Precision Integer Arithmetic
1 with 'gawk'.
· Checking for MPFR How to check if MPFR is available.
· POSIX Floating Point Problems Standards Versus Existing Practice.
· Floating point summary Summary of floating point discussion.
· Extension Intro What is an extension.
· Plugin License A note about licensing.
· Extension Mechanism Outline An outline of how it works.
· Extension API Description A full description of the API.
· Extension API Functions Introduction Introduction to the API functions.
· General Data Types The data types.
· Memory Allocation Functions Functions for allocating memory.
· Constructor Functions Functions for creating values.
· Registration Functions Functions to register things with
1 'gawk'.
· Extension Functions Registering extension functions.
· Exit Callback Functions Registering an exit callback.
· Extension Version String Registering a version string.
· Input Parsers Registering an input parser.
· Output Wrappers Registering an output wrapper.
· Two-way processors Registering a two-way processor.
· Printing Messages Functions for printing messages.
· Updating ERRNO Functions for updating 'ERRNO'.
· Requesting Values How to get a value.
· Accessing Parameters Functions for accessing parameters.
· Symbol Table Access Functions for accessing global
1 variables.
· Symbol table by name Accessing variables by name.
· Symbol table by cookie Accessing variables by "cookie".
· Cached values Creating and using cached values.
· Array Manipulation Functions for working with arrays.
· Array Data Types Data types for working with arrays.
· Array Functions Functions for working with arrays.
· Flattening Arrays How to flatten arrays.
· Creating Arrays How to create and populate arrays.
· Redirection API How to access and manipulate
1 redirections.
· Extension API Variables Variables provided by the API.
· Extension Versioning API Version information.
· Extension GMP/MPFR Versioning Version information about GMP and
1 MPFR.
· Extension API Informational Variables Variables providing information about
1 'gawk''s invocation.
· Extension API Boilerplate Boilerplate code for using the API.
· Changes from API V1 Changes from V1 of the API.
· Finding Extensions How 'gawk' finds compiled
1 extensions.
· Extension Example Example C code for an extension.
· Internal File Description What the new functions will do.
· Internal File Ops The code for internal file operations.
· Using Internal File Ops How to use an external extension.
· Extension Samples The sample extensions that ship with
1 'gawk'.
· Extension Sample File Functions The file functions sample.
· Extension Sample Fnmatch An interface to 'fnmatch()'.
· Extension Sample Fork An interface to 'fork()' and
1 other process functions.
· Extension Sample Inplace Enabling in-place file editing.
· Extension Sample Ord Character to value to character
1 conversions.
· Extension Sample Readdir An interface to 'readdir()'.
· Extension Sample Revout Reversing output sample output
1 wrapper.
· Extension Sample Rev2way Reversing data sample two-way
1 processor.
· Extension Sample Read write array Serializing an array to a file.
· Extension Sample Readfile Reading an entire file into a string.
· Extension Sample Time An interface to 'gettimeofday()'
1 and 'sleep()'.
· Extension Sample API Tests Tests for the API.
· gawkextlib The 'gawkextlib' project.
· Extension summary Extension summary.
· Extension Exercises Exercises.
· V7/SVR3.1 The major changes between V7 and
1 System V Release 3.1.
· SVR4 Minor changes between System V
1 Releases 3.1 and 4.
· POSIX New features from the POSIX standard.
· BTL New features from Brian Kernighan's
1 version of 'awk'.
· POSIX/GNU The extensions in 'gawk' not
1 in POSIX 'awk'.
· Feature History The history of the features in
1 'gawk'.
· Common Extensions Common Extensions Summary.
· Ranges and Locales How locales used to affect regexp
1 ranges.
· Contributors The major contributors to
1 'gawk'.
· History summary History summary.
· Gawk Distribution What is in the 'gawk'
1 distribution.
· Getting How to get the distribution.
· Extracting How to extract the distribution.
· Distribution contents What is in the distribution.
· Unix Installation Installing 'gawk' under
1 various versions of Unix.
· Quick Installation Compiling 'gawk' under Unix.
· Shell Startup Files Shell convenience functions.
· Additional Configuration Options Other compile-time options.
· Configuration Philosophy How it's all supposed to work.
· Non-Unix Installation Installation on Other Operating
1 Systems.
· PC Installation Installing and Compiling
1 'gawk' on Microsoft Windows.
· PC Binary Installation Installing a prepared distribution.
· PC Compiling Compiling 'gawk' for
1 Windows32.
· PC Using Running 'gawk' on Windows32.
· Cygwin Building and running 'gawk'
1 for Cygwin.
· MSYS Using 'gawk' In The MSYS
1 Environment.
· VMS Installation Installing 'gawk' on VMS.
· VMS Compilation How to compile 'gawk' under
1 VMS.
· VMS Dynamic Extensions Compiling 'gawk' dynamic
1 extensions on VMS.
· VMS Installation Details How to install 'gawk' under
1 VMS.
· VMS Running How to run 'gawk' under VMS.
· VMS GNV The VMS GNV Project.
· VMS Old Gawk An old version comes with some VMS
1 systems.
· Bugs Reporting Problems and Bugs.
· Bug address Where to send reports to.
· Usenet Where not to send reports to.
· Maintainers Maintainers of non-*nix ports.
· Other Versions Other freely available 'awk'
1 implementations.
· Installation summary Summary of installation.
· Compatibility Mode How to disable certain 'gawk'
1 extensions.
· Additions Making Additions To 'gawk'.
· Accessing The Source Accessing the Git repository.
· Adding Code Adding code to the main body of
1 'gawk'.
· New Ports Porting 'gawk' to a new
1 operating system.
· Derived Files Why derived files are kept in the Git
1 repository.
· Future Extensions New features that may be implemented
1 one day.
· Implementation Limitations Some limitations of the
1 implementation.
· Extension Design Design notes about the extension API.
· Old Extension Problems Problems with the old mechanism.
· Extension New Mechanism Goals Goals for the new mechanism.
· Extension Other Design Decisions Some other design decisions.
· Extension Future Growth Some room for future growth.
· Old Extension Mechanism Some compatibility for old extensions.
· Notes summary Summary of implementation notes.
· Basic High Level The high level view.
· Basic Data Typing A very quick intro to data types.
1
1 To my parents, for their love, and for the wonderful example they set
1 for me.
1
1 To my wife Miriam, for making me complete. Thank you for building
1 your life together with me.
1
1 To our children Chana, Rivka, Nachum and Malka, for enrichening our
1 lives in innumerable ways.
1