Skip to main content
awk command with examples

AWK command with examples

AWK – is a powerful text processing command. It searches for a pattern of text in a line/file and performs an action if it matches. AWK is an excellent filter and so it is an excellent tool for Log processing. So, in this article, we will cover few use cases of AWK command with examples.

Some of the key features of Awk are:

  • Scripting Language
  • It has variables, Conditionals and Loops
  • Arithmetic and String operators

 

Basic Structure:

The basic structure of an AWK command looks like this:

awk 'pattern {action}' input-file

It says, take each line of the input file; if the line contains the pattern apply the action specified to the line; and print the resulting line.

  • If the pattern is not specified, the action is applied to all lines.
  • If the action is not specified, the awk will print all the lines that match with the given patterns which is the default action.

 

Basic AWK Options and Functions:

TagDescription
-F or --field-separatorInput Field Separator (FS)
-f or --fileprogram file
NRThe total number of input records (Rows)
NFThe total number of fields(columns) in the current input record
-v var=val
--assign var=val
Assign the value val to the variable var, before execution of the program begins.
$0 The whole line
$1, $2...$n
Fields from 1 to n.
RS
Record Separator, a newline by default.
OFSField separator of the outputted data (default whitespace)
ORSRecord separator of the outputted data (default newline)
index(s,t)Position in string s where string t occurs, 0 if not found
length(s)Length of string s (or $0 if no arg)
int(x)Truncate x to integer value
split(s,a,fs)Split string s into array a split by fs, returning length of a
system(cmd)Execute cmd and return exit status
tolower(s)String s to lowercase
toupper(s)String s to uppercase
BEGINPreprocessing (Executes before processing the data)
ENDPostprocessing (Executes after processing the data)

 

AWK command with examples:

Create a file “sample.txt” with the following contents. We will be using this file for the below examples.

dnsmasq:x:488:65534:dnsmasq:/var/lib/empty:/bin/false 
mail:x:8:12:Mailer daemon:/var/spool/clientmqueue:/bin/false 
leo:x:506:506::/home/leo:/bin/bash
awk:x:911:911::/home/awk:/sbin/nologin
messagebus:x:499:499:User for D-Bus:/run/dbus:/bin/false 
mysql:x:60:494:MySQL database admin:/var/lib/mysql:/bin/false 
ntp:x:74:488:NTP daemon:/var/lib/ntp:/bin/false
mat:x:502:502::/home/mat:/bin/bash
leone:x:507:507::/home/leone:/bin/bash

Note: This file has fields separated by colons.

 

Example 1

The default field separator in AWK is a whitespace character (space or tab). Check the following example and see how AWK processes it:

[email protected]:~> awk '{print $1}' sample.txt  
dnsmasq:x:488:65534:dnsmasq:/var/lib/empty:/bin/false 
mail:x:8:12:Mailer 
leo:x:506:506::/home/leo:/bin/bash 
awk:x:911:911::/home/awk:/sbin/nologin 
messagebus:x:499:499:User 
mysql:x:60:494:MySQL 
ntp:x:74:488:NTP 
mat:x:502:502::/home/mat:/bin/bash
leone:x:507:507::/home/leone:/bin/bash

Here, the AWK will print each line till it encounters the first whitespace character.

 

Example 2

But the actual field separator in our “sample.txt” file is “:”, so we need to specify it using the “-F” option.

[email protected]:~> awk -F: '{print $1}' sample.txt 
dnsmasq
mail
leo
awk
messagebus
mysql
ntp
mat
leone

 

Example 3

Printing Multiple Fields/Columns.

[email protected]:~> awk -F: '{print $1 "\t" $NF}' sample.txt    
dnsmasq /bin/false  
mail    /bin/false  
leo     /bin/bash 
awk     /sbin/nologin 
messagebus      /bin/false  
mysql   /bin/false  
ntp     /bin/false 
mat     /bin/bash
leone   /bin/bash

$NF – prints the last field of each matching line.

 

Example 4

Printing Field in Any Order

[email protected]:~> awk -F: '{print $NF"\t" $1}' sample.txt  
/bin/false      dnsmasq 
/bin/false      mail 
/bin/bash       leo 
/sbin/nologin   awk 
/bin/false      messagebus 
/bin/false      mysql 
/bin/false      ntp 
/bin/bash       mat
/bin/bash       leone

 

Example 5

Printing Fields by Pattern

[email protected]:~> awk '/home/ {print $0}' sample.txt  
leo:x:506:506::/home/leo:/bin/bash 
awk:x:911:911::/home/awk:/sbin/nologin 
mat:x:502:502::/home/mat:/bin/bash
leone:x:507:507::/home/leone:/bin/bash

Here, the AWK prints all lines that match the pattern “home”.

Note: Here, $0 denotes the entire line in a field. You can also use the following command to get a similar output.

awk '/home/ {print}' sample.txt

 

Example 6

Count Lines

[email protected]:~> awk 'END{print NR}' sample.txt  
9

This emulates “wc -l”.

 

Example 7

Find all users who has user id greater than 500

[email protected]:~> awk -F: '$3 > 500 {print $1}' sample.txt  
leo 
awk 
mat 
leone

In the above example, first field ($3) is user id. So if $3 is greater than 500, then print its username (First Field).

 

Example 8

Print a particular line of a file

[email protected]:~> awk 'NR==9' sample.txt  
leone:x:507:507::/home/leone:/bin/bash

This will print the line 9 of the file sample.txt

 

Example 9

Print lines with more than 60 characters

[email protected]:~> awk 'length > 60' sample.txt  
mail:x:8:12:Mailer daemon:/var/spool/clientmqueue:/bin/false  
mysql:x:60:494:MySQL database admin:/var/lib/mysql:/bin/false

 

Example 10

To print all non-empty line.

awk 'NF > 0' sample.txt

 

Example 11

Next, we will use awk to print the UID and GID of all user’s starting with the string “leo”.

[email protected]:~> awk -F: '$1~/^leo/ {print $3 "\t" $4 }' sample.txt  
506     506 
507     507
  • “^” indicates start of a string.
  • “~” is used to match a regular expression match.

 

Example 12

Now, we will print the UID and GID of only the user “leo”.

[email protected]:~> awk -F: '$1~/^leo$/ {print $3 "\t" $4 }' sample.txt  
506     506
  • “$” indicates the end of a string.

 

Example 13

AWK Preprocessing with BEGIN

[email protected]:~> awk 'BEGIN {print "The File Contents:"} {print $0}' sample.txt  
The File Contents: 
dnsmasq:x:488:65534:dnsmasq:/var/lib/empty:/bin/false  
mail:x:8:12:Mailer daemon:/var/spool/clientmqueue:/bin/false  
leo:x:506:506::/home/leo:/bin/bash 
awk:x:911:911::/home/awk:/sbin/nologin 
messagebus:x:499:499:User for D-Bus:/run/dbus:/bin/false  
mysql:x:60:494:MySQL database admin:/var/lib/mysql:/bin/false  
ntp:x:74:488:NTP daemon:/var/lib/ntp:/bin/false 
mat:x:502:502::/home/mat:/bin/bash 
leone:x:507:507::/home/leone:/bin/bash

The contents inside BEGIN runs before processing the data.

 

Example 14

AWK Postprocessing using END

[email protected]:~> awk '{print $0} END {print "End of data"}' sample.txt  
dnsmasq:x:488:65534:dnsmasq:/var/lib/empty:/bin/false  
mail:x:8:12:Mailer daemon:/var/spool/clientmqueue:/bin/false  
leo:x:506:506::/home/leo:/bin/bash 
awk:x:911:911::/home/awk:/sbin/nologin 
messagebus:x:499:499:User for D-Bus:/run/dbus:/bin/false  
mysql:x:60:494:MySQL database admin:/var/lib/mysql:/bin/false  
ntp:x:74:488:NTP daemon:/var/lib/ntp:/bin/false 
mat:x:502:502::/home/mat:/bin/bash 
leone:x:507:507::/home/leone:/bin/bash 
End of data

The contents inside END runs after processing the data.

 

Example 15

BEGIN and END in a single AWK statement

[email protected]:~> awk 'BEGIN{print "Start";FS=":";OFS=" - "} {print $3,$4} END{print "End"}' sample.txt 
Start 
488 - 65534 
8 - 12 
506 - 506 
911 - 911 
499 - 499 
60 - 494 
74 - 488 
502 - 502 
507 - 507 
End

 

  • FS – Field Separator is “:”
  • OFS – Output Field Separator is ” – “

 

Example 16

System commands inside AWK using “system” function.

[email protected]:~> awk 'BEGIN { system("date")}' 
Mon Sep 18 19:45:01 IST 2017

System Function: Execute cmd and return the exit status.

 

Thanks for reading this blog post on “AWK command with examples”. These are very basic operations with AWK. In the next part, we will come up with more complex operations of AWK.

Thank you!

AWK command with examples
5 (100%) 3 votes

GeeksAlive

TheGeeksAlive is all about Linux, Server Administration, Web Hosting and other Open Source Technologies.