01 Introduction

In this episode I will describe how to calculate elapsed time in bash or other shell scripts.

While this may sound like a very simple and basic thing to do, there is a slightly more complex aspect to it if you wish to calculate elapsed time to a higher resolution than one second.

02

There are many reasons for calculating elapsed time in a shell script.

For example you may wish to simply report how long an operation took to run.

Another reason may be that you are trying to speed up a script and need to calculate benchmark data to see how different alternative methods perform.

03

What may seem like a simple task gets a bit more complicated if you want to do it for multiple different operating systems even if they are all unix related, as we shall see.

--------------------

04 Operating Systems Tested

For the purposes of this episode, I ran tests on the current version of the following operating systems.

Alma

Alpine

Debian

FreeBSD

OpenBSD

RaspberryPi

OpenSuse

Ubuntu 2604

Alma is a close copy of Red Hat that we can take as representing Red Hat style distros.

--------------------

05 Simple Low Resolution Timing

I will start with the simple and obvious method before describing the less obvious ones.

This uses the date command to get the current time in seconds since the unix epoch.

This is simply

date '+%s'

06

Save this to a variable using whatever method you prefer.

For example.

starttime=$(date '+%s')

07

Next, do whatever operations it is you wish to time.

Use the date command to get the current time again.

endtime=$(date '+%s')

08

Now simply subtract the start time from the end time using shell arithmetic.

This should be very obvious and basic.

--------------------

09 Higher Resolution Timing

However, suppose we wish to measure time to greater than one second of precision.

We need to do two things.

The first is to obtain the current time at a higher degree of precision.

The second is to conduct the calculations to a higher degree of precision.

10

Unfortunately, the standard time precision for POSIX shells seems to be 1 second.

Some shells offer a higher precision, but others do not.

Furthermore, standard shell arithmatic uses integer, which limits calculations to 1 second of precision.

--------------------

11 Bash High Resolution Shell Variable

Fortunately, bash is one that does offer a high precision date.

If you are using bash 5.0 or newer, there is a shell variable called EPOCHREALTIME which offers time since the the unix epoch (that is, since the first of January 1970, at 00:00:00 UTC) in seconds to 6 decimals of precision.

12

Example

echo $EPOCHREALTIME

1779634800.184926

13

This is related to the similar bash variable known as EPOCHSECONDS which gives the number of seconds since the unix epoch.

14

Example

echo $EPOCHSECONDS

1779634800

15

So if you are using bash, measuring time is very simple.

--------------------

16 But is it Really Bash?

Is your script however actually using bash?

Debian and derivatives actually have two shells.

The first, the interactive shell is bash.

The second, the non-interactive shell is dash, which stands for "debian almquist shell".

17

If you open a terminal, you get bash.

If your script starts with a "bin/bash" shebang line, you get bash.

However, if your script starts with a "bin/sh" shebang line, you get dash.

Some people find themselves getting caught out by this one when they try something out in a terminal but find that it doesn't work in their script which started with "bin/sh".

18

Many other, but not all, Linux distros use bash for both the interactive and non-interactive shells, so "bin/sh" and "bin/bash" work the same with those ones.

So if you intend to use bash, make sure your script calls for bash in the first line.

--------------------

19 The SHELL Variable

So how can a script tell what shell it is running under?

There is a shell variable called "SHELL" which will tell you the name of the shell.

Well, sort of.

20

On Debian and derivatives "SHELL" will say "bash" regardless of whether the actual shell is bash or dash.

On some other operating systems "SHELL" will simply say "sh" even if it is something else entirely.

So we need to do some additional levels of checking to see what we have.

21

To start with though, here's what each of the test distros reports for SHELL.

Alma : bash

Alpine : sh

Debian : bash

FreeBSD : sh

OpenBSD : ksh

Raspberry-Pi : bash

Suse : bash

ubuntu2604 : bash

--------------------

22 Bash Versus Dash

First, let's try to see which ones are bash and which ones are dash.

The first thing we can check is for the shell variable BASH_VERSION.

23

Example

echo $BASH_VERSION

If the shell is bash, then it will report a version string.

If the shell is not bash, then it will return an empty value.

24

Using this test, we can see that Alma and Opensuse are indeed using bash.

We however need to check Debian, Raspberry Pi, and Ubuntu when running in an "sh" script.

To check this we can use the "which" command to see what "sh" actually is.

25

Example

echo $( ls -l $(which sh ) | rev | cut -d" " -f1 | cut -d/ -f1 | rev )

26

"which sh" shows us the path to "sh"

However, that is a link so we need to use

"ls -l" to find the actual executable.

"rev" reverses the string.

27

"cut" takes the first element separated by spaces.

The second

"cut" takes the first element separated by the "/" characters.

The final "rev" takes that string and reverses it again to get it in the correct order.

28

In the case of Debian, Raspberry Pi, and Ubuntu it tells us that this is "dash".

--------------------

29 Openbsd

Openbsd reports its shell as "ksh", which stands for Korn Shell.

It is indeed Korn Shell, so we simply leave that one as is.

--------------------

30 Alpine and Freebsd

Next we have Alpine Linux and Freebsd, which both report as "sh".

In the case Freebsd there doesn't appear to be any further we can go that I am aware of.

It's simple "sh".

It is a basic POSIX shell which seems to be similar to the original unix shell, the Bourne Shell.

Older versions of Freebsd used a different shell known as tsch (the C shell), but I haven't tested that so I will ignore that here.

31

With Alpine Linux however, we can get the actual shell using the same method that we used for Debian Linux.

This reports as being "busybox".

32

Busybox is a limited shell intended for use in embedded systems.

Alpine was originally an embedded distro, but some people started using it for containers.

Alpine is Linux, but it is not GNU/Linux, and there are a number of areas which can trip you up if you are not aware of them.

So, be extra careful if you are using it for anything, and test everything.

--------------------

33 Summary of Actual Shells

Here is our revised list with the actual shell used when asking for "sh", so far as we can determine.

Alma : bash

Alpine : busybox

Debian : dash

FreeBSD : sh

OpenBSD : ksh

Raspberry-Pi : dash

Suse : bash

ubuntu2604 : dash

34

There are other shells, but none of them are the default shell for any of the distros on our list, so I haven't tested them.

--------------------

35 Solutions for Measuring Time

Now we need to find solutions for bash, dash, ksh, sh, and busybox.

--------------------

36 Bash

For bash, we can simply use EPOCHREALTIME, as mentioned above.

--------------------

37 Dash

For dash, we can use the date command.

This is a very conventional method, and is probably the first answer that anyone would give for this situation.

However, while it will work in most cases, it will not work in all cases, so it is not a universal solution.

38

To use date we simply call it with the correct format string.

This uses %s to get seconds since the epoch, and %N to get nanoseconds of the current second.

If you put a decimal separator between the two it will appear in the output.

You can use the correct decimal separator for your locale, but I won't go into that here.

Instead I will just assume a period or dot.

39

Example

date '+%s.%N'

1779634800.358916385

--------------------

40 Problems with Date on Alpine and Openbsd

Date will work for bash, dash, and sh on Freebsd.

However it will not work for ksh on Openbsd, or for busybox on Alpine.

41

With busybox on Alpine, it simply ignores the %N format specifier and prints out the epoch in seconds only followed by the decimal separator.

=

Example

date '+%s.%N'

1779634800.

42

With ksh on Openbsd it prints the epoch in seconds followed by the decimal separator and then the %N as a literal N.

date '+%s.%N'

1779634800.N

43

Fortunately we have alternatives for these two cases.

--------------------

44 Openbsd

Openbsd has the "ts" or timestamp utility installed by default.

ts prints a time stamp in front of every line it receives from standard input.

I won't go into details on all aspects of ts here, I'll leave that to someone else.

Instead I will focus on how to use it for our specific purposes here.

45

We need to provide a format specifier to ts, which in this case is "%.s"

We also need to provide something for standard input, or otherwise ts will simply sit there and wait for input.

So what we need to do is to echo nothing through a pipe to ts while also giving ts the proper format specifier.

46

Example

echo | ts "%.s"

47

This will output the epoch time in seconds to six decimals of precision.

ts is installed in Openbsd and Freebsd by default and can be used in either.

It can also be installed in many other distros.

--------------------

48 Busybox on Alpine

None of the methods discussed so far will work for busybox on Alpine though.

However there is a way, but it's a bit non obvious and somewhat hacky.

49

Busybox includes a command called "adjtimex".

This is normally used to adjust the time hardware.

However if it is run without arguments, it will report the current settings.

50

These include the current epoch time in seconds , and in another field the time in microseconds.

These are reported as key value pairs.

So what we need to do is to do the following

51

Run adjtimex

Capture the output.

Grep for "time.tv_sec"

Grep for "time.tv_usec"

Use cut to extract the time value in each case.

Use tr to get rid of excess spaces in each case.

Combine the two in a string with a decimal separator between them.

52

This takes a total of 4 lines of shell script.

I will just describe them breifly here, see the show notes for details.

53

First we want to capture the output of adjtimex in a single operation.

Run adjtimex and pipe the output through grep to capture lines containing "time.tv_"

and save this to a variable.

# Extract the current high resolution time from adjtimex.

# We want two key value pairs, identified by time.tv_sec and time.tv_usec.

tvals=$( adjtimex | grep "time.tv_" )

54

Next echo the contents of this variable and pipe it through grep, cut, and tr to get first the seconds and then the microseconds while also removing excess spaces.

Save these to two separate variables.

"time.tv_sec" is the time in seconds since the epoch.

"time.tv_usec" is the number of microseconds in the current second.

# Get the time since the unix epoch in seconds and micro-seconds.

timesec=$( echo "$tvals" | grep "time.tv_sec" | cut -d: -f2 | tr -d " " )

timeusec=$( echo "$tvals" | grep "time.tv_usec" | cut -d: -f2 | tr -d " " )

55

Adjtimex does not zero pad the microsecond time value to provide leading zeros, so we need to take care of this using printf before we can append it to the seconds value. We didn't need to do this with date where the %N format character does this automatically.

In this instance, the printf format string is '%06d'

padusec=$( printf '%06d' $timeusec )

Now, combine these into a single number with a decimal separator by using simple string concatenation.

# Combine them into a single number.

timehires="$timesec"".""$padusec"

--------------------

56 Summary of Methods

Let's summarize where we are so far in terms of methods we can use to get the current time as a high resolution number.

Alma : use EPOCHREALTIME or date

Debian (bash) : use EPOCHREALTIME or date

Raspberry-Pi (bash) : use EPOCHREALTIME or date

ubuntu2604 (bash) : use EPOCHREALTIME or date

Suse : use EPOCHREALTIME or date

Debian (dash) : use date

Raspberry-Pi (dash) : use date

ubuntu2604 (dash) : use date

Alpine : use adjtimex and parse the output

FreeBSD : use date or ts

OpenBSD : use ts

--------------------

57 Other alternatives

There are a few alternatives that we haven't discussed yet.

58 Bash with Dash

In the case of Debian, Raspberry Pi, and Ubuntu running dash, since bash is available it is possible to write a separate bash script which simply echos EPOCHREALTIME and then call it from the dash script and capture the output.

While this would work, there's probably not a lot of point to it.

If you can rely on bash being there, then just change the first line of the script and make it a bash script.

59 Adding Packages to Alpine

The ts or timestamp utility is a common unix utility that can be installed if it is not present by default.

This does produce high resolution timestamps on Alpine.

On Alpine Linux this comes as part of the "moreutils" package.

To add the package, use the following

sudo apk add moreutils

echo | ts "%.s"

1779634800.959948

60

You can also add the GNU coreutils, which will provide a high resolution date command which works like in the other examples.

To add the package use the following

sudo apk add coreutils

date '+%s.%N'

1779634800.212897332

61

If you can install more packages into your Alpine system, either of the above two is probably going to be preferable to parsing the output of adjtimex.

62 Custom Timestamp Programs

You can also write a very short program in python, perl, tcl, or some other language and have it output the current epoch time.

I won't discuss that here though.

--------------------

63 Calculating Time Differences

Shell arithmetic is integer only.

If we wish to use high resolution timing data, we need to do something so we don't lose the precision we have worked so hard to get.

There are several possible solutions.

64 Change the Time Base

One method is to change the time base from seconds to milli, micro, or nanoseconds.

This can be done by simply multiplying the time values by the appropriate amount (e.g. 1000, 1,000,000, etc.) before subtracting them.

This allows for integer arithmetic on high resolution values without losing precision.

65 Use the Shell bc Arbitrary Precision Calculator

The bc command line calculator will perform calculations using real numbers and is easy to use in scripts.

It is present by default in most distros.

echo "scale=9; $endtime - $starttime" | bc

where endtime and starttime are variables containing time values.

66

However, for some inexplicable reason, neither Debian nor Opensuse install it by default.

It is present in Ubuntu and Raspberry Pi which are Debian derivatives, and it can be added to distros which lack it.

67 Use awk

awk can also perform calculations using real numbers and it is present in nearly all distros including in all of the ones we tested here.

echo "$endtime $starttime" | awk '{printf "%.6f\n", $1 - $2}'

--------------------

68 Benchmarks

And of course no comparative evaluation would be complete without benchmarks where we see how each method compares to another in terms of speed.

In the benchmark test I ran each method in a loop through multiple iterations, measured the elapsed time, subtracted out the time for an empty loop, and then compared it to alternate methods.

For anything other than EPOCHREALTIME, the empty loop time is negligible and has no real effect on the results.

69

Rather interestingly I came across a bug which caused date to run very slowly if called immediately after using EPOCHREALTIME in bash.

The effect of the bug was to make the date benchmark test roughly 24 times slower.

This has been fixed in newer releases, but if you are using an older distro release then beware of this bug.

I was able to get around it either putting a sleep delay between benchmarking EPOCHREALTIME and benchmarking date, or by simply testing date before testing EPOCHREALTIME.

70

To be able to conduct additional tests I installed ts in Ubuntu and Alpine, and the GNU version of date in Alpine.

71 EPOCHREALTIME Versus date in Ubuntu 2604 bash

The EPOCHREALTIME method is 3103 times faster than date.

However, when the same test is run on Ubuntu 2404 when the date test is run before the EPOCHREALTIME test, EPOCHREALTIME is 1240 faster than date.

Other Linux distros show performance to Ubuntu 2404.

It appears that a side effect of fixing whatever the bug is has the effect of slowing down date.

However, this is probably not a significant issue in normal circumstances.

72 date versus ts in Ubuntu 2604 bash

The date method is 3.7 times faster than ts

73 date versus ts in Ubuntu 2604 dash

The date method is 4.9 times faster than ts

74 date versus ts in Freebsd sh

The date method is 2.5 times faster than ts

75 date versus adjtimex in Alpine Busybox

The date method is 6.0 times faster than adjtimex

76 date versus ts in Alpine Busybox

The date method is 20.0 times faster than ts

77 bc versus awk in Ubuntu 2604

I compared calculating the difference between two numbers when using bc versus awk.

The difference is negligible, with bc being only 7% faster than awk.

78 Conclusion for Benchmarks

Based on these results, if you need to measure elapsed time to high resolution and care about runing the command with as little overhead as possible, then the order of preference should be the following.

79

If you are using a newer version of bash, then use EPOCHREALTIME.

If that is not available, then use date, provided it allows for high resolution times.

If the above two cannot be used, then use ts.

If you are using Busybox and cannot install either GNU date or ts, then use adjtimex.

Date is the closest in terms of being the universal portable solution, but it does not work in all cases.

80

I have not compared different platforms to each other in terms of performance, as that would be a much more involved problem that is outside the scope of this episode.

However, different operating systems implement different commands in different ways.

81

For example, on Openbsd and Freebsd, ts appears to be an ELF binary. That is, it is executable machine code, possibly written in C.

On Ubuntu however, ts appears to be a perl script.

As a result of this, the advantage that date has over ts is much less in Freebsd than it is with Ubuntu (and likely other Linux distros) as on Freebsd it doesn't need to load a perl interpreter to run ts.

--------------------

82 Overall Conclusion

You no doubt thought that measuring elapsed time was going to be so simple, and how could someone get an entire podcast out of such a simple subject?

And yet here we are half an hour later with just a basic overview of the subject.

83

I hope you found this interesting and informative.

Please let us know in the comments if you think that I have done anything incorrectly, or if you have another way of doing things.

I hope to see you all again in another future episode of HPR.

--------------------