coreutils: dd invocation

1 
1 11.2 ‘dd’: Convert and copy a file
1 ==================================
1 
1 ‘dd’ copies a file (from standard input to standard output, by default)
1 with a changeable I/O block size, while optionally performing
1 conversions on it.  Synopses:
1 
1      dd [OPERAND]...
1      dd OPTION
1 
11    The only options are ‘--help’ and ‘--version’.  ⇒Common
 options.  ‘dd’ accepts the following operands, whose syntax was
1 inspired by the DD (data definition) statement of OS/360 JCL.
1 
1 ‘if=FILE’
1      Read from FILE instead of standard input.
1 
1 ‘of=FILE’
1      Write to FILE instead of standard output.  Unless ‘conv=notrunc’ is
1      given, ‘dd’ truncates FILE to zero bytes (or the size specified
1      with ‘seek=’).
1 
1 ‘ibs=BYTES’
1      Set the input block size to BYTES.  This makes ‘dd’ read BYTES per
1      block.  The default is 512 bytes.
1 
1 ‘obs=BYTES’
1      Set the output block size to BYTES.  This makes ‘dd’ write BYTES
1      per block.  The default is 512 bytes.
1 
1 ‘bs=BYTES’
1      Set both input and output block sizes to BYTES.  This makes ‘dd’
1      read and write BYTES per block, overriding any ‘ibs’ and ‘obs’
1      settings.  In addition, if no data-transforming ‘conv’ option is
1      specified, input is copied to the output as soon as it’s read, even
1      if it is smaller than the block size.
1 
1 ‘cbs=BYTES’
1      Set the conversion block size to BYTES.  When converting
1      variable-length records to fixed-length ones (‘conv=block’) or the
1      reverse (‘conv=unblock’), use BYTES as the fixed record length.
1 
1 ‘skip=N’
1      Skip N ‘ibs’-byte blocks in the input file before copying.  If
1      ‘iflag=skip_bytes’ is specified, N is interpreted as a byte count
1      rather than a block count.
1 
1 ‘seek=N’
1      Skip N ‘obs’-byte blocks in the output file before copying.  if
1      ‘oflag=seek_bytes’ is specified, N is interpreted as a byte count
1      rather than a block count.
1 
1 ‘count=N’
1      Copy N ‘ibs’-byte blocks from the input file, instead of everything
1      until the end of the file.  if ‘iflag=count_bytes’ is specified, N
1      is interpreted as a byte count rather than a block count.  Note if
1      the input may return short reads as could be the case when reading
1      from a pipe for example, ‘iflag=fullblock’ will ensure that
1      ‘count=’ corresponds to complete input blocks rather than the
1      traditional POSIX specified behavior of counting input read
1      operations.
1 
1 ‘status=LEVEL’
1      Transfer information is normally output to stderr upon receipt of
1      the ‘INFO’ signal or when ‘dd’ exits.  Specifying LEVEL will adjust
1      the amount of information printed, with the last LEVEL specified
1      taking precedence.
1 
1      ‘none’
1           Do not print any informational or warning messages to stderr.
1           Error messages are output as normal.
1 
1      ‘noxfer’
1           Do not print the final transfer rate and volume statistics
1           that normally make up the last status line.
1 
1      ‘progress’
1           Print the transfer rate and volume statistics on stderr, when
1           processing each input block.  Statistics are output on a
1           single line at most once every second, but updates can be
1           delayed when waiting on I/O.
1 
1 ‘conv=CONVERSION[,CONVERSION]...’
1      Convert the file as specified by the CONVERSION argument(s).  (No
1      spaces around any comma(s).)
1 
1      Conversions:
1 
1      ‘ascii’
1           Convert EBCDIC to ASCII, using the conversion table specified
1           by POSIX.  This provides a 1:1 translation for all 256 bytes.
1           This option implies ‘conv=unblock’; input is converted to
1           ASCII before trailing spaces are deleted.
1 
1      ‘ebcdic’
1           Convert ASCII to EBCDIC.  This is the inverse of the ‘ascii’
1           conversion.  This option implies ‘conv=block’; trailing spaces
1           are added before being converted to EBCDIC.
1 
1      ‘ibm’
1           This acts like ‘conv=ebcdic’, except it uses the alternate
1           conversion table specified by POSIX.  This is not a 1:1
1           translation, but reflects common historical practice for ‘~’,
1           ‘[’, and ‘]’.
1 
1           The ‘ascii’, ‘ebcdic’, and ‘ibm’ conversions are mutually
1           exclusive.  If you use any of these options, you should also
1           use the ‘cbs=’ option.
1 
1      ‘block’
1           For each line in the input, output ‘cbs’ bytes, replacing the
1           input newline with a space and padding with spaces as
1           necessary.
1 
1      ‘unblock’
1           Remove any trailing spaces in each ‘cbs’-sized input block,
1           and append a newline.
1 
1           The ‘block’ and ‘unblock’ conversions are mutually exclusive.
1 
1      ‘lcase’
1           Change uppercase letters to lowercase.
1 
1      ‘ucase’
1           Change lowercase letters to uppercase.
1 
1           The ‘lcase’ and ‘ucase’ conversions are mutually exclusive.
1 
1      ‘sparse’
1           Try to seek rather than write NUL output blocks.  On a file
1           system that supports sparse files, this will create sparse
1           output when extending the output file.  Be careful when using
1           this option in conjunction with ‘conv=notrunc’ or
1           ‘oflag=append’.  With ‘conv=notrunc’, existing data in the
1           output file corresponding to NUL blocks from the input, will
1           be untouched.  With ‘oflag=append’ the seeks performed will be
1           ineffective.  Similarly, when the output is a device rather
1           than a file, NUL input blocks are not copied, and therefore
1           this option is most useful with virtual or pre zeroed devices.
1 
1      ‘swab’
1           Swap every pair of input bytes.  GNU ‘dd’, unlike others,
1           works when an odd number of bytes are read—the last byte is
1           simply copied (since there is nothing to swap it with).
1 
1      ‘sync’
1           Pad every input block to size of ‘ibs’ with trailing zero
1           bytes.  When used with ‘block’ or ‘unblock’, pad with spaces
1           instead of zero bytes.
1 
1      The following “conversions” are really file flags and don’t affect
1      internal processing:
1 
1      ‘excl’
1           Fail if the output file already exists; ‘dd’ must create the
1           output file itself.
1 
1      ‘nocreat’
1           Do not create the output file; the output file must already
1           exist.
1 
1           The ‘excl’ and ‘nocreat’ conversions are mutually exclusive.
1 
1      ‘notrunc’
1           Do not truncate the output file.
1 
1      ‘noerror’
1           Continue after read errors.
1 
1      ‘fdatasync’
1           Synchronize output data just before finishing.  This forces a
1           physical write of output data.
1 
1      ‘fsync’
1           Synchronize output data and metadata just before finishing.
1           This forces a physical write of output data and metadata.
1 
1 ‘iflag=FLAG[,FLAG]...’
1      Access the input file using the flags specified by the FLAG
1      argument(s).  (No spaces around any comma(s).)
1 
1 ‘oflag=FLAG[,FLAG]...’
1      Access the output file using the flags specified by the FLAG
1      argument(s).  (No spaces around any comma(s).)
1 
1      Here are the flags.  Not every flag is supported on every operating
1      system.
1 
1      ‘append’
1           Write in append mode, so that even if some other process is
1           writing to this file, every ‘dd’ write will append to the
1           current contents of the file.  This flag makes sense only for
1           output.  If you combine this flag with the ‘of=FILE’ operand,
1           you should also specify ‘conv=notrunc’ unless you want the
1           output file to be truncated before being appended to.
1 
1      ‘cio’
1           Use concurrent I/O mode for data.  This mode performs direct
1           I/O and drops the POSIX requirement to serialize all I/O to
1           the same file.  A file cannot be opened in CIO mode and with a
1           standard open at the same time.
1 
1      ‘direct’
1           Use direct I/O for data, avoiding the buffer cache.  Note that
1           the kernel may impose restrictions on read or write buffer
1           sizes.  For example, with an ext4 destination file system and
1           a Linux-based kernel, using ‘oflag=direct’ will cause writes
1           to fail with ‘EINVAL’ if the output buffer size is not a
1           multiple of 512.
1 
1      ‘directory’
1 
1           Fail unless the file is a directory.  Most operating systems
1           do not allow I/O to a directory, so this flag has limited
1           utility.
1 
1      ‘dsync’
1           Use synchronized I/O for data.  For the output file, this
1           forces a physical write of output data on each write.  For the
1           input file, this flag can matter when reading from a remote
1           file that has been written to synchronously by some other
1           process.  Metadata (e.g., last-access and last-modified time)
1           is not necessarily synchronized.
1 
1      ‘sync’
1           Use synchronized I/O for both data and metadata.
1 
1      ‘nocache’
1           Request to discard the system data cache for a file.  When
1           count=0 all cached data for the file is specified, otherwise
1           the cache is dropped for the processed portion of the file.
1           Also when count=0, failure to discard the cache is diagnosed
1           and reflected in the exit status.
1 
1           Note data that is not already persisted to storage will not be
1           discarded from cache, so note the use of the “sync” options in
1           the examples below, which are used to maximize the
1           effectiveness of the ‘nocache’ flag.
1 
1           Here are some usage examples:
1 
1                # Advise to drop cache for whole file
1                dd if=ifile iflag=nocache count=0
1 
1                # Ensure drop cache for the whole file
1                dd of=ofile oflag=nocache conv=notrunc,fdatasync count=0
1 
1                # Advise to drop cache for part of file
1                # Note the kernel will only consider complete and
1                # already persisted pages.
1                dd if=ifile iflag=nocache skip=10 count=10 of=/dev/null
1 
1                # Stream data using just the read-ahead cache.
1                # See also the ‘direct’ flag.
1                dd if=ifile of=ofile iflag=nocache oflag=nocache,sync
1 
1      ‘nonblock’
1           Use non-blocking I/O.
1 
1      ‘noatime’
11           Do not update the file’s access timestamp.  ⇒File
           timestamps.  Some older file systems silently ignore this
1           flag, so it is a good idea to test it on your files before
1           relying on it.
1 
1      ‘noctty’
1           Do not assign the file to be a controlling terminal for ‘dd’.
1           This has no effect when the file is not a terminal.  On many
1           hosts (e.g., GNU/Linux hosts), this option has no effect at
1           all.
1 
1      ‘nofollow’
1           Do not follow symbolic links.
1 
1      ‘nolinks’
1           Fail if the file has multiple hard links.
1 
1      ‘binary’
1           Use binary I/O.  This option has an effect only on nonstandard
1           platforms that distinguish binary from text I/O.
1 
1      ‘text’
1           Use text I/O.  Like ‘binary’, this option has no effect on
1           standard platforms.
1 
1      ‘fullblock’
1           Accumulate full blocks from input.  The ‘read’ system call may
1           return early if a full block is not available.  When that
1           happens, continue calling ‘read’ to fill the remainder of the
1           block.  This flag can be used only with ‘iflag’.  This flag is
1           useful with pipes for example as they may return short reads.
1           In that case, this flag is needed to ensure that a ‘count=’
1           argument is interpreted as a block count rather than a count
1           of read operations.
1 
1      ‘count_bytes’
1           Interpret the ‘count=’ operand as a byte count, rather than a
1           block count, which allows specifying a length that is not a
1           multiple of the I/O block size.  This flag can be used only
1           with ‘iflag’.
1 
1      ‘skip_bytes’
1           Interpret the ‘skip=’ operand as a byte count, rather than a
1           block count, which allows specifying an offset that is not a
1           multiple of the I/O block size.  This flag can be used only
1           with ‘iflag’.
1 
1      ‘seek_bytes’
1           Interpret the ‘seek=’ operand as a byte count, rather than a
1           block count, which allows specifying an offset that is not a
1           multiple of the I/O block size.  This flag can be used only
1           with ‘oflag’.
1 
1      These flags are not supported on all systems, and ‘dd’ rejects
1      attempts to use them when they are not supported.  When reading
1      from standard input or writing to standard output, the ‘nofollow’
1      and ‘noctty’ flags should not be specified, and the other flags
1      (e.g., ‘nonblock’) can affect how other processes behave with the
1      affected file descriptors, even after ‘dd’ exits.
1 
1    The numeric-valued strings above (N and BYTES) can be followed by a
1 multiplier: ‘b’=512, ‘c’=1, ‘w’=2, ‘xM’=M, or any of the standard block
1 size suffixes like ‘k’=1024 (⇒Block size).
1 
1    Any block size you specify via ‘bs=’, ‘ibs=’, ‘obs=’, ‘cbs=’ should
1 not be too large—values larger than a few megabytes are generally
1 wasteful or (as in the gigabyte..exabyte case) downright
1 counterproductive or error-inducing.
1 
1    To process data that is at an offset or size that is not a multiple
1 of the I/O block size, you can use the ‘skip_bytes’, ‘seek_bytes’ and
1 ‘count_bytes’ flags.  Alternatively the traditional method of separate
1 ‘dd’ invocations can be used.  For example, the following shell commands
1 copy data in 512 KiB blocks between a disk and a tape, but do not save
1 or restore a 4 KiB label at the start of the disk:
1 
1      disk=/dev/rdsk/c0t1d0s2
1      tape=/dev/rmt/0
1 
1      # Copy all but the label from disk to tape.
1      (dd bs=4k skip=1 count=0 && dd bs=512k) <$disk >$tape
1 
1      # Copy from tape back to disk, but leave the disk label alone.
1      (dd bs=4k seek=1 count=0 && dd bs=512k) <$tape >$disk
1 
1    For failing disks, other tools come with a great variety of extra
1 functionality to ease the saving of as much data as possible before the
1 disk finally dies, e.g.  GNU ‘ddrescue’
1 (https://www.gnu.org/software/ddrescue/).  However, in some cases such a
1 tool is not available or the administrator feels more comfortable with
1 the handling of ‘dd’.  As a simple rescue method, call ‘dd’ as shown in
1 the following example: the options ‘conv=noerror,sync’ are used to
1 continue after read errors and to pad out bad reads with NULs, while
1 ‘iflag=fullblock’ caters for short reads (which traditionally never
1 occur on disk based devices):
1 
1      # Rescue data from an (unmounted!) partition of a failing disk.
1      dd conv=noerror,sync iflag=fullblock </dev/sda1 > /mnt/rescue.img
1 
1    Sending an ‘INFO’ signal (or ‘USR1’ signal where that is unavailable)
1 to a running ‘dd’ process makes it print I/O statistics to standard
1 error and then resume copying.  In the example below, ‘dd’ is run in the
1 background to copy 5GB of data.  The ‘kill’ command makes it output
1 intermediate I/O statistics, and when ‘dd’ completes normally or is
1 killed by the ‘SIGINT’ signal, it outputs the final statistics.
1 
1      # Ignore the signal so we never inadvertently terminate the dd child.
1      # Note this is not needed when SIGINFO is available.
1      trap '' USR1
1 
1      # Run dd with the fullblock iflag to avoid short reads
1      # which can be triggered by reception of signals.
1      dd iflag=fullblock if=/dev/zero of=/dev/null count=5000000 bs=1000 & pid=$!
1 
1      # Output stats every second.
1      while kill -s USR1 $pid 2>/dev/null; do sleep 1; done
1 
1    The above script will output in the following format:
1 
1      3441325+0 records in
1      3441325+0 records out
1      3441325000 bytes (3.4 GB, 3.2 GiB) copied, 1.00036 s, 3.4 GB/s
1      5000000+0 records in
1      5000000+0 records out
1      5000000000 bytes (5.0 GB, 4.7 GiB) copied, 1.44433 s, 3.5 GB/s
1 
1    The ‘status=progress’ option periodically updates the last line of
1 the transfer statistics above.
1 
1    On systems lacking the ‘INFO’ signal ‘dd’ responds to the ‘USR1’
1 signal instead, unless the ‘POSIXLY_CORRECT’ environment variable is
1 set.
1 
1    An exit status of zero indicates success, and a nonzero value
1 indicates failure.
1