[Question]
  [
What general tips do you have for golfing in sed? I'm looking for ideas which can be applied to code-golf problems and which are also at least somewhat specific to sed (e.g. "remove comments" is not an answer).
Please post one tip per answer.
  
[Answer]
If you need to use [labels](https://www.gnu.org/software/sed/manual/sed.html#Programming-Commands) then for sure you'll want your label names to be as short as possible. In fact taken to the extreme, you may even use the empty string as a label name:
```
:    # define label ""
p    # print pattern space
b    # infinite loop! - branch to label ""
```
[Answer]
The [GNU sed documentation](https://www.gnu.org/software/sed/manual/sed.html) describes the `s` command as *"sed's Swiss Army Knife"*. But if all you want to do is replace all instances of one character with another, then the `y` command is what you need:
```
y/a/b/
```
is one char shorter than:
```
s/a/b/g
```
[Answer]
When repeatedly replacing in a loop:
```
loop:
s/foo/bar/g
tloop
```
it's usually unnecessary to replace globally, as the loop will eventually replace all occurrences:
```
# GNU sed
:
s/foo/bar/
t
```
Note also the GNU extension above: a label can have an empty name, saving more precious bytes. In other implementations, a label cannot be empty, and jumping without a label transfers flow to the end of script (i.e. same as `n`).
[Answer]
Consider using extended regex syntax (in GNU sed). The `-r` option costs one byte in scoring, but using it just once to eliminate the backslashes from a pair of `\(...\)` has already paid for itself.
[Answer]
As mentioned in `man sed` (GNU), you can use any character as a delimiter for regular expressions by using the syntax
```
\%regexp%
```
where `%` is a placeholder for any character.
This is useful for commands like
```
/^http:\/\//
```
which are shorter as
```
\%^http://%
```
What is mentioned in the [GNU sed manual](https://www.gnu.org/software/sed/manual/sed.html#Regular-Expressions) but **not** in `man sed` is that you can change the delimiters of `s///` and `y///` as well.
For example, the command
```
ss/ssg
```
removes all slashes from the pattern space.
[Answer]
# Read the whole input at once with `-z`
Often you need to operate on the whole input at once instead of one line at a time. The `N` command is useful for that:
```
:
$!{N;b}
```
...but usually you can skip it and use the `-z` flag instead.
The `-z` flag makes sed use NUL (`\0`) as its input line separator instead of `\n`, so if you know your input won’t contain `\0`, it will read all of the input at once as a single “line”:
```
$ echo 'foo
> bar
> baz' | sed -z '1y/ao/eu/'
fuu
ber
bez
```
[Try it online!](https://tio.run/##S0oszvj/PzU5I19BPS0/nyspsQiIq9QVahSKU1MUdKsU1A0r9RPz9VNL9dX//wcA "Bash – Try It Online")
[Answer]
There's no built-in arithmetic, but calculations can be done in unary or in unary-coded decimal. The following code converts decimal to UCD, with x as the unit and 0 as the digits separator:
```
s/[1-9]/0&/g
s/[5-9]/4&/g
y/8/4/
s/9/4&/g
s/4/22/g
s/[37]/2x/g
s/[26]/xx/g
s/[1-9]/x/g
```
and here's the conversion back to decimal:
```
s/0x/-x/g
s/xx/2/g
y/x/1/
s/22/4/g
s/44/8/g
s/81/9/g
s/42/6/g
s/21/3/g
s/61/7/g
s/41/5/g
s/-//g
```
These are both taken from [an answer to "Multiply two numbers without using any numbers"](/a/49442).
Plain old unary can be converted using this pair of loops from [this answer to "{Curly Numbers};"](/a/58007), where the unit is `;`. I've used `v` and `x` to match Roman for `5` and `10`; `b` comes from "bis".
```
# unary to decimal
:d
/;/{
s/;;;;;/v/g
s/vv/x/g
/[;v]/!s/x\+/&0/
s/;;/b/g
s/bb/4/
s/b;/3/
s/v;/6/
s/vb/7/
s/v3/8/
s/v4/9/
y/;bvx/125;/
td
}
# Decimal to unary
:u
s/\b9/;8/
s/\b8/;7/
s/\b7/;6/
s/\b6/;5/
s/\b5/;4/
s/\b4/;3/
s/\b3/;2/
s/\b2/;1/
s/\b1/;0/
s/\b0//
/[^;]/s/;/&&&&&&&&&&/g
tu
```
[Answer]
# Append a newline in one byte
The `G` command appends a newline and the contents of the hold space to the pattern space, so if your hold space is empty, instead of this:
```
s/$/\n/
```
You can do this:
```
G
```
# Prepend a newline in three bytes
The `H` command appends a newline and the contents of the pattern space to the hold space, and `x` swaps the two, so if your hold space is empty, instead of this:
```
s/^/\n/
```
You can do this:
```
H;x
```
This will pollute your hold space, so it only works once. For two more bytes, though, you could clear your pattern space before swapping, which is still a savings of two bytes:
```
H;z;x
```
[Answer]
If not explicitly banned by the question, the consensus for [this meta question](http://meta.codegolf.stackexchange.com/questions/5343/can-numeric-input-output-be-in-unary) is that numerical input may be in unary. This saves you the 86 bytes of decimal to unary as per [this answer](https://codegolf.stackexchange.com/a/51329/11259).
[Answer]
I know this is an old thread, but I just found those clumsy decimal to UCD converters, with almost a hundred bytes, some even messing the hold space or requiring special faulty `sed` versions.
For **decimal to UCD** I use (68 bytes; former best posted here 87 bytes)
```
s/$/\n9876543210/
:a
s/\([1-9]\)\(.*\n.*\)\1\(.\)/\3x\2\1\3/
ta
P;d
```
**UCD to decimal** is (also 66 bytes; former best posted here 96)
```
s/$/\n0123456789/
:a      
s/\([0-8]\)x\(.*\n.*\)\1\(.\)/\3\2\1\3/
ta      
P;d
```
* `\n` in the replacement is not portable. You can use a different character instead and save two bytes, but you'll need more bytes to remove the appendix instead of `P;d`; see next remark. Or, if your hold space is empty, do `G;s/$/9876543210/` without byte penalty.
* If you need further processing, you'll need some more bytes for `s/\n.*//` instead of `P;d`.
* You could save two bytes each for those buggy old GNU `sed` versions
* No, you can't save those six backslashes as extended regular expressions don't do backreferences
[Answer]
## Empty regexes are equivalent to the previously encountered regex
(thanks to [Riley](https://codegolf.stackexchange.com/users/57100/riley) for discovering this from an [anagol submission](http://golf.shinh.org/reveal.rb?ASCII+Pylon/mitchs+%28tails%29_1524733941&sed))
Here is an example where we are tasked with creating 100 `@`s in an empty buffer.
```
s/$/@@@@@@@@@@/;s/.*/&&&&&&&&&&/ # 31 bytes
s/.*/@@@@@@@@@@/;s//&&&&&&&&&&/  # 30 bytes
```
The second solution is 1 byte shorter and uses the fact that empty regexes are filled in with the last encountered regex. Here, for the second substitution, the last regex was `.*`, so the empty regex here will be filled with `.*`. This also works with regexes in `/conditionals/`.
Note that it is the previously *encountered* regex, so the following would also work.
```
s/.*/@@@@@@@@@@/;/@*/!s/$/@/;s//&&&&&&&&&&/
```
The empty regex gets filled with `@*` instead of `$` because `s/$/@/` is never reached.
[Answer]
# Unary-UCD-Decimal conversions
The programs below are adapted from solutions in [anarchy golf](http://golf.shinh.org/). The solutions below work on GNU sed 4.2.2, where empty labels are allowed. It is possible I've missed better solutions on anarchy golf.
## Unary to Decimal, `sed -r` at 47 bytes
From solutions by [%20](http://golf.shinh.org/reveal.rb?Collatz+Problem/%2520_1480130793&sed) and [tails](http://golf.shinh.org/reveal.rb?Bijective+base+10+add/tails_1536134169&sed). Works only for integers greater than 0. The chosen unary digit (`-` in the below program) occurs 3 times in the program.
```
:
s/-/<<123456789-01>/
s/(.)<.*\1(-*.).*>/\2/
t
```
[Try it online!](https://tio.run/##K05N@f/fiqtYX1ffxsbQyNjE1MzcwlLXwNBOHyiooadpo6cVY6ihq6Wnqadlpx9jpM9V8v@/LtHgX35BSWZ@XvF/3SIA "sed 4.2.2 – Try It Online") and [(bash wrapper for easier testing)](https://tio.run/##FcgxCoQwEEDR3lNMsaAGJuPEddeV4EnSKAloo2LSCLn7uFb/8ecpLnKFmJcwecAX53QCenBue3QBZtqPRDF4mtftKeAJpQxFJCRr2bTv7vPtf9jwSP9Z6dpq5bhCpWutRnKGiiSlSNsYvgE "Bash – Try It Online")
## Decimal to UCD, `sed` at 39 bytes
From a solution by [%20](http://golf.shinh.org/reveal.rb?Collatz+Problem/%2520_1480130793&sed). Works on all integers, including negative ones.
```
:
s/[1-9]/&;/g
y/123456789/012345678/
t
```
[Try it online!](https://tio.run/##K05N@f/fiqtYP9pQ1zJWX81aP52rUt/QyNjE1MzcwlLfAMbU5yr5/9/QwNLsX35BSWZ@XvF/3SIA "sed 4.2.2 – Try It Online")
## Decimal to Unary (via UCD), `sed` 55 bytes
From a solution by [tails](http://golf.shinh.org/reveal.rb?Boxception/tails_1626672646&sed). Works on all integers, including negative ones.
```
:
s/[1-9]/&;/g
y/123456789/012345678/
s/;0/9;/
t
s/0//g
```
[Try it online!](https://tio.run/##K05N@f/fiqtYP9pQ1zJWX81aP52rUt/QyNjE1MzcwlLfAMbUB6qxNtC3tNbnKgEyDfT10///NzKwBAA "sed 4.2.2 – Try It Online") and [(bash wrapper)](https://tio.run/##S0oszvivn19Qol@cmqKflJkHohXU/1txFetHG@paxuqrWeunc1XqGxoZm5iamVtY6hvAmPpANdYG@pbW@lwlQKaBvn76f/WatPycFAVdw5rSvMxCBd3k/7pGBpYA "Bash – Try It Online")
[Answer]
Instead of clearing the pattern space with `s/.*//`, use the `z` command (lowercase) if you go with GNU sed. Besides the lower bytes count, it has the advantage that it won't start the next cycle as the command `d` does, which can be useful in certain situations.
[Answer]
Let's talk about the `t` and `T` commands, that although they are explained in the man page, it's easy to forget about it and introduce bugs accidently, especially when the code gets complicated.
**Man page statement for `t`:**
> 
> If a `s///` has done a successful substitution since the last input line was read and since the last t or T command, then branch to label.
> 
> 
> 
**Example showing what I mean:** Let's say you have a list of numbers and you want to count how many negatives there are. Partial code below:
```
1{x;s/.*/0/;x}                   # initialize the counter to 0 in hold space
s/-/&/                           # check if number is negative
t increment_counter              # if so, jump to 'increment_counter' code block
b                                # else, do nothing (start a next cycle)
:increment_counter
#function code here
```
Looks ok, but it's not. If the first number is positive, that code will still think it was negative, because the jump done via `t` for the first line of input is performed regardless, since there was a successful `s` substitution when we initialized the counter! Correct is: `/-/b increment_counter`.
If this seemed easy, you could still be fooled when doing multiple jumps back and forth to simulate functions. In our example the `increment_counter` block of code for sure would use a lot of `s` commands. Returning back with `b main` might cause another check in "main" to fall in the same trap. That is why I usually return from code blocks with `s/.*/&/;t label`. It's ugly, but useful.
[Answer]
Expanding upon this [tip answer](https://codegolf.stackexchange.com/a/51329/59010), regarding the conversions between decimal and plain unary number formats, I present the following alternative methods, with their advantages and disadvantages.
**Decimal to plain unary:** 102 + 1(r flag) = 103 bytes. I counted `\t` as a literal tab, as 1 byte.
```
h
:
s:\w::2g
y:9876543210:87654321\t :
/ /!s:$:@:
/\s/!t
x;s:-?.::;x
G;s:\s::g
/\w/{s:@:&&&&&&&&&&:g;t}
```
[**Try it online!**](https://tio.run/nexus/sed#@5/BZcVVbBVTbmVllM5VaWVpYW5mamJsZGhgBWPFlChYcekr6CsWW6lYOQCZMcX6iiVcFdbFVrr2elZW1hVc7kB2TLGVVTpQsly/uhioTA0OrNKtS2r//zc0@pdfUJKZn1f8X7cIAA)
Advantage: it is 22 bytes shorter and as extra, it works with negative integers as input
Disadvantage: it overwrites the hold space. However, since it's more likely that you'd need to convert the input integer right at the start of the program, this limitation is rarely felt.
**Plain unary to decimal:** 102 + 1(r flag) = 103 bytes
```
s:-?:&0:
/@/{:
s:\b9+:0&:
s:.9*@:/&:
h;s:.*/::
y:0123456789:1234567890:
x;s:/.*::
G;s:\n::
s:@::
/@/t}
```
[**Try it online!**](https://tio.run/nexus/sed#PYy7EkAwFET7fIgihhtvWYV0fkKlosGIgjF@XdwobHV25uw6i6hFoCDI0AVh0Q86hAo8xloaEOPYcJEEiBMqSbO8KKta4yeeH6xQLFnpmPoZ/sDg@91v54zPs6z7tMzWRdsL)
Advantage: it is 14 bytes shorter. This time both tip versions work for negative integers as input.
Disadvantage: it overwrites the hold space
For a complicated challenge, you'll have to adapt these snippets to work with other information that may exist in the pattern space or hold space, besides the number to convert. The code can be golfed more, if you know you only work with positive numbers or that zero alone is not going to be a valid input / output.
An example of such challenge answer, where I created and used these snippets, is the [Reciprocal of a number (1/x)](https://codegolf.stackexchange.com/a/115528/59010).
[Answer]
# Make use of sed's line-handling ability
With flexible challenge I/O, it can pay to have input/output separated by newlines instead of any other character by taking advantage of sed's commands for handling lines (like `D`, `N`, `n`, `G`, `H`, `P`, `s`'s `m` flag) instead of only being limited to `s` substitutions.
This can also open the opportunity for using `D` for looping instead of labels and goto, especially in sed versions that don't permit empty labels.
[Answer]
# The `L` command in old GNU sed versions
Used, for example, in the second solution in <https://codegolf.stackexchange.com/a/220633/>. Older versions of GNU sed like GNU sed 4.2.2 have the `L` command, which was later removed in newer versions. From [the archived docs](https://web.archive.org/web/20161224125356/https://www.gnu.org/software/sed/manual/sed.html#Extended-Commands),
> 
> `L` *n*
> 
> 
> This GNU `sed` extension fills and joins lines in pattern space to produce output lines of (at most) *n* characters, like `fmt` does; if *n* is omitted, the default as specified on the command line is used. This command is considered a failed experiment and unless there is enough request (which seems unlikely) will be removed in future versions.
> 
> 
> 
[Answer]
# `#n` at first to imply `-n`
> 
> The "#" and the remainder of the line are ignored (treated as a comment), with the single exception that if the first two characters in the file are #n, the default output is suppressed; this is the equivalent of specifying -n on the command line.
> 
> 
> 
[Source: sed (from SUSv2)](https://pubs.opengroup.org/onlinepubs/7908799/xcu/sed.html)
This is useful if you prefer NOT to output something by default.
...But is it really useful? `-n` adds either 1 or 2 bytes but `#n` and LF adds 3.
[Answer]
# `i`, `a`, and `c`
(The answer focuses on GNU sed because GNU sed's invocation of these commands is slightly shorter than that of POSIX, but otherwise I think the functionality should be the same).
These insert to, append to, and change the pattern space respectively. The lines they add are printed and so are not edited into the pattern space, meaning that the program will not be able to manipulate these lines. But they are shorter than `s`, if you want to insert some unchanging lines. Compare the following two lines:
```
s/^/text\n/ # 11 bytes
itext       #  5 bytes
```
Now I want to focus on `c`. `c text` replaces the pattern space with `text`. This text will be printed immediately, since the `c` makes sed move on to the next line. This means that the commands follows a call to `c` are effectively ignored. This behaviour of `c` can be useful in challenges where there are only few possible outputs, particularly when combined with the conditional `/.../`.
An example is the ['Hello, World!'](https://codegolf.stackexchange.com/q/55422) challenge, where `c` is shorter than `s`:
```
s/^/Hello, World!/ # 18 bytes
cHello, World!     # 14 bytes
```
Another example is [this challenge](https://codegolf.stackexchange.com/q/187816) ([sed answer](https://codegolf.stackexchange.com/a/190946)) to swap the strings `Good` and `Bad`. The outputs are restricted to being `Good` (when the input is `Bad`) or `Bad` (when the input is `Good`).
`s` gives 21 bytes:
```
s/Goo/Ba/;t;s/Ba/Goo/
```
Using `c` (in combination with `/.../`) gives 13 instead:
```
/B/cGood
cBad
```
If the input matches `/B/`, i.e. it the input is `Bad`, `Good` is printed and the program skips processing this input line. So the program only ever reaches the second line if the input doesn't match `B`, i.e. if the input is `Good`. Then in this case the output is set to `Bad`.
[Answer]
In sed, the closest thing to a function that you can have is a label. A function is useful because you can execute its code multiple times, thus saving a lot of bytes. In sed however you would need to specify the return label and as such you can't simply call this "function" multiple times throughout your code the way you would do it in other languages.
The workaround I use is to add in one of the two memories a flag, which is used to select the return label. This works best when the function code only needs a single memory space (the other one).
**Example showing what I mean:** taken from a project of mine to write a small game in sed
```
# after applying the player's move, I overwrite the pattern space with the flag "P"
s/.*/P/
b check_game_status
:continue_turn_from_player
#code
```
---
```
b calculate_bot_move
:return_bot_move
# here I call the same function 'check_game_status', but with a different flag: "B"
s/.*/B/
b check_game_status
:continue_turn_from_bot
#code (like say 'b update_screen')
```
---
```
:check_game_status   # this needs just the hold space to run
#code
/^P$/b continue_turn_from_player
/^B$/b continue_turn_from_bot
```
The labels should be golfed of course to just one letter, I used full names for a better explanation.
[Answer]
Mostly useless step:
```
y|A-y|B-z|
```
This will only translate `A` to `B` and `y` to `z` (... and `-` to `-` ;), but nothing else,
so
```
sed -e 'y|A-y|B-z|' <<<'Hello world!'
```
will just return:
```
Hello world!
```
You could ensure this will be useless, for sample by using this on lower-case hexadecimal values (containing only `0`, `1`, `2`, `3`, `4`, `5`, `6`, `7`, `8`, `9`, `a`, `b`, `c`, `d`, `e` or `f`.)
A little worst:
```
sed '; ;/s/b;y|A-y|B-z|;s ;s/ //; ; ;' <<<'Hello world'
Hello world
```
Why did this **not** suppress the space?
[Answer]
# Combine `s` substitutions
The `s` command takes many bytes (4 + 1 for the statement separator), so combining them can save bytes.
[An example](https://codegolf.stackexchange.com/a/221378): the following is at 17 bytes
```
s/\S+ //
s/\S+$//
```
while combining the two substitutions gives 15 bytes
```
s/^\S+ |\S+$//g
```
[Answer]
# Use back references
Most people know that you can mark some parts of your regex with `\(\)` and later refer to it as `\1` (or `\2` for the second and so on) in your substitution.
But you hardly ever see *back references*, using `\1` in the regex itself. For extended regular expressions (option `-E` to `sed`), this has been removed from the standard, but GNU `sed` supports it anyhow, so it works at tio.run, for example.
For example, this ERE `(.)\1` matches a double character or this one `(.).*\1` appearing twice in the pattern space. [See here](https://codegolf.stackexchange.com/a/266985/72726) for an example, how this makes things easier (and much shorter!).
Complicated task like [this one](https://codegolf.stackexchange.com/questions/266951/get-the-bible-verse/266969#266969) looking for matches in a file or [this one](https://codegolf.stackexchange.com/questions/266223/find-the-conflict-free-sitting-arrangement/266251#266251) or [this one](https://codegolf.stackexchange.com/questions/265307/play-a-game-of-memory/265363#265363) would almost be impossible without this feature.
# Math with back references
And it's great for teaching `sed` to count. I know you can solve some of those problems using `y`, but as soon as you have more than one thing to increment or decrement, this is the way to go. See examples [here](https://codegolf.stackexchange.com/questions/264903/order-by-earliest-lower-digit/265011#265011) or [here](https://codegolf.stackexchange.com/questions/38325/minimum-excluded-number/265157#265157) or [try the 151-byte decimal add online](https://tio.run/##XcyxDsIgFIXhnae4Boe2CT1woRbs7FsQpxrj0jYBX1@UbjY505ecPz1m9VzepSTcobRh64bL6IOEuBqR0DR9@1vftdEqRBM54J9ZVra6MkmQ0ZhyvUoNWbHvJAHbBMJpPiRpT9Ixed6TENmUYskJp4kFs/WaxuCt@6xbfq1LKur2BQ "sed – Try It Online") to examine how it works.
[Answer]
Okay, here's a weird functioning of (GNU) sed's regex. For example, if the pattern space is only `qqq`, then the extended regex `s/^|q|$/<&>/g` gives `<q><q><q>`. [Try it online!](https://tio.run/##K05N@f@/WD@uprBGRd9GzU4//f//wsLCf/kFJZn5ecX/dYsA "sed 4.2.2 – Try It Online")
I'm not sure, but I think it is because if a character matches, then do the empty strings surrounding it. So because the terminal `q` matches, the match also includes the end of the pattern space, so `$` by itself doesn't get matched because otherwise it would overlap with the previous match (and likewise for `^` and even I think for word boundaries and the like).
An example where this is useful is in the following (rather contrived) task:
* an input of `a` should give `ab`,
* and inputs matching `/a(c+)/` should give `ab\1b`, for example, an input of `accc` should give `abcccb`
Without the trick I can get 11 bytes (with the `-E` flag)
```
s/a|c+/&b/g
```
but using it gives 10 bytes
```
s/a|$/&b/g
```
[Try it online!](https://tio.run/##K05N@f@/WD@xRkVfLUk//f//RK7EZCACYwiR/C@/oCQzP6/4v24RAA "sed 4.2.2 – Try It Online")
(Do let me know if there is a better example that uses this behaviour).
]