Hi everyone !
I’m in need for some assistance for string manipulation with sed and regex. I tried a whole day to trial & error and look around the web to find a solution however it’s way over my capabilities and maybe here are some sed/regex gurus who are willing to give me a helping hand !
With everything I gathered around the web, It seems it’s rather a complicated regex and sed substitution, here we go !
What Am I trying to achieve?
I have a lot of markdown guides I want to host on a self-hosted forgejo based git markdown. However the classic markdown links are not the same as one github/forgejo…
Convert the following string:
[Some text](#Header%20Linking%20MARKDOWN.md)
Into
[Some text](#header-linking-markdown.md)
As you can see those are the following requirement:
- Pattern:
[
]( - Only edit what’s between parentheses
- Replace
space (%20)
with-
- Everything as lowercase
- Links are sometimes in nested parentheses
- e.g. (look here
[
) ](
- e.g. (look here
- Do not change a line that begins with
https
(external links)
While everything is probably a bit complex as a whole the trickiest part is probably the nested parentheses :/
What I tried
The furthest I got was the following:
sed -Ei 's|\(([^\)]+)\)|\L&|g' test3.md #make everything between parentheses lowercase
sed -i '/https/ ! s/%20/-/g' test3.md #change every %20 occurrence to -
These sed/regx substitution are what I put together while roaming the web, but it has a lot a flaws and doesn’t work with nested parentheses. Also this would change every %20
occurrence in the file.
The closest solution I found on stackoverflow looks similar but wasn’t able to fit to my needs. Actually my lack of regex/sed understanding makes it impossible to adapt to my requirements.
I would appreciate any help even if a change of tool is needed, however I’m more into a learning processes, so a script or CLI alternative is very appreciated :) actually any help is appreciated :D !
Thanks in advance.
Don’t worry or apologize about your English. I’m having no trouble understanding. :)
I’m going to take the second part first and come back with another comment to address the
%20
andhttps
bits.So these variations, like
[1.3 Subtitles](BDMV_svt-av1_encode_anime.md#1.3%20Subtitles)
, are where you would start to craft a new expression. Trying to catch every variation in a single expression would get to complicated and more likely to fail and/or modify text you don’t want modified.So in this case, here’s the expression I’d use:
sed -ri 's|(]\(.+[0-9]+)\.([0-9]+.+\))|\1-\2|' somefile
And the breakdown:
sed -ri
calls sed with the expanded regular expressions capabilities and to edit the file in place's|
- Begins the pattern match|modify expression(
- This very first opening parentheses is a special metacharacter that is used to group a sub-expression within the larger expression. By doing this we can create variables that we can refer to in the modification portion of the command.]\(
- Find the closing bracket character and an opening parentheses character, which we know will be the beginning of a markdown url. The backslash precedes the open parentheses to escape it and indicate it needs to look for the actual open parentheses character.+
- Find any character (indicated by the.
) one or more times (indicated by the+
). This will find any characters until it gets to the next specified character in the expression[0-9]+
- This is two parts. The first part is[0-9]
. The brackets are metacharacters in regex that enclose a character set to match from. In this case the character set is the numbers zero to nine. What this means on its own is thatsed
will look for one occurrence of any number between zero and nine. The+
tellssed
to find one or more occurrences of a number between one and nine until it gets to the next portion of the pattern. I did this because I don’t know the upper bounds of the documentation numeration you’re working with in the links. If all the links only contain single digit numbers before the decimal, you can remove the+
.)
- This closing parentheses marks the end of the subexpression that we want to refer to. In this case, the sub expression is capturing from the closing bracket up to (but not including) the decimal in the number.\.
- This tellssed
to find the period/dot/decimal character in the number. It’s preceded by the backslash because the period/dot/decimal character is a metacharacter in regular expressions.(
- This is the beginning of a new subexpression[0-9]+
- The numeral capture repeats to find the number after the period/dot/decimal. Similarly to the number before the decimal, if the number after the decimal is only ever single digit, the+
can be removed..+
- Find any character (indicated by the.
) one or more times (indicated by the+
). This will find any characters until it gets to the next specified character in the expression, taking us to the end of the url\)
- Find the closing parentheses of the url. The backslash precedes the closing parentheses to escape it and indicate it needs to look for the actual open parentheses character.)
- This closes our second subexpression, which captures everything from the number after the decimal to the closing parentheses of the link.|
- Indicates the end of the pattern matching portion of the expression/command. and the beginning of the modification part of the command/expression.\1
- This is how we refer to or call the subexpressions. The syntax is a backslash followed by a number, and the number indicates the sequential position of the subexpression. So\1
refers to this portion of the regex in the command above:(]\(.+[0-9]+)
. This section of the expression is capturing everything from the closing bracket up to (but not including) the period/dot/decimal character. By using it in this position in the substitution/modification, we’re just using it as a variable, so in the substitution, it’s going to put everything it finds in the first subexpression first in the new/modified string of text.-
- This tellssed
to put a dash immediately after the first subexpression in this new/modified string of text, effectively replacing the period/dot/decimal in the number portion of the url.\2
- This is calling the second subexpression, which is this portion of the pattern matching regex:[0-9]+.+\)
. This captures everything in the url from the number after the period/dot/decimal (not including the decimal), to the closing parthenses of the markdown url. Used in this position of the substitution it tells sed to place it after the dash in the new/modified text.|'
- This indicates the end of the modification portion of the command and closes the match|substitution expression.somefile
- The file to be worked onHere is the full command again:
sed -ri 's|(]\(.+[0-9]+)\.([0-9]+.+\))|\1-\2|' somefile
Altogether what this does is: Begin the first subexpression that starts with finding a closing bracket followed by an opening parentheses followed by any character one or more times until finding at least one or more numbers between zero and nine until it finds a decimal, and then close and remember what was found for this sub expression (not including the decimal). Then begin the second subexpression that starts with finding a number between zero and nine one or more times, and then find any character any number of times until a closing parentheses is found. Then close and remember what was found in this subexpression. Replace everything with subexpression one followed by a dash followed by subexpression two.
If you also need this markdown link text to be converted to lowercase, just add
\L
to the replacement section before the\1
like so:sed -ri 's|(]\(.+[0-9]+)\.([0-9]+.+\))|\L\1-\2|' somefile