Why does delayed expansion fail when inside a piped block of code?
I wasn't sure if I should edit my question, or post this as an answer.
I already vaguely knew that a pipe executes both the left and the right side each in its own CMD.EXE "session". But Aacini's and jeb's responses forced me to really think about and investigate what is happening with pipes. (Thank you jeb for demonstrating what is happening when piping into SET /P!)
I developed this investigative script - it helps explain a lot, but also demonstrates some bizarre and unexpected behavior. I'll post the script, followed by the output. Finally I will provide some analysis.
@echo off
cls
setlocal disableDelayedExpansion
set var1=value1
set "var2="
setlocal enableDelayedExpansion
echo on
@echo NO PIPE - delayed expansion is ON
echo 1: %var1%, %var2%, !var1!, !var2!
(echo 2: %var1%, %var2%, !var1!, !var2!)
@echo(
@echo PIPE LEFT SIDE - Delayed expansion is ON
echo 1L: %%var1%%, %%var2%%, !var1!, !var2! | more
(echo 2L: %%var1%%, %%var2%%, !var1!, !var2!) | more
(setlocal enableDelayedExpansion & echo 3L: %%var1%%, %%var2%%, !var1!, !var2!) | more
(cmd /v:on /c echo 4L: %%var1%%, %%var2%%, !var1!, !var2!) | more
cmd /v:on /c echo 5L: %%var1%%, %%var2%%, !var1!, !var2! | more
@endlocal
@echo(
@echo Delayed expansion is now OFF
(cmd /v:on /c echo 6L: %%var1%%, %%var2%%, !var1!, !var2!) | more
cmd /v:on /c echo 7L: %%var1%%, %%var2%%, !var1!, !var2! | more
@setlocal enableDelayedExpansion
@echo(
@echo PIPE RIGHT SIDE - delayed expansion is ON
echo junk | echo 1R: %%var1%%, %%var2%%, !var1!, !var2!
echo junk | (echo 2R: %%var1%%, %%var2%%, !var1!, !var2!)
echo junk | (setlocal enableDelayedExpansion & echo 3R: %%var1%%, %%var2%%, !var1!, !var2!)
echo junk | (cmd /v:on /c echo 4R: %%var1%%, %%var2%%, !var1!, !var2!)
echo junk | cmd /v:on /c echo 5R: %%var1%%, %%var2%%, !var1!, !var2!
@endlocal
@echo(
@echo Delayed expansion is now OFF
echo junk | (cmd /v:on /c echo 6R: %%var1%%, %%var2%%, !var1!, !var2!)
echo junk | cmd /v:on /c echo 7R: %%var1%%, %%var2%%, !var1!, !var2!
Here is the output
NO PIPE - delayed expansion is ON
C:\test>echo 1: value1, , !var1!, !var2!
1: value1, , value1,
C:\test>(echo 2: value1, , !var1!, !var2! )
2: value1, , value1,
PIPE LEFT SIDE - Delayed expansion is ON
C:\test>echo 1L: %var1%, %var2%, !var1!, !var2! | more
1L: value1, %var2%, value1,
C:\test>(echo 2L: %var1%, %var2%, !var1!, !var2! ) | more
2L: value1, %var2%, !var1!, !var2!
C:\test>(setlocal enableDelayedExpansion & echo 3L: %var1%, %var2%, !var1!, !var2! ) | more
3L: value1, %var2%, !var1!, !var2!
C:\test>(cmd /v:on /c echo 4L: %var1%, %var2%, !var1!, !var2! ) | more
4L: value1, %var2%, value1, !var2!
C:\test>cmd /v:on /c echo 5L: %var1%, %var2%, !var1!, !var2! | more
5L: value1, %var2%, value1,
Delayed expansion is now OFF
C:\test>(cmd /v:on /c echo 6L: %var1%, %var2%, !var1!, !var2! ) | more
6L: value1, %var2%, value1, !var2!
C:\test>cmd /v:on /c echo 7L: %var1%, %var2%, !var1!, !var2! | more
7L: value1, %var2%, value1, !var2!
PIPE RIGHT SIDE - delayed expansion is ON
C:\test>echo junk | echo 1R: %var1%, %var2%, !var1!, !var2!
1R: value1, %var2%, value1,
C:\test>echo junk | (echo 2R: %var1%, %var2%, !var1!, !var2! )
2R: value1, %var2%, !var1!, !var2!
C:\test>echo junk | (setlocal enableDelayedExpansion & echo 3R: %var1%, %var2%, !var1!, !var2! )
3R: value1, %var2%, !var1!, !var2!
C:\test>echo junk | (cmd /v:on /c echo 4R: %var1%, %var2%, !var1!, !var2! )
4R: value1, %var2%, value1, !var2!
C:\test>echo junk | cmd /v:on /c echo 5R: %var1%, %var2%, !var1!, !var2!
5R: value1, %var2%, value1,
Delayed expansion is now OFF
C:\test>echo junk | (cmd /v:on /c echo 6R: %var1%, %var2%, !var1!, !var2! )
6R: value1, %var2%, value1, !var2!
C:\test>echo junk | cmd /v:on /c echo 7R: %var1%, %var2%, !var1!, !var2!
7R: value1, %var2%, value1, !var2!
I tested both the left and right side of the pipe to demonstrate that processing is symmetric on both sides.
Tests 1 and 2 demonstrate that parentheses don't have any impact on delayed expansion under normal batch circumstances.
Tests 1L,1R: Delayed expansion works as expected. Var2 is undefined, so %var2% and !var2! output demonstrates that the commands are executed in a command line context, and not a batch context. In other words, command line parsing rules are used instead of batch parsing. (see How does the Windows Command Interpreter (CMD.EXE) parse scripts?) EDIT - !VAR2! is expanded in the parent batch context
Tests 2L,2R: The parentheses disable the delayed expansion! Very bizarre and unexpected in my mind. Edit - jeb considers this an MS bug or design flaw. I agree, there doesn't seem to be any rational reason for the inconsistent behavior
Tests 3L,3R: setlocal EnableDelayedExpansion does not work. But this is expected because we are in a command line context. setlocal only works in a batch context.
Tests 4L,4R: Delayed expansion is initially enabled, but parentheses disable it. CMD /V:ON re-enables delayed expansion and everything works as expected. We still have command line context and output is as expected.
Tests 5L,5R: Almost the same as 4L,4R except delayed expansion is already enabled when CMD /V:on is executed. %var2% gives expected command line context output. But !var2! output is blank which is expected in a batch context. This is another very bizarre and unexpected behavior. Edit - actually this makes sense now that I know !var2! is expanded in the parent batch context
Tests 6L,6R,7L,7R: These are analogous to tests 4L/R,5L/R except now delayed expansion starts out disabled. This time all 4 scenarios give the expected !var2! batch context output.
If someone can provide a logical explanation for results of 2L,2R and 5L,5R then I will select that as the answer to my original question. Otherwise I will probably accept this post as the answer (really more of an observation of what happens than an answer) Edit - jab nailed it!
Addendum: In response to jeb's comment - here is more evidence that piped commands within a batch execute in a command line context, not a batch context.
This batch script:
@echo on
call echo batch context %%%%
call echo cmd line context %%%% | more
gives this output:
C:\test>call echo batch context %%
batch context %
C:\test>call echo cmd line context %% | more
cmd line context %%
Final Addendum
I've added some additional tests and results that demonstrate all the findings so far. I also demonstrate that FOR variable expansion takes place before the pipe processing. Finally I show some interesting side effects of the pipe processing when a multi-line block is collapsed into a single line.
@echo off
cls
setlocal disableDelayedExpansion
set var1=value1
set "var2="
setlocal enableDelayedExpansion
echo on
@echo(
@echo Delayed expansion is ON
echo 1: %%, %%var1%%, %%var2%%, !var1!, ^^^!var1^^^!, !var2!, ^^^!var2^^^!, %%cmdcmdline%% | more
(echo 2: %%, %%var1%%, %%var2%%, !var1!, ^^^!var1^^^! !var2!, %%cmdcmdline%%) | more
for %%a in (Z) do (echo 3: %%a %%, %%var1%%, %%var2%%, !var1!, ^^^!var1^^^! !var2!, %%cmdcmdline%%) | more
(
echo 4: part1
set "var2=var2Value
set var2
echo "
set var2
)
(
echo 5: part1
set "var2=var2Value
set var2
echo "
set var2
echo --- begin cmdcmdline ---
echo %%cmdcmdline%%
echo --- end cmdcmdline ---
) | more
(
echo 6: part1
rem Only this line remarked
echo part2
)
(
echo 7: part1
rem This kills the entire block because the closing ) is remarked!
echo part2
) | more
Here is the output
Delayed expansion is ON
C:\test>echo 1: %, %var1%, %var2%, !var1!, ^!var1^!, !var2!, ^!var2^!, %cmdcmdline% | more
1: %, value1, %var2%, value1, !var1!, , !var2!, C:\Windows\system32\cmd.exe /S /D /c" echo 1: %, %var1%, %var2%, value1, !var1!, , !var2!, %cmdcmdline% "
C:\test>(echo 2: %, %var1%, %var2%, !var1!, ^!var1^! !var2!, %cmdcmdline% ) | more
2: %, value1, %var2%, !var1!, !var1! !var2!, C:\Windows\system32\cmd.exe /S /D /c" ( echo 2: %, %var1%, %var2%, !var1!, ^!var1^! !var2!, %cmdcmdline% )"
C:\test>for %a in (Z) do (echo 3: %a %, %var1%, %var2%, !var1!, ^!var1^! !var2!, %cmdcmdline% ) | more
C:\test>(echo 3: Z %, %var1%, %var2%, !var1!, ^!var1^! !var2!, %cmdcmdline% ) | more
3: Z %, value1, %var2%, !var1!, !var1! !var2!, C:\Windows\system32\cmd.exe /S /D /c" ( echo 3: Z %, %var1%, %var2%, !var1!, ^!var1^! !var2!, %cmdcmdline% )"
C:\test>(
echo 4: part1
set "var2=var2Value
set var2
echo "
set var2
)
4: part1
var2=var2Value
"
var2=var2Value
C:\test>(
echo 5: part1
set "var2=var2Value
set var2
echo "
set var2
echo --- begin cmdcmdline ---
echo %cmdcmdline%
echo --- end cmdcmdline ---
) | more
5: part1
var2=var2Value & set var2 & echo
--- begin cmdcmdline ---
C:\Windows\system32\cmd.exe /S /D /c" ( echo 5: part1 & set "var2=var2Value
var2=var2Value & set var2 & echo
" & set var2 & echo --- begin cmdcmdline --- & echo %cmdcmdline% & echo --- end cmdcmdline --- )"
--- end cmdcmdline ---
C:\test>(
echo 6: part1
rem Only this line remarked
echo part2
)
6: part1
part2
C:\test>(echo %cmdcmdline% & (
echo 7: part1
rem This kills the entire block because the closing ) is remarked!
echo part2
) ) | more
Tests 1: and 2: summarize all the behaviors, and the %%cmdcmdline%% trick really helps to demonstrate what is taking place.
Test 3: demonstrates that FOR variable expansion still works with a piped block.
Tests 4:/5: and 6:/7: show interesting side effects of the way pipes work with multi-line blocks. Beware!
I've got to believe figuring out escape sequences within complex pipe scenarios will be a nightmare.
As Aacini shows, it seems that many things fail within a pipe.
echo hello | set /p var=
echo here | call :function
But in reality it's only a problem to understand how the pipe works.
Each side of a pipe starts its own cmd.exe in its own ascynchronous thread.
That is the cause why so many things seem to be broken.
But with this knowledge you can avoid this and create new effects
echo one | ( set /p varX= & set varX )
set var1=var2
set var2=content of two
echo one | ( echo %%%var1%%% )
echo three | echo MYCMDLINE %%cmdcmdline%%
echo four | (cmd /v:on /c echo 4: !var2!)
Update 2019-08-15:
As discovered at Why does `findstr` with variable expansion in its search string return unexpected results when involved in a pipe?, cmd.exe is only used if the command is internal to cmd.exe, if the command is a batch file, or if the command is enclosed in a parenthesized block. External commands not enclosed within parentheses are launched in a new process without the aid of cmd.exe.
EDIT: In depth analysis
As dbenham shows, both sides of the pipes are equivalent for the expansion phases.
The main rules seems to be:
The normal batch parser phases are done
.. percent expansion
.. special character phase/block begin detection
.. delayed expansion (but only if delayed expansion is enabled AND it isn't a command block)
Start the cmd.exe with C:\Windows\system32\cmd.exe /S /D /c"<BATCH COMMAND>"
These expansions follows the rules of the cmd-line parser not the the batch-line parser.
.. percent expansion
.. delayed expansion (but only if delayed expansion is enabled)
The <BATCH COMMAND> will be modified if it's inside a parenthesis block.
(
echo one %%cmdcmdline%%
echo two
) | more
Called as C:\Windows\system32\cmd.exe /S /D /c" ( echo one %cmdcmdline% & echo two )", all newlines are changed to & operator.
Why the delayed expansion phase is affected by parenthesis?
I suppose, it can't expand in the batch-parser-phase, as a block can consist of many commands and the delayed expansion take effect when a line is executed.
(
set var=one
echo !var!
set var=two
) | more
Obviously the !var! can't be evaluated in the batch context, as the lines are executed only in the cmd-line context.
But why it can be evaluated in this case in the batch context?
echo !var! | more
In my opionion this is a "bug" or inconsitent behaviour, but it's not the first one
EDIT: Adding the LF trick
As dbenham shows, there seems to be some limitation through the cmd-behaviour that changes all line feeds into &.
(
echo 7: part1
rem This kills the entire block because the closing ) is remarked!
echo part2
) | more
This results intoC:\Windows\system32\cmd.exe /S /D /c" ( echo 7: part1 & rem This ...& echo part2 ) "
The rem will remark the complete line tail, so even the closing bracket is missing then.
But you can solve this with embedding your own line feeds!
set LF=^
REM The two empty lines above are required
(
echo 8: part1
rem This works as it splits the commands %%LF%% echo part2
) | more
This results to C:\Windows\system32\cmd.exe /S /D /c" ( echo 8: part1 %cmdcmdline% & rem This works as it splits the commands %LF% echo part2 )"
And as the %lf% is expanded while parsing the parenthises by the parser, the resulting code looks like
( echo 8: part1 & rem This works as it splits the commands
echo part2 )
This %LF% behaviour works always inside of parenthesis, also in a batch file.
But not on "normal" lines, there a single <linefeed> will stop the parsing for this line.
EDIT: Asynchronously is not the full truth
I said that the both threads are asynchronous, normally this is true.
But in reality the left thread can lock itself when the piped data isn't consumed by the right thread.
There seems to be a limit of ~1000 characters in the "pipe" buffer, then the thread is blocked until the data is consumed.
@echo off
(
(
for /L %%a in ( 1,1,60 ) DO (
echo A long text can lock this thread
echo Thread1 ##### %%a > con
)
)
echo Thread1 ##### end > con
) | (
for /L %%n in ( 1,1,6) DO @(
ping -n 2 localhost > nul
echo Thread2 ..... %%n
set /p x=
)
)