1

What I want

My actual long term goal is bypassing all the output of a git process into another function (in order to make a progress bar - see my older question for more details about that).

What I've got so far

I thought my problem was solved after this question but it comes out now that only partially.

I have already figured out that git

  • produces output only to stderr by design
  • I had to use the --progress option of git in order to force git to really print out the progress.

Now if I run

                                  #or also &> output.file
git clone --progress http://someRepo > output.file 2>&1

I can see the full progress e.g. by running at the same time in a second terminal

tail -f output.file

I get

Cloning into 'someRepo' ...
remote: Counting objects: 2618, done.
remote: Compressing objects: 100% (14/14), done.
Receiving objects:   4% (106/2618), 42.11 MiB | 5.67 MiB/s

which gets updated in realtime until finishing

Cloning into 'someRepo' ...
remote: Counting objects: 2618, done.
remote: Compressing objects: 100% (14/14), done.
remote: Total 2618 (delta 2), reused 12 (delta 1), pack-reused 2603
Receiving objects: 100% (2618/2618), 258.95 MiB | 5.71 MiB/s, Done.
Resolving differences: 100% (1058/1058), Done.
Checking connectivity ... Done.

So far so well.

My Problem

Now I do not want to put this output into a file anymore but rather bypass it into a function. So I tried

                                  #or also |&
git clone --progress http://someRepo 2>&1 | {
    while read -r line
    do
        # echo is for testing only
        # later I want to pass this to another function
        # EDIT: I added bypassed in order to see what is bypassed
        echo "bypassed: ${line}"
    done
}

But running it like this I only get as output displayed (EDIT: I added the bypassed: to the echo line in order to see how it gets passed.)

bypassed: Cloning into 'someRepo'
bypassed: remote: Counting objects: 2618, done.
remote: Compressing objects: 100% (14/14), done.

and only at the very end when the download finishes I get the rest all at once jumping to this output (EDIT: I added the bypassed: to the echo line in order to see how it gets passed.)

bypassed: Cloning into 'someRepo' ...
bypassed: remote: Counting objects: 2618, done.
remote: Compressing objects: 100% (14/14), done.
remote: Total 2618 (delta 2), reused 12 (delta 1), pack-reused 2603
Receiving objects: 100% (2618/2618), 258.95 MiB | 5.71 MiB/s, Done.
Resolving differences: 100% (1058/1058), Done.
bypassed: Checking connectivity ... Done.

So apparently the 5 lines of the progress block are all passed in at once .. maybe this is a problem for the pipe and the while loop?

So what am I missing? How to I have to run the command in order to get the output also "echoed" / bypassed into my function later in realtime just the same way the tail -f output.file gets updated in realtime?

EDIT

I also already tried stdbuf and even unbuffer as suggested here and here

stdbuf -i0 -o0 -e0 git clone --progress http://someRepo |& ...

stdbuf -oL git clone --progress http://someRepo |& ...


unbuffer git clone --progress http://someRepo |& ...

but this didn't change the output.

Community
  • 1
derHugo
  • 3,376
  • 5
  • 34
  • 52

2 Answers2

4

This is a very interesting question and I had to experiment for a while, but the answer is actually super simple!

How does git's progress output work? It shows a single status line with a progress percentage and updates it all the time until it finished. It does that by not printing a line feed \n but a carriage return \r character at the end of the progress line, which causes the output cursor to jump back to the beginning of the line again, ready to overwrite the last written line again with an updated value.

You can observe this by piping git's output through cat -A, which does not interpret invisible characters but shows them, so that e.g. \r becomes ^M and line breaks additionally get denoted with a $:

Cloning into 'MyRepository'...$
remote: Counting objects: 2317, done.        $
Receiving objects:   0% (1/2317)   ^MReceiving objects:   1% (24/2317)   ^MReceiving objects:   2% (47/2317)   ^MReceiving objects:   3% (70/2317)   ^MReceiving objects:   4% (93/2317)   ^MReceiving objects:   5% (116/2317)   
[...]

Now why does this affect read? It's obvious, as help read says (extract, emphasis mine):

Read a line from the standard input and split it into fields.

read waits for line breaks (\n) before it finishes. The output of git did not contain any line breaks during the progress display, hence read did not finish. Only after the progress display was completed and git really printed the next line, read took in all the output, including all intermediate states and carriage returns and you echoed it inside your loop. If you pipe the echo or done through cat -A as well, you see the same output full of ^Ms, like above.

What you can do to allow read to capture all the intermediate progress lines is to translate all carriage returns into real line breaks, by piping the stuff through tr \\r \\n. That way each state gets printed on a new line instead of overwriting the previous line.

So, the complete loop you used could look like this:

git clone --progress http://someRepo 2>&1 | tr \\r \\n |
    while read -r line ; do
        echo "bypassed: ${line}"
    done

On my system, the output shown in terminal by this solution is still not totally fluent and stutters a bit, but I was not able to improve that any further. At least you now have the real "progress" output.

Byte Commander
  • 110,243
0

Here is a complete solution, not very nice, but working well !

total=0
{
git clone --progress http:yourRepo 2>&1 | tr \\r \\n |
    while read -r line ; do
        cur=`grep -oP '\d+(?=%)' <<< ${line}`
        total=$((total+cur))
        percent=$(bc <<< "scale=2;100*($total/11767)")
        echo "$percent/1" | bc
    done
} | whiptail --title "Cloning" --gauge "Cloning Git Repository" 8 78 0
clo
  • 1