==> /tmp/build/80754af9/pytorch_1625259906368/_build_env/bin/git log -n1 <==
commit 56b43f4fec1f76953f15a627694d4bba34588969
Author: Pritam Damania <9958665+pritamdamania87@users.noreply.github.com>
Date:   Tue Mar 23 19:28:21 2021 -0700

    Perform appropriate CUDA stream synchronization in distributed autograd. (#53929) (#54358)
    
    Summary:
    Pull Request resolved: https://github.com/pytorch/pytorch/pull/53929
    
    The local autograd engine performs appropriate stream synchronization
    between autograd nodes in the graph to ensure a consumer's stream is
    synchronized with the producer's stream before executing the consumer.
    
    However in case of distributed autograd, the SendRpcBackward function receives
    gradients over the wire and TensorPipe uses its own pool of streams for this
    purpose. As a result, the tensors are received on TensorPipe's stream pool but
    SendRpcBackward runs on a different stream during the backward pass and there
    is no logic to synchronize these streams.
    
    To fix this, I've enhanced DistEngine to synchronize these streams
    appropriately when it receives grads over the wire.
    ghstack-source-id: 124055277
    
    (Note: this ignores all push blocking failures!)
    
    Test Plan:
    1) Added unit test which reproduced the issue.
    2) waitforbuildbot.
    
    Reviewed By: walterddr, wanchaol
    
    Differential Revision: D27025307
    
    fbshipit-source-id: 2944854e688e001cb3989d2741727b30d9278414
    
    Co-authored-by: Pritam Damania <pritam.damania@fb.com>
==> /tmp/build/80754af9/pytorch_1625259906368/_build_env/bin/git describe --tags --dirty <==
v1.8.1-dirty
==> /tmp/build/80754af9/pytorch_1625259906368/_build_env/bin/git status <==
HEAD detached at v1.8.1
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   CMakeLists.txt
	modified:   cmake/Dependencies.cmake
	modified:   torch/csrc/DataLoader.cpp
	modified:   torch/utils/cpp_extension.py

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	conda_build.sh
	metadata_conda_debug.yaml

no changes added to commit (use "git add" and/or "git commit -a")
