mevatron mevatron - 8 days ago 7
Git Question

Docker container unable to clone from github.com

I'm currently attempting to run docker containers on a Raspberry Pi 2 using the HypriotOS version 0.6.1. Unfortunately, on my home network I am unable to clone from GitHub.

root@aaeac2ab909d:/home/meteor# GIT_CURL_VERBOSE=1 git clone --depth 1 https://github.com/4commerce-technologies-AG/meteor.git
Cloning into 'meteor'...
* Couldn't find host github.com in the .netrc file; using defaults
* Hostname was NOT found in DNS cache
* Trying 192.30.252.129...
* Connected to github.com (192.30.252.129) port 443 (#0)
* found 173 certificates in /etc/ssl/certs/ca-certificates.crt
* server certificate verification OK
* common name: github.com (matched)
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: RSA
* certificate version: #3
* subject:
* start date: Tue, 08 Apr 2014 00:00:00 GMT

* expire date: Tue, 12 Apr 2016 12:00:00 GMT

* issuer: C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert SHA2 Extended Validation Server CA
* compression: NULL
* cipher: AES-128-GCM
* MAC: AEAD
> GET /4commerce-technologies-AG/meteor.git/info/refs?service=git-upload-pack HTTP/1.1
User-Agent: git/2.1.4
Host: github.com
Accept: */*
Accept-Encoding: gzip
Pragma: no-cache

< HTTP/1.1 200 OK
* Server GitHub Babel 2.0 is not blacklisted
< Server: GitHub Babel 2.0
< Content-Type: application/x-git-upload-pack-advertisement
< Transfer-Encoding: chunked
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Vary: Accept-Encoding
< X-GitHub-Request-Id: 4B8A08F2:2C58:14CD5B7:56972928
<
* Connection #0 to host github.com left intact
* Couldn't find host github.com in the .netrc file; using defaults
* Found bundle for host github.com: 0x83cdf8
* Re-using existing connection! (#0) with host github.com
* Connected to github.com (192.30.252.129) port 443 (#0)
> POST /4commerce-technologies-AG/meteor.git/git-upload-pack HTTP/1.1
User-Agent: git/2.1.4
Host: github.com
Accept-Encoding: gzip
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result
Content-Length: 205

* upload completely sent off: 205 out of 205 bytes
< HTTP/1.1 200 OK
* Server GitHub Babel 2.0 is not blacklisted
< Server: GitHub Babel 2.0
< Content-Type: application/x-git-upload-pack-result
< Transfer-Encoding: chunked
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Vary: Accept-Encoding
< X-GitHub-Request-Id: 4B8A08F2:2C58:14CD5EE:56972928
<
* Connection #0 to host github.com left intact
* Couldn't find host github.com in the .netrc file; using defaults
* Found bundle for host github.com: 0x83cdf8
* Re-using existing connection! (#0) with host github.com
* Connected to github.com (192.30.252.129) port 443 (#0)
> POST /4commerce-technologies-AG/meteor.git/git-upload-pack HTTP/1.1
User-Agent: git/2.1.4
Host: github.com
Accept-Encoding: gzip
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result
Content-Length: 214

* upload completely sent off: 214 out of 214 bytes
< HTTP/1.1 200 OK
* Server GitHub Babel 2.0 is not blacklisted
< Server: GitHub Babel 2.0
< Content-Type: application/x-git-upload-pack-result
< Transfer-Encoding: chunked
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Vary: Accept-Encoding
< X-GitHub-Request-Id: 4B8A08F2:2C58:14CD635:56972929
<
remote: Counting objects: 2610, done.
remote: Compressing objects: 100% (2235/2235), done.
* GnuTLS recv error (-54): Error in the pull function.B/s
* Closing connection 0
error: RPC failed; result=56, HTTP code = 200| 2.90 MiB/s
fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed


This error occurs every time like clock-work. I took the system to a new network, and it worked just fine, which indicates something is wrong with my home network.

I suspect this has something to do with my routers local DNS settings using DNSMasq, but I'm not positive. Below are my DNSMasq options on my router:

Router DNSMasq settings



dnsmasq-settings

Host /etc/resolv.conf



nameserver 127.0.1.1
search home.lan


Container /etc/resolv.conf



domain home.lan
search home.lan
nameserver 192.168.11.1


Interestingly, tonight I decided to try running my container with the
--net=host
flag wondering if it had something to do with docker's bridge and my network settings. This apparently also works flawlessly!

Successful container git clone



root@black-pearl:/home/meteor# GIT_CURL_VERBOSE=1 git clone --depth 1 https://github.com/4commerce-technologies-AG/meteor.git
Cloning into 'meteor'...
* Couldn't find host github.com in the .netrc file; using defaults
* Hostname was NOT found in DNS cache
* Trying 192.30.252.128...
* Connected to github.com (192.30.252.128) port 443 (#0)
* found 173 certificates in /etc/ssl/certs/ca-certificates.crt
* server certificate verification OK
* common name: github.com (matched)
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: RSA
* certificate version: #3
* subject:
* start date: Tue, 08 Apr 2014 00:00:00 GMT

* expire date: Tue, 12 Apr 2016 12:00:00 GMT

* issuer: C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert SHA2 Extended Validation Server CA
* compression: NULL
* cipher: AES-128-GCM
* MAC: AEAD
> GET /4commerce-technologies-AG/meteor.git/info/refs?service=git-upload-pack HTTP/1.1
User-Agent: git/2.1.4
Host: github.com
Accept: */*
Accept-Encoding: gzip
Pragma: no-cache

< HTTP/1.1 200 OK
* Server GitHub Babel 2.0 is not blacklisted
< Server: GitHub Babel 2.0
< Content-Type: application/x-git-upload-pack-advertisement
< Transfer-Encoding: chunked
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Vary: Accept-Encoding
< X-GitHub-Request-Id: 4B8A08F2:59C7:1E862A2:56972ABF
<
* Connection #0 to host github.com left intact
* Couldn't find host github.com in the .netrc file; using defaults
* Found bundle for host github.com: 0xb83df8
* Re-using existing connection! (#0) with host github.com
* Connected to github.com (192.30.252.128) port 443 (#0)
> POST /4commerce-technologies-AG/meteor.git/git-upload-pack HTTP/1.1
User-Agent: git/2.1.4
Host: github.com
Accept-Encoding: gzip
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result
Content-Length: 205

* upload completely sent off: 205 out of 205 bytes
< HTTP/1.1 200 OK
* Server GitHub Babel 2.0 is not blacklisted
< Server: GitHub Babel 2.0
< Content-Type: application/x-git-upload-pack-result
< Transfer-Encoding: chunked
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Vary: Accept-Encoding
< X-GitHub-Request-Id: 4B8A08F2:59C7:1E86326:56972ABF
<
* Connection #0 to host github.com left intact
* Couldn't find host github.com in the .netrc file; using defaults
* Found bundle for host github.com: 0xb83df8
* Re-using existing connection! (#0) with host github.com
* Connected to github.com (192.30.252.128) port 443 (#0)
> POST /4commerce-technologies-AG/meteor.git/git-upload-pack HTTP/1.1
User-Agent: git/2.1.4
Host: github.com
Accept-Encoding: gzip
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result
Content-Length: 214

* upload completely sent off: 214 out of 214 bytes
< HTTP/1.1 200 OK
* Server GitHub Babel 2.0 is not blacklisted
< Server: GitHub Babel 2.0
< Content-Type: application/x-git-upload-pack-result
< Transfer-Encoding: chunked
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Vary: Accept-Encoding
< X-GitHub-Request-Id: 4B8A08F2:59C7:1E863C3:56972AC0
<
remote: Counting objects: 2610, done.
remote: Compressing objects: 100% (2235/2235), done.
* Connection #0 to host github.com left intact | 2.47 MiB/s
remote: Total 2610 (delta 245), reused 1412 (delta 149), pack-reused 0
Receiving objects: 100% (2610/2610), 14.26 MiB | 2.34 MiB/s, done.
Resolving deltas: 100% (245/245), done.
Checking connectivity... done.
Checking out files: 100% (2413/2413), done.


I've tried disabling DNSMasq on the router, but that didn't seem to help, so I was wondering if anyone had any ideas on something to try to get docker's native bridge mode working?

Answer

I forgot that I asked this question here. But, for those of you who also encounter this issue.

The solution was discovered by @aaronlehmann on GitHub Issue #1090. This issue was (possibly still is) affecting AWS instances, but also appears to affect some residential connections as well.

There are two possible fixes:

  • Turn on conntrack's "be liberal" flag: echo 1 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal. This causes conntrack/NAT to treat packets outside the TCP window as part of the flow being tracked, instead of marking them invalid and causing them to be handled by the host.
  • Add a rule to drop invalid packets instead of allowing them to trigger RSTs: iptables -I INPUT -m conntrack --ctstate INVALID -j DROP

For me, the ip_conntrack_tcp_be_liberal would never hold its setting after reboot. This required manually running the above command each time I wanted to use docker, so the best solution for me was saving

iptables -I INPUT -m conntrack --ctstate INVALID -j DROP

into my iptables.

All problems then disappeared!