Installing using HTTP proxy fails utterly

[rcarmo]

I've been trying to install Cascalog from behind a proxy using leiningen, and both 1.x and 2.x-SNAPSHOT fail with variations of the following output:

$ lein deps
Could not transfer artifact cascading:cascading-hadoop:pom:2.1.6 from/to conjars (http://conjars.org/repo/): Connection to http://conjars.org refused
Could not transfer artifact cascading.kryo:cascading.kryo:pom:0.4.6 from/to conjars (http://conjars.org/repo/): Connection to http://conjars.org refused
Could not transfer artifact com.twitter:maple:pom:0.2.2 from/to conjars (http://conjars.org/repo/): Connection to http://conjars.org refused
This could be due to a typo in :dependencies or network issues.
If you are behind a proxy, try setting the 'http_proxy' environment variable.

These are the only dependencies that fail. Everything else is downloaded just fine, and repeating this exercise on a machine outside the proxy works. But being unable to install things in a closed network (where the proxy works perfectly fine otherwise) is preventing me from adopting Cascalog.

Things I've tried so far:

• Setting http_proxy globally (it's our standard config), per-user and on the command line

• Setting :jvm-opts in project.clj and ~/.lein/profiles.clj

• Doing the same using :java-opts

• Running leiningen manually and setting everything on the command line:

  http_proxy=http://proxy:3128 java -Dhttp.proxyHost=proxy -Dhttp.proxyPort=3128 -client -Xbootclasspath/a:/home/user/.lein/self-installs/leiningen-2.3.3-standalone.jar -XX:+TieredCompilation -XX:TieredStopAtLevel=1 -Dfile.encoding=UTF-8 -Dmaven.wagon.http.ssl.easy=false -Dleiningen.original.pwd=/home/user/cascalog-test -Dleiningen.script=/usr/local/bin/lein -classpath :/home/user/.lein/self-installs/leiningen-2.3.3-standalone.jar clojure.main -m leiningen.core.main deps
  

None of these worked, and all fail with the traceback below, but only for some dependencies in conjars. I am quickly on the verge of concluding that org.apache.http.impl.client.AbstractHttpClient does not honor any proxy settings at all, but am at a loss as to why it's being used for these dependencies alone and everything else loads fine - including cascalog-core, which is also at conjars.

Examining the .pom files also failed to yield any insights.

Caused by: org.sonatype.aether.resolution.ArtifactDescriptorException: Failed to read artifact descriptor for cascading:cascading-hadoop:jar:2.1.6
        at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:296)
        at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor(DefaultArtifactDescriptorReader.java:186)
        at org.sonatype.aether.impl.internal.DefaultDependencyCollector.process(DefaultDependencyCollector.java:412)
        at org.sonatype.aether.impl.internal.DefaultDependencyCollector.process(DefaultDependencyCollector.java:544)
        at org.sonatype.aether.impl.internal.DefaultDependencyCollector.collectDependencies(DefaultDependencyCollector.java:240)
        ... 43 more
Caused by: org.sonatype.aether.resolution.ArtifactResolutionException: Could not transfer artifact cascading:cascading-hadoop:pom:2.1.6 from/to conjars (http://conjars.org/repo/): Connection to http://conjars.org refused
        at org.sonatype.aether.impl.internal.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:538)
        at org.sonatype.aether.impl.internal.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:216)
        at org.sonatype.aether.impl.internal.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:193)
        at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:281)
        ... 47 more
Caused by: org.sonatype.aether.transfer.ArtifactTransferException: Could not transfer artifact cascading:cascading-hadoop:pom:2.1.6 from/to conjars (http://conjars.org/repo/): Connection to http://conjars.org refused
        at org.sonatype.aether.connector.wagon.WagonRepositoryConnector$4.wrap(WagonRepositoryConnector.java:951)
        at org.sonatype.aether.connector.wagon.WagonRepositoryConnector$4.wrap(WagonRepositoryConnector.java:941)
        at org.sonatype.aether.connector.wagon.WagonRepositoryConnector$GetTask.run(WagonRepositoryConnector.java:669)
        at org.sonatype.aether.util.concurrency.RunnableErrorForwarder$1.run(RunnableErrorForwarder.java:60)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        ... 1 more
Caused by: org.apache.maven.wagon.TransferFailedException: Connection to http://conjars.org refused
        at org.apache.maven.wagon.shared.http4.AbstractHttpClientWagon.fillInputData(AbstractHttpClientWagon.java:892)
        at org.apache.maven.wagon.StreamWagon.getInputStream(StreamWagon.java:116)
        at org.apache.maven.wagon.StreamWagon.getIfNewer(StreamWagon.java:88)
        at org.apache.maven.wagon.StreamWagon.get(StreamWagon.java:61)
        at org.sonatype.aether.connector.wagon.WagonRepositoryConnector$GetTask.run(WagonRepositoryConnector.java:601)
        ... 4 more
Caused by: org.apache.http.conn.HttpHostConnectException: Connection to http://conjars.org refused
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190)
        at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
        at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:640)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
        at org.apache.maven.wagon.shared.http4.AbstractHttpClientWagon.execute(AbstractHttpClientWagon.java:746)
        at org.apache.maven.wagon.shared.http4.AbstractHttpClientWagon.fillInputData(AbstractHttpClientWagon.java:886)
        ... 8 more
Caused by: java.net.ConnectException: Connection timed out
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
        at java.net.Socket.connect(Socket.java:529)
        at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
        ... 15 more

[sw1nn]

Hi,

• Setting http_proxy globally (it's our standard config), per-user and
  on the command line

Looking here:

https://github.com/technomancy/leiningen/blob/95bdc72f805d59494ec58e7a63012fe15971d038/leiningen-core/src/leiningen/core/classpath.clj#L99

&

https://github.com/technomancy/leiningen/blob/95bdc72f805d59494ec58e7a63012fe15971d038/leiningen-core/src/leiningen/core/utils.clj#L18

It looks like the environment variable should be set to:

http_proxy=proxy:3128

(ie. no http://)

failing that you could either download manually and install in
${HOME}/.m2/repository

or create a maven project, setting the proxy settings in
${HOME}/.m2/settings.xml and pull the deps in that way. (see
http://maven.apache.org/guides/mini/guide-proxies.html)

They will then be available next time you run lein.

HTH

[rcarmo]

I failed to mention I had tried http_proxy=proxy:3128 as well. It, too, does not work, and is inconsistent with the usual handling of that variable by native binaries such as wget, curl, etc.

It is also inconsistent with other dependencies (especially cascalog-core) being downloaded without any issues from the same server, from clojars, and from a custom Maven repo (I'm using Cloudera Hadoop packages and have that repo declared in project.clj - I omitted that since the issue is specific to those few Cascalog dependencies).

As to downloading manually, I have rsynced the entirety of my local .m2 directory to the server and it still tries to fetch the dependencies for some reason (I haven't run strace on that scenario yet, but based on previous experiments it tries to establish a network connection as well).

[sorenmacbeth]

Just a note - cascalog artifacts aren't hosted at conjars.org, they are hosted by clojars.org.

[zenogantner]

I have exactly the same problem as rcarmo.

I do not agree with sw1nn that the http_proxy variable cannot have a protocol prefix: http:// is only prefixed in case an exception is thrown:
https://github.com/technomancy/leiningen/blob/95bdc72f805d59494ec58e7a63012fe15971d038/leiningen-core/src/leiningen/core/utils.clj#L18

I am using Leiningen 2.3.4 on Java 1.7.0_25 OpenJDK 64-Bit Server VM.

[sritchie]

I bet this'd be fixed if @chriswensel upgraded conjars to the latest code?

[zenogantner]

Thank you for the information, @sritchie
Is there an ETA for this?

Right now, the way to install Cascalog propagated by https://github.com/nathanmarz/cascalog does not work.

[zenogantner]

I assume the error is because of a 404.
If this is the case, then the error message is quite misleading -- maybe leiningen should add specific handling for such cases.

[sritchie]

I'm not actively working on this - the best way forward is to open an issue on conjars (which hosts the deps that are causing trouble).

[sritchie]

@cwensel, had oyu seen this one?

[sritchie]

@zenogantner if this is still an issue, I can transfer this over to Cascading.

[cwensel]

I'm unclear at to what the actual issue is and how to resolve it other than the previously mentioned update the clojars code. on that point, it won't happen anytime soon. they made some major architectural changes that made migrating the db meta-data over challenging (so i was told) so we reverted the attempt. it will happen, just not soon.

we are adding https support. I thought it was in this week, but turns out it is not. i'll press the team, maybe that's the solution.

[cwensel]

OK, I don't think this has anything to do with clojars codebase since /repo is served by Apache, and a connection timed out error is a network error.

but as mentioned above, we are adding HTTPS via a load balancer which may make the interface to lein a better citizen.