Linking to javadoc.io using Javadoc -link option

I have investigated the problem, the issue here is that a user agent must be set (an empty string is ok) in order for the connection to javadoc.io to complete successfully.

I worked the problem around and wrote a Gradle plugin that may be of help for those who rely on that build system.

Unfortunately, the work around can not get ported to the regular javadoc -link command invocation.


It's strange: I could see in the browser e.g. http://static.javadoc.io/org.pegdown/pegdown/1.6.0/package-list but when I add http://static.javadoc.io/org.pegdown/pegdown/1.6.0 as javadoc's link option it says

Error fetching URL: http://static.javadoc.io/org.pegdown/pegdown/1.6.0/package-list

I use next workaround:

  1. With maven-dependency-plugin unapack the javadoc of desired dependency.
  2. Link it with linkoffline option.

    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-dependency-plugin</artifactId>
        <version>2.8</version>
        <executions>
            <execution>
                <id>unpack-javadoc</id>
                <phase>package</phase>
                <goals>
                    <goal>unpack</goal>
                </goals>
                <configuration>
                    <artifactItems>
                        <artifactItem>
                            <groupId>org.pegdown</groupId>
                            <artifactId>pegdown</artifactId>
                            <classifier>javadoc</classifier>
                            <version>${pegdownVersion}</version>
                            <overWrite>false</overWrite>
                            <outputDirectory>${project.build.directory}/pegdown-javadoc</outputDirectory>
                        </artifactItem>
                    </artifactItems>
                </configuration>
            </execution>
        </executions>
    </plugin>
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-javadoc-plugin</artifactId>
        <configuration>
            <links>
                <link>http://www.slf4j.org/apidocs/</link>
            </links>
            <offlineLinks>
                <offlineLink>
                    <url>http://static.javadoc.io/org.pegdown/pegdown/${pegdownVersion}</url>
                    <location>${project.build.directory}/pegdown-javadoc</location>
                </offlineLink>
            </offlineLinks>
        </configuration>
    </plugin>
    

I ended up just using -linkoffline to get around this issue, which I suppose has the nice property of not needing internet connectivity at build time, though if anyone has further thoughts on how to make this work with -link I'm all ears.


From the command line, use an argument like -J-Dhttp.agent=javadoc.

In Maven, use something like:

<additionalJOption>-J-Dhttp.agent=maven-javadoc-plugin-${pom‌​.name}</additionalJO‌​ption>

The background: As Danilo Pianini suggests in another answer, the problem is the User-Agent header. However, the problem isn't an empty User-Agent; it's the default Java User-Agent, which looks something like "Java/1.8.0_112":

$ URL=https://static.javadoc.io/org.checkerframework/checker-qual/2.2.2/package-list

# default Java User-Agent:
$ wget -U Java/1.8.0_112 "$URL" 2>&1 | grep response
HTTP request sent, awaiting response... 403 Forbidden

# no User-Agent:
$ wget -U '' "$URL" 2>&1 | grep response
HTTP request sent, awaiting response... 200 OK

# custom User-Agent:
$ wget -U javadoc "$URL" 2>&1 | grep response
HTTP request sent, awaiting response... 200 OK

So the fix is to tell Javadoc to use a different User-Agent. Java won't let you omit the User-Agent, so you'll have to provide a value, which Java will prepend to its default agent.

As best I can tell, the blocking of Javadoc isn't intentional: Javadoc just (probably unwisely) uses the default Java User-Agent, and the content delivery network that javadoc.io uses blocks that by default.

(One more note about Maven: Everything works fine with -link. It also works fine with -linkoffline if you download the package-list file and tell Javadoc to read it from disk. However, if you use -linkoffline but tell Javadoc to fetch package-list from the javadoc.io URL (this is an unusual thing to do), it may fail. The problem: Maven tries to pre-validate the package-list file but, under some versions of Java, fails because it rejects the SSL certificate of javadoc.io, a certificate that Javadoc itself accepts.)

(Oh, and it appears to be important to use a URL specifically from static.javadoc.io, not javadoc.io. Also, I would recommend https, not http, in case http://static.javadoc.io someday starts issuing redirects to https://static.javadoc.io, as Javadoc currently doesn't handle such redirects. Also, https is a good thing :))

Tags:

Java

Javadoc