This page is intended for all sorts of random topics that are worth being recorded to allow further troubleshooting.

Jenkins

Everything related to Jenkins

GitHub webhooks

Note : Github webhook for apache/incubator-mxnet can only be created by the Apache Infra team. A request needs to be made for setting up a new webhook / updating existing webhook. For ex : INFRA-20085 - Getting issue details... STATUS

GitHub webhooks allow to be notified of GitHub events instead of polling. This allows us to operate without running into rate limiting. The configuration in Jenkins has a few catches that can save you quite some time:

Create a webhook at https://github.com/ACCOUNT/REPOSITORY/settings/hooks/. You can use this configuration for reference:

Note : Make sure to include the trailing `/` in the Payload URL section for configuring the webhook on Github. Refer : 302 Github webhook with Jenkins error on StackOverflow.

You will have to create a secret manually. Note it down as it is required to be entered into Jenkins.

In Jenkins, navigate to the general "Configure System" page (like http://jenkins.mxnet-ci-dev.amazon-ml.com/configure). You might be tempted to navigate to "GitHub Pull Request Builder" which looks like this, but this is not the one we are looking for! The shared secret here is something different and might cost you a few hours to figure out!

 Wrong field

Instead, navigate to the tab called "GitHub". It looks like follows:

Click on "Advanced..." and you will see the following:

On "Shared Secret", press "Add". A new window will open up. In the "Kind" dropdown, select "Secret text". Then input something like the following and insert the previously noted down secret:

Now go back to the GitHub webhook and press "Redeliver":

Jenkins Job does not get triggered by WebHook

In case the web hook has properly been set up, go to the system log (e.g. http://jenkins.mxnet-ci-dev.amazon-ml.com/log/all) and search for something along the following lines:

Received PULL_REQUEST for https://github.com/MXNetEdge/incubator-mxnet from 192.30.252.40 ⇒ http://jenkins.mxnet-ci-dev.amazon-ml.com:8080/github-webhook

If you don't see this message, this means that the we bhook has not been set up properly. consult the previous guide how to continue. This message is a prerequisite for this guide.

To analyze a certain job, go to that job and look for the entry "Repository Events" (e.g. http://jenkins.mxnet-ci-dev.amazon-ml.com/job/test-marco-mxnet-sanity/indexing/events). You will see something along the following lines:

[Tue Nov 20 14:52:10 UTC 2018] Received Pull request #5 opened in repository MXNetEdge/incubator-mxnet CREATED event from 192.30.252.35 ⇒ http://jenkins.mxnet-ci-dev.amazon-ml.com:8080/github-webhook/ with timestamp Tue Nov 20 14:52:05 UTC 2018
14:52:11 Connecting to https://api.github.com using MXNetEdgeBot/****** Examining MXNetEdge/incubator-mxnet

  Checking branches...

  Getting remote branches...

    Checking branch master

  16 branches were processed

  Checking pull-requests...

  Getting remote pull request #5...

    Checking pull request #5
      ‘ci/jenkins/Jenkinsfile’ not found
    Does not meet criteria

    Checking pull request #5
      ‘ci/jenkins/Jenkinsfile’ not found
    Does not meet criteria

  2 pull requests were processed

Finished examining MXNetEdge/incubator-mxnet

[Tue Nov 20 14:52:12 UTC 2018] Pull request #5 opened in repository MXNetEdge/incubator-mxnet CREATED event from 192.30.252.35 ⇒ http://jenkins.mxnet-ci-dev.amazon-ml.com:8080/github-webhook/ with timestamp Tue Nov 20 14:52:05 UTC 2018 processed in 1.7 sec

Here you can see, for example, that the PR did not contain the requested file. In this particular case, the background was a bit different. See the guide below for details

Jenkinsfile not found or changes not reflected

If you create a new Jenkinsfile and it does not get picked up by Jenkins with the following message if you submit a pull request

Checking pull request #5
  ‘ci/jenkins/Jenkinsfile’ not found
Does not meet criteria

this might not mean that the file is missing but rather be related to the security restrictions Jenkins has. The background here is the fact that Jenkins does not evaluate Jenkinsfiles from people without write permissions to the repository. Request somebody with write permissions to open the pull request instead.

GitHub commit/PR status does not get published 

In case the commit status is not being properly published, go to the Jenkins log and look for the following line:

[Set GitHub commit status (universal)] PENDING on repos [] (sha:xxxxxxx) with context:test/mycontext

Note that the repos-array is empty. This is because the GitHubCommitStatusSetter is not able to resolve repositories if they have not been configured within Jenkins. In order to fix this, go to 

Manage Jenkins > Configure System > GitHub > GitHub Servers

and add the following config:

For credentials, make sure to use credentials that have permissions to add the commit status. At Apache, this would be committer credentials.

What you will want to see in the logs is the following:

[Set GitHub commit status (universal)] PENDING on repos [GHRepository@362b786c[description=Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more,homepage=http://mxnet.io,name=incubator-mxnet,fork=true,size=40782,milestones={},language=C++,commits={},responseHeaderFields={null=[HTTP/1.1 200 OK], Access-Control-Allow-Origin=[*], Access-Control-Expose-Headers=[ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type], Cache-Control=[private, max-age=60, s-maxage=60], Content-Encoding=[gzip], Content-Security-Policy=[default-src 'none'], Content-Type=[application/json; charset=utf-8], Date=[Tue, 20 Nov 2018 13:36:45 GMT], ETag=[W/"417522734a6356ef821736c139646b34"], Last-Modified=[Tue, 20 Nov 2018 13:03:10 GMT], OkHttp-Received-Millis=[1542721005396], OkHttp-Response-Source=[NETWORK 200], OkHttp-Selected-Protocol=[http/1.1], OkHttp-Sent-Millis=[1542721005230], Referrer-Policy=[origin-when-cross-origin, strict-origin-when-cross-origin], Server=[GitHub.com], Status=[200 OK], Strict-Transport-Security=[max-age=31536000; includeSubdomains; preload], Transfer-Encoding=[chunked], Vary=[Accept, Authorization, Cookie, X-GitHub-OTP, Accept-Encoding], X-Accepted-OAuth-Scopes=[repo], X-Content-Type-Options=[nosniff], X-Frame-Options=[deny], X-GitHub-Media-Type=[github.v3; format=json], X-GitHub-Request-Id=[E560:56B6:12219B6:147A932:5BF40DED], X-OAuth-Scopes=[admin:repo_hook, repo, repo:status], X-RateLimit-Limit=[5000], X-RateLimit-Remaining=[4480], X-RateLimit-Reset=[1542722488], X-XSS-Protection=[1; mode=block]},url=https://api.github.com/repos/MXNetEdge/incubator-mxnet,id=123927478]] (sha:d731aa6) with context:ci/jenkins/test-marco-mxnet-sanity
Setting commit status on GitHub for https://github.com/MXNetEdge/incubator-mxnet/commit/d731aa63a6fdfd502fc7aa63df1fa437f7b8407c