CurrencyFair releases OneSignal-Java-SDK as Open Source + Continuous Deployment with Travis CI and OSSRH

Introduction

We at CurrencyFair have recently released an update to our system that enables customers to receive push notifications about a deposit being cleared or an Auto Transaction being completed - you can read more about those features on our marketing blog. For those push notifications we decided to use OneSignal in the backend - a high volume, cross platform push notification delivery provider. Their API is really nice and they offer SDKs for many, many development environments. Go on, check them out too! Adobe, Uber and Line, among many others, also use them!

However, one of the drawbacks when using OneSignal was the lack of a native Java library to programmatically compose and send push notifications to customers. And so we decided to write our own and ultimately publish it as Open Source! You can find the source code here.

Technicalities

Development

OneSignal provides a well-documented REST API. We decided to use OpenFeign together with FasterXML’s jackson to communicate with their endpoints.

Why Feign?

Feign was originally developed by Netflix and is extremely easy to use - Feign aims to connect your code to http APIs with minimal overhead and code. Via customizable decoders and error handling, you should be able to write to any text-based http api.

Why jackson?

A simple quote:

(…) Jackson Project, formerly known as the standard JSON library for Java (or JVM platform in general), or, as the “best JSON parser for Java.” Or simply as “JSON for Java.” More than that, Jackson is a suite of data-processing tools for Java (and the JVM platform), including the flagship streaming JSON parser / generator library (…)

Continuous Deployment

After reading many tutorials and after many tries we finally nailed our own version of deploying our releases: we use Travis CI for building and Central Repository to host the artifacts. It wasn’t an easy task so here are the steps you would want to follow to reach same objectives.

Create a Jira Account at Sonatype.
Create a project ticket.

Add general information to your pom.xml that is required by Sonatype for your release to appear in Central Repository.

 <?xml version="1.0" encoding="UTF-8"?>
 <project xmlns="http://maven.apache.org/POM/4.0.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
     <groupId>org.example</groupId>
     <artifactId>my-library</artifactId>
     <version>0.1.0-SNAPSHOT</version>
     <name>my-library</name>
     <url>https://example.org</url>
     <description>My awesome java library</description>

     <developers>
       <developer>
         <id>jd</id>
         <name>Joe Doe</name>
         <email>joe.doe@example.org</email>
         <url>https://example.org</url>
       </developer>
     </developers>

     <licenses>
       <license>
         <name>MIT/Apache/BSD/GPL...</name>
         <url>URL</url>
         <distribution>repo</distribution>
       </license>
     </licenses>
 </project>

Add distribution management to your pom.xml (notice the <id>ossrh</id>, it’ll be used later).

 <distributionManagement>
   <snapshotRepository>
     <id>ossrh</id>
     <url>https://oss.sonatype.org/content/repositories/snapshots</url>
   </snapshotRepository>
   <repository>
     <id>ossrh</id>
     <url>https://oss.sonatype.org/service/local/staging/deploy/maven2/</url>
   </repository>
 </distributionManagement>

Add maven build and release plugins. You are required to upload a signed jar, sources jar and javadoc jar when releasing to Central Repository. Notice ossrh being used in here.

 <properties>
     <nexus-staging-maven-plugin.version>1.6.7</nexus-staging-maven-plugin.version>
     <maven-gpg-plugin.version>1.6</maven-gpg-plugin.version>
     <maven-source-plugin.version>3.0.1</maven-source-plugin.version>
     <maven-javadoc-plugin.version>2.10.4</maven-javadoc-plugin.version>
 </properties>

 <build>
     <plugins>
         <plugin>
             <groupId>org.sonatype.plugins</groupId>
             <artifactId>nexus-staging-maven-plugin</artifactId>
             <version>${nexus-staging-maven-plugin.version}</version>
             <extensions>true</extensions>
             <configuration>
                 <serverId>ossrh</serverId>
                 <nexusUrl>https://oss.sonatype.org/</nexusUrl>
                 <autoReleaseAfterClose>true</autoReleaseAfterClose>
             </configuration>
         </plugin>

         <plugin>
             <groupId>org.apache.maven.plugins</groupId>
             <artifactId>maven-gpg-plugin</artifactId>
             <version>${maven-gpg-plugin.version}</version>
             <executions>
                 <execution>
                     <id>sign-artifacts</id>
                     <phase>verify</phase>
                     <goals>
                         <goal>sign</goal>
                     </goals>
                 </execution>
             </executions>
         </plugin>

         <plugin>
             <groupId>org.apache.maven.plugins</groupId>
             <artifactId>maven-source-plugin</artifactId>
             <version>${maven-source-plugin.version}</version>
             <executions>
                 <execution>
                     <id>attach-sources</id>
                     <goals>
                         <goal>jar-no-fork</goal>
                     </goals>
                 </execution>
             </executions>
         </plugin>

         <plugin>
             <groupId>org.apache.maven.plugins</groupId>
             <artifactId>maven-javadoc-plugin</artifactId>
             <version>${maven-javadoc-plugin.version}</version>
             <executions>
                 <execution>
                     <id>attach-javadocs</id>
                     <goals>
                         <goal>jar</goal>
                     </goals>
                 </execution>
             </executions>
         </plugin>
     </plugins>
 </build>

Create .travis directory (or any other name) in root of your project, it’ll be used for deployment scripts.

Create .travis/gpg.sh and make it executable.

 $ touch .travis/gpg.sh
 $ chmod +x .travis/gpg.sh

Content:

 #!/usr/bin/env bash

 set -e

 # create a random passphrase
 export GPG_PASSPHRASE=$(echo "$RANDOM$(date)" | md5sum | cut -d\  -f1)

 # configuration to generate gpg keys
 cat >gen-key-script <<EOF
     %echo Generating a basic OpenPGP key
     Key-Type: RSA
     Key-Length: 4096
     Subkey-Type: 1
     Subkey-Length: 4096
     Name-Real: Joe Doe
     Name-Email: joe.doe@example.org
     Expire-Date: 0y
     Passphrase: ${GPG_PASSPHRASE}
     %commit
     %echo done
 EOF

 # create a local keypair with given configuration
 gpg --batch --gen-key gen-key-script


 # export created GPG key
 #
 # example output
 # sec   4096R/EDD32E8B 2017-01-20
 # uid                  Joe Doe <joe.doe@example.com>
 # ssb   4096R/CC1613B2 2017-01-20
 # ssb   4096R/55B7CAA2 2017-01-20
 export GPG_KEYNAME=$(gpg -K | grep ^sec | cut -d/  -f2 | cut -d\  -f1 | head -n1)

 # cleanup local configuration
 shred gen-key-script

 # publish the gpg key
 # (use keyserver.ubuntu.com as travis request keys from this server,
 #  we avoid synchronization issues, while releasing)
 gpg --keyserver keyserver.ubuntu.com --send-keys ${GPG_KEYNAME}

 # wait for the key beeing accessible
 while(true); do
   date
   gpg --keyserver keyserver.ubuntu.com  --recv-keys ${GPG_KEYNAME} && break || sleep 30
 done

This script will generate a signing key for us on the fly when a deployment job is about to take place on Travis CI.

Create .travis/mvnsettings.xml. Notice the ossrh being used again.

 <settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">

     <servers>
         <server>
             <!-- Maven Central Deployment -->
             <id>ossrh</id>
             <username>${env.SONATYPE_USERNAME}</username>
             <password>${env.SONATYPE_PASSWORD}</password>
         </server>
     </servers>

     <profiles>
         <profile>
             <id>ossrh</id>
             <activation>
                 <activeByDefault>true</activeByDefault>
             </activation>
             <properties>
                 <gpg.executable>gpg</gpg.executable>
                 <gpg.passphrase>${env.GPG_PASSPHRASE}</gpg.passphrase>
             </properties>
         </profile>
     </profiles>

 </settings>

Remember exporting GPG_PASSPHRASE in the previous point, in gpg.sh? Travis CI will substitute it in here in place of${env.GPG_PASSPHRASE}. We’ll get to SONATYPE_* in the next points.

Create Travis CI account: open Travis CI, choose Sign Up and connect with your Github profile. Go to Accounts page and select the repo that you would like to be continuously integrated.
Open settings of your repo in Travis (More options -> Settings), you can configure there a couple of things, we are interested in Environment Variables.
Add SONATYPE_USERNAME with the username you registered your Sonatype Jira account with and SONATYPE_PASSWORD with the Sonatype Jira password.

Add .travis.yml to the root of your project. It is very important that you override install instruction from mvn to use the mvnsettings.xml you have created a short time ago.

sudo: false

cache:
  directories:
    - ~/.m2/repository

language: java
jdk:
  - oraclejdk8

install:
  - mvn --settings .travis/mvnsettings.xml install -DskipTests=true -Dgpg.skip -Dmaven.javadoc.skip=true -B -V

deploy:
  -
    provider: script
    script: .travis/deploy.sh
    skip_cleanup: true
    on:
      repo: CurrencyFair/OneSignal-Java-SDK
      branch: master
      jdk: oraclejdk8
  -
    provider: script
    script: .travis/deploy.sh
    skip_cleanup: true
    on:
      repo: CurrencyFair/OneSignal-Java-SDK
      tags: true
      jdk: oraclejdk8
  -
    provider: script
    script: .travis/after_deploy.sh
    skip_cleanup: true
    on:
      repo: CurrencyFair/OneSignal-Java-SDK
      tags: true
      jdk: oraclejdk8

env:
  global:
    - GIT_NAME="Your name"
    - GIT_EMAIL="you@yourdomain.com"

The idea is to:

make a build of your repo every time a new Pull Request appears and new commit is pushed to such Pull Request
make a build and deployment of your repo to Central Repository Snapshot every time a new commit gets added to master branch
make a build and deployment of your repo to Central Repository Release every time a new tag is pushed to your repository
update version in pom.xml and push to Github repo once a tag-triggered release happened

Create a ssh key that will be used for approving Travis pushing into your repo.
```
$ ssh-keygen -t rsa -b 4096 -C "<your_email>" -f github_deploy_key -N ''
```
This will generate 2 keys in 2 files:
- public key in github_deploy_key.pub
- private key in github_deploy_key
Add public key to Github repo: go to https://github.com/<username>/<repository>/settings/keys, click Add deploy key, name it, paste the content of github_deploy_key.pub in the textarea, select Allow write access and click Add key. You can now remove github_deploy_key.pub from your disk.
This is very important now that you execute this and the next steps not in your fork but a directly cloned original repo (most important for Github Organizations).

Install Travis CLI
```
$ gem install travis
```
Login into Travis (you should execute this inside your repo directory), you will be asked for your github credentials
```
$ travis login
```
Encrypt the github_deploy_key with Travis CLI.
```
$ travis encrypt-file github_deploy_key
```
You will see output similar to
```
encrypting github_deploy_key for <username>/<repository>
storing result as github_deploy_key.enc
storing secure env variables for decryption

openssl aes-256-cbc -K $encrypted_XXXXXXXXXXXX_key -iv $encrypted_XXXXXXXXXXXX_iv -in github_deploy_key.enc -out github_deploy_key -d

Pro Tip: You can add it automatically by running with --add.

Make sure to add github_deploy_key.enc to the git repository.
Make sure not to add github_deploy_key to the git repository.
Commit all changes to your .travis.yml.
```
If you go to https://travis-ci.org/<username>/<repository>/settings you will see there are two new environment variables encrypted_XXXXXXXXXXXX_key and encrypted_XXXXXXXXXXXX_iv. If you don’t see them double-check that you are executing this step not from forked repo directory. This is security measure enforced by Travis-CI - all encrypted files are local only to given fork and are unavailable to other forks and push requests. If you didn’t check this and continued on with the rest of this post then you would see
```
iv undefined
```
in the Travis job log. Be smart, don’t spent hours on looking for what’s wrong as I did because nobody else wrote this clear enough.
For the sake of order, move the github_deploy_key.enc to .travis directory.
```
$ mv github_deploy_key.enc .travis/
```
And delete the private key
```
$ rm -f github_deploy_key
```

Add .travis/deploy.sh script and make it executable.

$ touch .travis/deploy.sh
$ chmod +x .travis/deploy.sh

Content:

#!/usr/bin/env bash
set -e

# update current version number to a TAG version if this is a tag build
if [ ! -z "$TRAVIS_TAG" ]
then
    echo "on a tag -> set pom.xml <version> to $TRAVIS_TAG for release"
    mvn --settings .travis/mvnsettings.xml org.codehaus.mojo:versions-maven-plugin:2.3:set -DnewVersion=$TRAVIS_TAG
else
    echo "not on a tag -> keep snapshot version in pom.xml"
fi

# cleanup and generate gpg keys
if [ ! -z "$TRAVIS" -a -f "$HOME/.gnupg" ]; then
    shred -v ~/.gnupg/*
    rm -rf ~/.gnupg
fi

source .travis/gpg.sh

# DEPLOY \o/
mvn clean deploy --settings .travis/mvnsettings.xml -DskipTests=true --batch-mode --update-snapshots

# cleanup gpg keys, just to be safe
if [ ! -z "$TRAVIS" ]; then
    shred -v ~/.gnupg/*
    rm -rf ~/.gnupg
fi

Let’s go through this script step by step:

update current version number in pom.xml to a TAG version if this is a tag build ($TRAVIS_TAG is an environment variable Travis CI provides when a build is triggered by a tag pushed to repository)
cleanup and generate gpg keys
call the mvn deploy with extra settings - this will generate jar, -sources.jar, -javadoc.jar artifacts, will sign them all with gpg and finally deploy to Central Repository (Snapshot or Release depending on version number - it’s really important that you stick to Semantic Versioning)

Add .travis/after_deploy.sh script and make it executable. Remember it will be only called on tag push.

$ touch .travis/after_deploy.sh
$ chmod +x .travis/after_deploy.sh

Content:

#!/usr/bin/env bash
set -e

increment_version_to_next_snapshot ()
# http://stackoverflow.com/a/8659330/5226815
{
  declare -a part=( ${1//\./ } )
  declare    new
  declare -i carry=1

  for (( CNTR=${#part[@]}-1; CNTR>=0; CNTR-=1 )); do
    len=${#part[CNTR]}
    new=$((part[CNTR]+carry))
    [ ${#new} -gt $len ] && carry=1 || carry=0
    [ $CNTR -gt 0 ] && part[CNTR]=${new: -len} || part[CNTR]=${new}
  done
  new="${part[*]}"
  new_snapshot="${new// /.}-SNAPSHOT"
}

# Get the deploy key by using Travis's stored variables to decrypt deploy_key.enc
openssl aes-256-cbc \
    -K $encrypted_dd04a63e1379_key \
    -iv $encrypted_dd04a63e1379_iv \
    -in .travis/github_deploy_key.enc \
    -out .travis/github_deploy_key -d
chmod 600 .travis/github_deploy_key
eval `ssh-agent -s`
ssh-add .travis/github_deploy_key
git config --global user.name "$GIT_NAME"
git config --global user.email "$GIT_EMAIL"

# increment version to the next snapshot version
increment_version_to_next_snapshot $TRAVIS_TAG
echo "on a tag -> set pom.xml <version> to next SNAPSHOT $new_snapshot"
mvn --settings .travis/mvnsettings.xml org.codehaus.mojo:versions-maven-plugin:2.3:set -DnewVersion=$new_snapshot
mvn --settings .travis/mvnsettings.xml org.codehaus.mojo:versions-maven-plugin:2.3:commit

# Save some useful information
REPO=$(git config remote.origin.url)
REPO=${REPO/https:\/\/github.com\//git@github.com:}
REPO_NAME=$(basename $REPO)
TARGET_DIR=$(mktemp -d /tmp/$REPO_NAME.XXXX)
git clone --branch master ${REPO} ${TARGET_DIR}
yes | cp -f pom.xml $TARGET_DIR/
cd $TARGET_DIR
# Finally add changed pom.xml and commit
git add pom.xml
git commit -m "[skip ci] [Travis-CI] next snapshot version via Travis build $TRAVIS_BUILD_NUMBER"
# Now that we're all set up, we can push
git push $REPO master

Step by step actions of this script:

prepare increment_version_to_next_snapshot function that increments semver version from tag and postfixes it with -SNAPSHOT
decrypt the encrypted github_deploy_key.enc key and add it to ssh agent
configure git to use the name and email setup in .travis.yml
increment the version in pom
clone the repository again, with working refs, master branch known and ssh access
copy the pom.xml file from old location to the newly cloned repo
commit the changed file, add [skip ci] so that Travis-CI will not fire again when we…
push that commit to origin

Commit and push everything, your Travis CI job should now start.

You can now

git tag -a 1.0.0 -m "Version 1.0.0 - first public release"

your application :)

tech.CurrencyFair

CurrencyFair releases OneSignal-Java-SDK as Open Source + Continuous Deployment with Travis CI and OSSRHPosted 20 Jan 2017 by Mateusz Bajorek

Introduction

Technicalities

Development

Why Feign?

Why jackson?

Continuous Deployment