At this point, we’re about done discussing Omni’s push architecture! We’ve already seen Go, the APNs data structures and connection, and the pipeline processor that manages hundreds of thousands of push notifications for OmniFocus customers every day. All that’s left is to take a quick tour of some other parts of the stack that support this infrastructure.
As mentioned earlier, much of Omni Sync Server is built atop FreeBSD, and the
push provider is no exception. Since Go targets FreeBSD as a “first-class
citizen” for compilation and execution, and there’s a port for the Go language,
it’s easy to get a Go toolchain up and running. Simply run pkg install lang/go
to get all the tools needed for building Go programs.
At Omni, we keep a FreeBSD system in the rotation in our automated build system. Every few hours, it checks out the latest source for the push provider from our internal SVN server, builds it, and archives the result. Along with the actual Go sources for the provider, we provide a Makefile and package information, so that the archived product is an xzipped FreeBSD package. This way, we can take advantage of FreeBSD’s existing package management system for easy deployment and upgrades of the push provider.
Next up, we needed a way to hang on to notification information: what clients were registered with the provider, how they’re grouped, and some statistics tracking. We also needed to integrate with Omni Sync Server for a staged rollout of push: during testing, we enabled push only for some sync servers, in order to measure the kind of extra load that push would levy on our sync system. (Thankfully, this period was very brief, and push is enabled for every customer now.)
All of this information was very well suited for a relational database. However, we’re in a bit of a transitional period there as well. For older applications, like Omni Sync Server, we already have MySQL set up and running well. For newer code, we’re trying to use PostgreSQL wherever possible.
As a result, the push provider wound up talking to both databases! We store all the new push-related information – things like device and group IDs, as well as counts of notifications sent – in PostgreSQL. We also interfaced with a read-only replica of the Omni Sync Server database for checking customers’ sync servers. (While this might not seem like the optimal solution, we’ve been happily using MySQL in production for a few years, and there wasn’t any sense in potentially destabilizing everybody’s sync experience for a PostgreSQL migration. If it ain’t broke, don’t fix it!)
The Web portion of the provider was fairly straightforward. Rather than try to wrap Web access in Apache or nginx, or write a separate Web interface that called a push API, we used Go’s built-in HTTP and HTML templating support to handle all incoming HTTP requests and expose a simple but serviceable administrative interface.
Of course, security – and future compatibility – were big concerns for push. On
top of plain HTTP, Go’s standard library also provided a flexible implementation
for handling HTTPS through the crypto/tls package. With this, we were able to
ensure that the provider passed Apple’s strict ATS requirements, and that all our connections to Apple’s push
service were encrypted as well.
In building the push provider, our engineers stood on the shoulders of giants. Lots of hard work went into each of the components mentioned in this post, and thanks in large part to their efficiency and utility, we were able to start work on the provider and deploy it to our customers in less than three months. Of course, our customers were invaluable in this process as well - hundreds of helpful people volunteered for the OmniFocus beta program, and helped us see how push would work together with Omni Sync Server.
As of this writing, the push provider has already sent over ten million push notifications. We’re looking forward to the next ten!
