Posted on

Using Make with React and RequireJS

RequireJS is a great library for building modular JavaScript clients that conform to the AMD API specification. It gives your JS an import-like mechanism by which you avoid global-namespace issues and makes your code behave more like a server-side language such as Python or Java. It also includes an optimization tool that can concatenate your JavaScript into a single file and minify it, which helps reduce HTTP overhead.

I recently wrote about React, which is a library targeted at constructing UIs. React uses a special syntax called JSX for specifying DOM components in XML, and it compiles down to vanilla JavaScript. JSX can either be precompiled or compiled in the browser at runtime. The latter option has obvious performance implications and probably shouldn’t be used in production, but it works great for quickly hacking something together.

However, if you’re using RequireJS and opt to defer JSX compilation to the browser, you’ll have problems loading your JSX modules since they aren’t valid JavaScript. Fortunately, there are RequireJS plugins to work around this, such as require-jsx, which lets you simply do the following:

The require-jsx plugin just performs the compilation when a module is loaded.

The other option, as I hinted at, is to precompile your JSX. This offloads the JSX transformation and allows Require’s optimizer to minify your entire client. React has a set of complementary tools, aptly named react-tools, which includes a command-line utility for performing this compilation.

The jsx tool can also watch directories, doing the compilation whenever the source changes, with the `–watch` option.

Require now has no problem loading our React components since they are plain ole JavaScript—no special JSX plugins necessary. This also means we can easily hook in minification using Require’s r.js:

You can use whatever build tool you want to tie all these things together. I personally prefer Make because it’s simple and ubiquitous.

Running `make js` will install my Bower dependencies, perform JSX compilation, and then minify the client. This workflow works well and makes it easy to setup different build steps, such as pip installing Python requirements, running integration and unit tests, and performing deploys.

Posted on

Building User Interfaces with React

If you follow me on Twitter, you’ve probably heard me raving about React. React is described as a “JavaScript library for building user interfaces” and was open sourced by Facebook about a year ago. Everybody and their mom has a JavaScript framework, so what makes React so interesting? Why would you use it over mainstays like Backbone or Angular?

There are a few things that make React worth looking at. First, React is a library, not a framework. It makes no assumptions about your frontend stack, and it plays nicely with existing codebases, regardless of the tech you’re using. This is great because you can use React incrementally for new or legacy code. Write your whole UI with it or use it for a single feature. All you need is a DOM node to mount.

React is delightfully simple (contrasted with Angular, which is a nightmare for beginners, and Backbone, which is relatively simple but still has several core concepts). It’s built around one idea: the Component, which is merely a reusable unit of UI. React was designed from the ground-up to be composable—Components are composed of other Components. Everything in the DOM is a Component, so your UI consists of a hierarchy of them.

Components can be built using JSX, an XML-like syntax that compiles down to regular JavaScript. As such, they can also be specified using plain-old JavaScript. The result is the same, but JSX makes it easy to visualize your DOM.

React does not do two-way data binding ((It’s worth noting that React provides a small add-on for getting the conciseness of two-way binding with the correctness of its one-way binding model.)). This is by design. It uses the von Neumann model of dataflow, which means data flows in only one direction. Two-way data binding makes it difficult to reason about your code. The advantage of the one-way model that React adopts is that it essentially turns your UI into a deterministic state machine. On the surface, it behaves as if the entire UI is simply re-rendered based on the current state of your data model. If you know what state your data is in, you know exactly what state your UI is in. Your UI is predictable. The React mantra is “re-render, don’t mutate.”

Re-rendering the entire DOM sounds expensive, but this is where React really shines. In order to draw your UI, it maintains a virtual DOM which can then be diffed. React’s diffing algorithm determines the minimum set of Components that need to be updated. It also batches reads and writes to the real DOM. This makes React fast.

Data is modeled two ways in a React component, props and state, which highlights the one-way data flow described earlier. Props consist of data that is passed from parent to child. A Component’s props can only be set by its parent. State, on the other hand, is an internal data structure that is accessed and modified only from within a Component. A Component is re-rendered when either its props or state is updated.

Once again, this makes it really easy to reason about your code (and unit test). Also note the use of the onClick handler. React provides a synthetic event system that gives you cross-browser compatible event listeners that you can attach to Components.

React and Backbone’s Router is a surprisingly powerful, yet effortless, combination for building single-page applications.

React makes it trivial to build small web apps, but because of its affinity for reusability and data modeling, it also scales well for large, complex UIs. You don’t have to use it for a new project, just start replacing small pieces of your UI with React Components. This makes it a lot easier for developers to adopt.

Posted on

Real-Time Client Notifications Using Redis and Socket.IO

Backbone.js is great for building structured client-side applications. Its declarative event-handling makes it easy to listen for actions in the UI and keep your data model in sync, but what about changes that occur to your data model on the server? Coordinating user interfaces for data consistency isn’t a trivial problem. Take a simple example: users A and B are viewing the same data at the same time, while user A makes a change to that data. How do we propagate those changes to user B? Now, how do we do it at scale, say, several thousand concurrent users? What about external consumers of that data?

One of our products at WebFilings called for real-time notifications for a few reasons:

  1. We needed to keep users’ view of data consistent.
  2. We needed a mechanism that would alert users to changes in the web client (and allow them to subscribe/unsubscribe to certain events).
  3. We needed notifications to be easily consumable (beyond the scope of a web client, e.g. email alerts, monitoring services, etc.).

I worked on developing a pattern that would address each of these concerns while fitting within our platform’s ecosystem, giving us a path of least resistance with maximum payoff.

Polling sucks. Long-polling isn’t much better. Server-Sent Events are an improvement. They provide a less rich API than the WebSocket protocol, which supports bi-directional communication, but they do have some niceties like handling reconnects and operating over traditional HTTP. Socket.IO provides a nice wrapper around WebSockets while falling back to other transport methods when necessary. It has a rich API with features like namespaces, multiplexing, and reconnects, but it’s built on Node.js, which means it doesn’t plug into our Python stack very easily.

The solution I decided on was a library called gevent-socketio, which is a Python implementation of the Socket.IO protocol built on gevent, making it incredibly simple to hook in to our existing Flask app. 

The gevent-socketio solution really only solves a small part of the overarching problem by providing a way to broadcast messages to clients. We still need a way to hook these messages in to our Backbone application and, more important, a way to publish and subscribe to events across threads and processes. The Socket.IO dispatcher is just one of potentially many consumers after all.

The other piece of the solution is to use Redis for its excellent pubsub capabilities. Redis allows us to publish and subscribe to messages from anywhere, even from different machines. Events that occur as a result of user actions, task queues, or cron jobs can all be captured and published to any interested parties as they happen. We’re already using Redis as a caching layer, so we get this for free. The overall architecture looks something like this:

pubsub

Let’s dive into the code.

Hooking gevent-socketio into our Flask app is pretty straightforward. We essentially just wrap it with a SocketIOServer.

The other piece is registering client subscribers for notifications:

NotificationsNamespace is a Socket.IO namespace we will use to broadcast notification messages. We use gevent-socketio’s BroadcastMixin to multicast messages to clients.

When a connection is received, we spawn a greenlet that listens for messages and broadcasts them to clients in the notifications namespace. We can then build a minimal API that can be used across our application to publish notifications.

Wiring notifications up to the UI is equally simple. To facilitate communication between our Backbone components while keeping them decoupled, we use an event-dispatcher pattern relying on Backbone.Events. The pattern looks something like this:

This pattern makes it trivial for us to allow our views, collections, and models to subscribe to our Socket.IO notifications because we just have to pump the messages into the dispatcher pipeline.

Now our UI components can subscribe and react to client- and server-side events as they see fit and in a completely decoupled fashion. This makes it very easy for us to ensure our client-side views and models are updated automatically while also letting other services consume these events.

Posted on

No More Ninjas

How does a software company attract talent? Compensation? That’s how they attract people. Free lunches and foosball tables? Keep guessing.

The most effective way for a company to bag top-tier engineers is simple: let developers be developers.

Engineers Drive Innovation

Software is a symbiotic creature, and there are two ways a product is built: from the top down and the bottom up. Top-down development means that requirements are curated by and flow from the business, product owners, and managers, downward to the engineers. With the latter, developers explore new ideas and use technology that may be outside the organization’s standard gamut. Ideas for new products or features are born and pushed up to product owners where they can be cultivated.

Both of these methods must strike a balance for a company and its products to be successful. Without a focused vision, a product will fail. Without embracing new ideas and technology, a company will become irrelevant.

Open Source is King

How an organization perceives OSS is paramount to developers. Companies that contribute and maintain open source projects tend to attract talented software engineers. Just look at some of the top repositories on GitHub. Unfortunately, there can sometimes be an eternal struggle between engineers and lawyers.

It’s equally important that a company supports the use of OSS in its own products. I’ve worked on projects where the use of open source code was discouraged in favor of homegrown solutions. This is dangerously senseless considering most mature, open source projects are battle tested and have large community support.

Developers want to work at companies that embrace open source, which is a push-pull dynamic.

Let Me Work, Dammit

I do all of my development on a MacBook Pro running OSX. Some of my coworkers use similar hardware configurations with various flavors of Linux. Back when I worked with .NET, I used Windows. The CLI is now my home, however, and Vim is Old Reliable sitting in the driveway. Some prefer IntelliJ, some PyCharm, and yet others Sublime Text—to each his own.

What I’m getting at is everyone has their preferred setup for building software. If your product does not have technical constraints resulting from the stack you’re using, don’t force me out of my element. Let me use the environment I’m comfortable with because it’s the environment I’m productive with.

No More Ninjas

Tech recruiters: no more code ninjas, no more Jedis, no more rock stars, no more gurus. These words are an unfortunate by-product of the startup culture. You may have the best of intentions, but to me, these words are red flags and marginally patronizing.

Perhaps I’m overly cynical, but I tend to translate job postings of this nature as “Seeking to exploit naive, young programmer to build a startup by working unsustainable hours. You’ll get equity!” It’s the hard sell.

We get it, you’re looking for skilled programmers, but no self-respecting programmer will call him or herself a rock star. And if they do, not only does it blow expectations out of proportion, it makes them look like a dick. Let your company, its products, and its culture speak for themselves. If you aren’t finding the “rock stars” you were looking for, it wasn’t meant to be.

Let Developers Be Developers

Yes, I’m biased. Developers are opinionated—I certainly am. Nonetheless, I would hazard to guess many top tech companies tend to address several of the points I’ve touched on. Call it headstrong, obstinate, whatever, but these are things I like to get a feel for when interviewing with a company. Others may disagree, but I suspect you wouldn’t be too hard-pressed to find many who take the craft seriously share similar views.

Posted on

The Art of Software

I stared at my coffee as a friend asked a mildly profound question: “what is your greatest passion in life?” Strangely, I thought of an exchange that occurred just a few months earlier while I was at the bank, opening an account for my software consulting company.

“What is the nature of your company’s business?” the banker asked.

“Building software,” my partner and I managed to respond with. Now suddenly fascinated, she looked away from her computer and at us, as if we had abruptly sprouted wings.

“Like Facebook?”

I did my best to suppress a laugh and simply nodded. I could hardly blame her. The 30-something-year-old banker’s knowledge of the domain likely consisted of the horrendous software they used at the bank, Angry Birds, and whatever insight she gleaned from watching The Social Network. About equivalent to my knowledge of banking.

I looked up from my coffee. Naturally, my answer would immediately gravitate towards software, but how to frame it in a way that made sense?

Building something that impacts people—making their lives easier, more enjoyable, or something to that effect—that’s really the goal of any engineer, but it’s an amazing ability that software developers possess. The modern-day alchemist, quite literally turning electrons into social media and flight simulators and bank transactions—things that delight or captivate; things that help us exist. We’re the janitors of data, or is it plumbers? But we’re also artists.

Software is a craft, an art, and not one that everyone is capable of doing (but certainly capable of learning). But what is art? Something that exhibits beauty in aesthetics, perhaps visually, perhaps audibly. We think of Michelangelo’s painting of the Sistine Chapel or Beethoven’s Fifth Symphony. We probably don’t think of Twitter or Facebook and certainly not the ATM you withdrew money from.

Web designers may argue that websites can be beautiful, and they’re right. Software can be both a visual and aural stimulus, but these are really just superficial observations. The same way I say “Huh, that’s pretty cool” after seeing the Sistine Chapel. Others may appreciate it more, or less.

Many would argue that the purpose of art is to evoke emotion from its audience. There’s probably a lot of truth to that, but to me, art exists on several different levels. There’s the immediate aesthetic beauty—what the eyes see and what the ears hear. There’s the emotions that it evokes from its audience—this makes me happy, this makes me sad, this makes me intrigued. And then there’s the appreciation of the craft: painters admiring a painting, not because of its obvious beauty, but because of the technical prowess it demonstrates. Software is no different.

I read something once that said software developers are never impressed. I don’t think that’s entirely true, or at least the entirety of the truth. Beyond the superficial veneer of beauty, there are the inner workings of incredibly complex systems, riddled with algorithms, design patterns, and all the things that make programmers swoon. And they run like well-oiled machines. That is art, but art that only others who share the trade can enjoy. There is elegance in code like there is elegance in the prose of a novel. The difference is in those who are able to marvel at it.

The great thing about engineering, software or otherwise, is the problem solving involved. There is no end to the amount of exceptionally difficult problems that need solving, and the way in which those problems are solved is an art in itself.

I doubt many people associate the words “art” and “software,” but there is as much creativity and craftsmanship in building software as there is engineering. It’s this drive, to build great products that affect people, that I consider my passion and my art.