The following is a guest post by Jonah Lopin (@jonahlopin), founder at Crayon. Jonah is a HubSpot alumnus, and I'm an angel investor in the company.
Here at Crayon, we love humans.
In fact, all our moms and many of our best friends are humans!
But when it comes to solving problems, we prefer code.
Humans are awesome at certain things, like closing enterprise sales deals, delivering rousing speeches, and comforting small children. For these tasks and lots of others, humans are far superior to code.
But in a business context, humans have some serious drawbacks:
1. They’re hard to recruit (Crayon is hiring, by the way)
2. They’re expensive (compared to servers)
3. They sometimes get sick and can’t come to work (☺)
For a product-driven software company, there’s something else to consider. It’s subtle, but important.
Software is better than humans at providing an elegant solution to complex problems at scale.
Tomasz from Redpoint put together some fascinating data back in 2012 that showed billion dollar public SaaS companies have revenue per employee around 200k/year. Jeremiah Owyang puts revenue per employee at Google & Facebook around $1m per employee, about 5x higher! Clearly, Facebook & Google solve more problems with code than a typical B2B software company.
The fact that Google & Facebook solve their primary problems with code rather than people doesn’t just impact their metrics and margins, it shapes the elegance and efficiency of the solutions they provide. The subtle advantage of solving problems with code rather than humans is that it tends to make your core product better over time.
Crayon is small, but we dream big. We solve problems with code, not humans, because we want to serve millions of users as elegantly as possible. It’s just down-right hard to do that with too many humans in the mix, especially if you want to scale quickly. (And especially if you’re not Uber.)
The rest of this post is about a big problem we had and how we solved it with code.
The Problem: Some Web Pages Don’t Look So Hot
Crayon is a visual inspiration platform. We help marketers get great ideas so they can do better marketing.
We’re programmatically adding about half a million pages to our system every day. As I write this article, we’ve got about 13 million designs in the system, and we’ll have close to 100 million by the end of the year.
We let users vote designs up and down, so most categories on Crayon like Startup Home Pages, B2B Pricing Pages and Landing Pages have always looked pretty good.
But we still had a problem in deeper categories: folks would be happily browsing, only to have the smooth inspirational vibe unceremoniously broken by a crappy looking page. How could we get the “best” designs to the top of the result set and the “least inspiring” designs to the bottom?
We Challenged Ourselves to Solve it With Code
Can code separate “inspiring” marketing designs from “crappy” ones?
Many said it couldn’t be done… but if they were right, would I be writing this article?
Here’s How We Solved It
Step 1: Pick Training Sets
We picked a set of 200 “inspiring” marketing designs and a set of 200 “uninteresting” marketing designs.
Yes, this is a bit of a fuzzy thing to do because it’s based on human judgment. But after running the services team at HubSpot for 5.5 years, and working closely with more than 8,000 professional marketers, I felt qualified to judge which marketing designs were likely to be interesting and instructive to other marketers. So sue me!
Step 2: Make Some Guesses About Why The Good Ones Are Good
This part is like picking a basketball team without being able to watch the candidates play basketball. What characteristics of the players would you look at? For instance, you might pick taller players or players wearing high-tops.
In the style of Jeff Foxworthy’s comedy routine You Might Be a Redneck:
If your Html is all based on tables… you might not have a great marketing design (tweet)
If you’ve got lots of inline styles… you might not have a great marketing design (tweet)
If you don’t use media queries in your CSS… you might not have a great marketing design (tweet)
You get the idea.
Step 3. Test Your Guesses
We wrote some code to test each “guess” from step 2 against each design in the training sets from step 1. We were hoping to find things that were “true” for the “inspiring” pages, and “untrue” for the “uninteresting pages”.
We looked at 45 discrete “guesses”, and found 25 of them were predictive of “inspiring” marketing designs. Success!
Some of the best factors were things like:
- Setting the viewport meta tag
- Including Facebook Open Graph tags
- Using the Bootstrap framework
- Specifying Apple touch icons
Note that these factors don’t directly predict which pages are “inspiring”. Rather, these factors indicate that someone clueful created the page, which is directly predictive of a page being “inspiring”.
Some of the things we thought might work, but weren’t predictive at all include:
- Whether a page uses JQuery doesn’t matter… it’s just too ubiquitous
- The total text on the page doesn't matter. It turns out there are equally as many crappy short pages as there are long pages (tweet)
The final step was some mathematical mojo based on how strong the signals were in step 2 to come up with an overall “inspiringness score” for each page.
The Results
Pages like the chatterbox.co homepage did very well:
While our friends at biomarkerstrategies.com didn’t fare as well:
Overall, here’s what we got:
How awesome is that?
The top 10% of Crayon search results just went from 50% “inspiring” to a whopping 93%!
We have 13 million designs in the system today, and reviewing those manually would take a 10 person team about 5 years. Ouch! But we’ll have 100 million designs by the end of the year, and that’s just the beginning. If we didn’t solve this problem with code, we’d be in serious trouble. And no humans were involved… except for the one human writing this article.
Your thoughts?
Does your business have a tough problem you plan to solve with code rather than humans?
Have you used machine learning to deliver elegant solutions to customers at scale?
Please continue the converstaion in the comments.
Go solve something with code! (tweet)