Ten Years On – The Million Monkeys Project

Blog Summary: (AI Summaries by Summarizes)
  • The author randomly recreated every work of Shakespeare using their own computer and cloud computing resources.
  • The project began with an inauspicious start and required careful calculations to randomly recreate every work of Shakespeare.
  • The author submitted the project to various sites' tip lines and received coverage from Gizmodo and Fox News.
  • The waiting period for the project to complete was longer than expected due to the Coupon Collector's Problem.
  • The author submitted the project to every mainstream and tech news outlet they could think of and received coverage from the BBC, leading to a chain reaction of coverage.

I want to tell you a story about how my life changed. It wasn’t a cult, new religion, or programming language. A million monkeys changed my life.

Ten years ago, I randomly recreated every work of Shakespeare. It was quite a project on the technical, news, and personal levels. Things would never be the same again. I want to tell you some of the backstories and how the outcomes of this personal project could apply in your own life.

In The Beginning

Let’s start at the beginning. Have you ever heard the saying that a million monkeys on a million typewriters will eventually create Shakespeare? I didn’t have a million monkeys available, so I used my own computer and, what was then novel, cloud computing resources. It took about a month and a half, but the monkeys created every work of Shakespeare.

The whole project began with an inauspicious start. I first conceived of the project and told a few people about it, and no one cared, not even my friends and family. Undeterred, I started the project. My original idea was to use Hadoop MapReduce to create the longest random group of characters ever found, which was 25 or 26 characters long. Given the math, that would have taken far longer than I wanted to spend on the project. Given the yawns from friends and family, I wasn’t about to start something that would have taken forever. I went back to the drawing board.

I started with the back of envelope calculations on the math this time. Given the exponential nature of the growth, I had to be very careful about the problem of randomly recreating every work of Shakespeare. For example, 26^8 was too small and trivial, where 26^10 was too big. Then there are computer science-specific parts to the problem, such as did Shakespeare use tabs or spaces? I decided to make the problem as simple as possible. I removed all of the punctuation and spaces and made everything lower case. I decided that nine characters at a time was the right-sized problem.

I went back to my friends and family with the updated project plan. Still yawns. Well, I wanted to get better with Hadoop MapReduce, and I went for it. I made one more change that I hoped would be engaging to people. I created and updated an image for every single work of Shakespeare that people could use to track progress and look in on their favorite work of Shakespeare (I was right about that, and people came back to check). This image was on a blog post announcing the project.

I started the MapReduce job and waited and checked on it every so often. The waiting is the hardest part.

No one in my family and friend circle may have cared, but I thought other geeks might like it, especially the Simpsons fans. I submitted the project to various sites’ tip lines. Maybe I’d get some coverage, I thought. Gizmodo and Fox News were the first ones to give any coverage. That was the most number of hits my blog had ever received. That was enough for me, and it showed there were some legs to the project.

The Coupon Collector problem

Then more waiting. There was another mathematical problem that I didn’t know about. There’s a problem called the Coupon Collector’s Problem. It means that initially, you will create lots of unfound nine character groups, and over time, it will take longer and longer to find unfound groups. The image above shows the problem. About halfway in, there’s a long tail of finding the remaining groups.

The Finding

The waiting stopped after about a month, and the monkeys created the first work of Shakespeare. Other works followed until every work of Shakespeare was recreated. Cool, what now?

I spent two hours submitting the project to every mainstream and tech news outlet I could think of. If you’re going to something interesting like this, you have to be a little PT Barnum and get out your megaphone. Otherwise, people won’t ever hear about the cool stuff you’re doing.

It was late that night, and I had worked all day then submitted for hours. I went to bed.

The following morning things changed. I had all these missed calls on my cell phone from international numbers with country codes I’d never seen before. They left messages of “Hi, my name is so and so from the BBC. We want to interview you about your project. Please call us back ASAP.” I was asleep, and they went to press anyway. There was my project covered in the BBC. That started the chain reaction of coverage.

There were so many articles and requests for interviews. It was all such a blur, and I had to go to work and try to field as many calls and emails as I could. It was a surreal experience seeing your name in the news for something positive. Going viral was my 15 minutes of fame, and I loved every minute of it.

The Job

Sometimes expectation and reality differ dramatically. There I was on the front of tech news and mainstream media. But I’m still the same old G at work.

My workmates would call out, “there’s Jesse on *insert site name*.” There was a least a touch of jealousy in these callouts.

If you’re a business owner and your employee is in The Wall Street Journal, you’d be at least a little excited, right? After all, one of the company’s core values was “innovative.” It turns out I received little to no fanfare from the company.

I think this really cemented a theme I saw in the company. This company wasn’t the place for me. I asked around to see what happened for other people who did something interesting at that company. They received paltry accolades relative to the effort put in. It was nothing, and the core value of innovation turned out to be just a word on a paper and written on the walls.

I was really a square peg trying to fit into this company’s round hole. It was time to find a company that appreciated and valued my talents.

The Search

I started my job search in earnest about a month after the project finished. I always had impostor syndrome (though I didn’t know it or know what that was). I thought that since I’m entirely self-taught that I could never get into the big-name companies. I decided to challenge these notions this time because the monkeys gave me that courage.

I also had to figure out how to leverage the project. I couldn’t just sit back and wait for things to come to me. I like to think of this time in my life like Apollo 11. My project was the rocket, and the notoriety was a liftoff. I wasn’t to the Moon yet. I had to orbit the Earth 1.5 more times before making my way to the Moon. During those orbits, I was getting experience in the art of technical interviews and learning how to leverage my project.

I interviewed at some big names such as Google, VMWare, and others.

Cloudera

Throughout this process, Cloudera was the company I wanted to work at, and they were the ones doing the most with Hadoop.

It turned out most of the people at Cloudera had already heard of me and the project. What was better was the company actually valued and appreciated work like that. It was such a difference from the company I was at. It gave me a glimpse into what could be if I could get over my impostor syndrome.

I joined Cloudera’s training team.

The Rest Is History?

No, wait. The rest isn’t history, and I still had to press on and find even more ways to leverage my project.

This extra work is where I had to learn how to make my own luck. I didn’t know anyone, and I was just this guy from Nevada with maybe one company name you’d recognize. Even with the project, I wasn’t a name. I spoke at my first Strata conference by talking about the Million Monkeys project.

I had to push and meet as many people as I could. Meeting people took some doing and effort. It turned out that some of them had heard of the project too. I’ve made some great friends at that conference. Much of my success now can be traced back to 2-3 people I met at that conference. Yes, it is about who you know. There’s a massive power to meeting people and being genuine. So start meeting people. I started with nothing and a quiet programmer’s demeanor. I’m still an introvert, but I came to understand the importance of personal connections. Read Never Eat Alone.

The Lessons

I see people trying to start personal projects the wrong way. They are looking for the cheapest, lowest effort, and least time-consuming route. These projects won’t go viral, and that probably isn’t your goal. Low-effort projects show and look more like HelloWorld than something to be proud of. The low effort ones I’ve seen make me question if the poor quality of the project is due to the lack of time or lack of ability. My experience is that low-effort projects are worse than doing one at all.

Make sure you have clearly laid out the goal for your project to be of any real help to you. Is your goal to get a new job? Play around and learn a new technology? Each plan has different levels of involvement and outcomes. The Million Monkeys Project was the most circuitous route to anything, but it showed I had put the time in and mastered the subject. People can see effort and mastery quickly.

People worry about putting effort into a project that goes nowhere. Who will it impress? It will impress the right person if you’re doing it right. There’s something of a “takes one to know one.” Another person who values creativity and mastery will quickly see that skill in your personal project. This person is who you’re creating your project for. I kept trying to create something that would excite my friends and family. They weren’t the people I should have targeted. I should have been focusing on the people that would appreciate a project like this.

There were different levels of mastery to the project. On the technical, there were levels of mastery that others saw. The way I used a Bloom filter or the pseudo-random number generator. Each one of these showed an attention to detail and ability. If you want to use your personal project to land a new job, this is what it needs to do. There should be no question that you not only have the skills required (e.g., Hadoop MapReduce) but there’s a more profound level that sets you apart from any competition in the interviewer’s mind.

The project touched a religious nerve that I didn’t realize existed. In a roundabout way, the project connects the evolution versus creation topic. This subject is covered more in Richard Dawkins’s books, but if something complex, like a book, can be created randomly, even more, complex things, such as amino acids and life, can evolve randomly. I came out of this project thinking that random processes mapped onto rules (physical, chemical, genetic) can create more complex outcomes than we expect.

I grew up in a religious cult that constantly told you to doubt yourself and your abilities. It’s still something I deal with, and that little voice is still in my head. In a project like this, there were all kinds of places for my self-doubt to come up. You don’t know anything about Hadoop. What if someone looks over your code and says how terrible it is? What if someone says something nasty about you? What if someone sees an error in my math or method? It turned out that most of these problems just lived in my head. As The Bard said, the internet is a vast anonymous place, and you will find saucy knaves.

Even as I write this post ten years later, I trip out that a million monkeys changed my life this much. It took effort and leverage, but I got to different levels than I ever imagined; with the right idea, talent, and hard work, you can do it too. I hope this post and honest sharing of my experience motivates others to try something stupid and see how it goes.

Related Posts

The Difference Between Learning and Doing

Blog Summary: (AI Summaries by Summarizes)There are several types of learning videos: hype, low effort, novice, and professional.It is important to avoid hype, low-effort, and

The Data Discovery Team

Blog Summary: (AI Summaries by Summarizes)The concept of a “data discovery team” is introduced, which focuses on searching for data in an enterprise data reality.Data