The last two years (2010-2011) I spent working at Quizlet were an incredible learning experience.
Like I did in Jan 2010, I wanted to reflect on some of the technologies I learned and things I did over the last 2 years…
- Learned tons about scalability! Scaled Quizlet to serve 11 million visits per months (60 million pageviews, 5 million uniques), serving mostly UGC content with an average page response time of < 50ms.
- Got very comfortable with understanding and tweaking advanced configuration options of Apache, PHP, & MySQL for desired functionality and scalability requirements
- Went from a command line beginner to feeling very comfortable in Linux and Solaris, frequently writing bash/shell scripts for common sysadmin tasks, and trying to automate as much as possible.
- Worked with large data sets: our database had several hundred million rows and was in the dozens of gigabytes in size.
- Played a dev-ops role and helped transition all production servers to use Puppet for configuration management. Focused on having a process for all server related changes and created the ability to have over a dozen different servers recreate-able in very little time.
- Learned how to make MySQL fast on a high-trafficked and large database, do replication and have a quick recovery plan. Dealt with configuring databases, designing schemas, and optimizing SQL statements. Also managed our relationship with Percona’s consultants (leading MySQL experts) and learned from them.
- Automated backups and ensured we had quick access to them in the right places plus securely storing them off-site to Amazon S3 for safety. We had people relying on their data being accessible for studying and thankfully were able to recover quickly from incidents without losing any data.
- Sphinx Search is a very powerful open source search server good for indexing and quickly searching over MySQL. I learned how to configure it for our purposes and use its various features to power search on millions of rows of Quizlet content in a couple hundred milliseconds.
- Implemented and used several performance monitoring tools to keep tabs on a heavily trafficked website: Nagios, Munin, New Relic, etc.
- Learned all about (and implemented) email deliverability best practices: SPF records, DKIM, rDNS, Feedback Loops, IP address reputation, etc.
- Used several cloud service providers (Joyent, Slicehost, Amazon AWS). Setup reverse proxy CDNs for optimal performance of static files.
- Major focus on serious performance optimizations across entire stack. Spent time obsessing over how to get an extra 20ms out of a page load from our backend while keeping code clean. Resulted in an average page load of < 50ms across Quizlet.
- I already knew PHP well, but I’d say now I’m more of an expert at properly structuring object-oriented PHP5 on a large codebase.
- Learned all about and then implemented website localization & internationalization (i18n) for Quizlet, helping create a framework for user-generated translation submissions that launched in 7 languages.
- Got deep into Unicode and character encodings and implemented a language detection system in PHP based on n-grams and character encodings.
- Spent time optimizing with AdSense - A/B tested with different ad placement, targeting, and IAB ad sizes for better inventory.
- Wrote a very fast A/B testing framework and used it to optimize our upgrade page for better monetization
- Used the Stripe API to accept credit card payments (replacing our previous PayPal solution).
- Lots of cross-browser CSS – making things look good in browsers both old and new.
- Helped lead the development team with best practices and good processes for coding and communication.
- Became really good at Git and transitioned our team from SVN/Trac to Git/GitHub. Used git submodules to manage many of our dependancies and became a better open source citizen by submitting pull requests and bug reports in more projects.
- Did a lot of code reviewing on trac and GitHub with peers and new hires / interns.
- Created a safer deployment process which first runs unit tests, PHP Lint, and other sanity checks before deploying.
- Started daily standups with the eng team to keep everyone up-to-date and on track.
- Along with development work, I was doing a lot of Product Management: feature triage and prioritization, tracking roadmaps, deciding among tradeoffs, etc.
- Huge focus on the user interface: tried to make every new feature high quality and easy to use. Spent a lot of time redoing the user flow of features that already existed to improve their ease of use. Removed clutter and made features that “just work”.
- Used Photoshop to work on visual improvements — designing or redesigning almost all the main pages and many shared UI components on Quizlet.
- Brainstormed and implemented game mechanics and other ways to make studying on Quizlet more fun and social.
- Focused heavily on recruiting other developers, helped create a process for finding potential candidates and putting them through a funnel. Did a lot of technical interviewing by phone/Skype and in-person. Helped build a great team of full-time people and interns.
- Team building. Helped manage and guide new employees, getting them up to speed and supporting their projects.
- Did a fair amount of biz dev as well. Negotiated deals for licensing software, worked out details of contracts. Talked with potential partners.
- Managed projects that other developers were working on, pushing for both code quality, fast completion, and a high quality user experience.
- Wrote a bunch of Quizlet blog posts about product announcements.
I’m sure I’m missing lots of stuff, and I should probably take even more time to write down some higher level lessons learned about startups. But for now this will do: startups are awesome and the best way to learn a lot of stuff really quickly by doing it. And if you’re lucky like I’ve been, it will be with an awesome team working on a great product used by tons of people.
I’ve recently moved onto a new opportunity that I’m very excited about. But I’m really glad to have been at Quizlet the last two years and I look forward to seeing it continuing to succeed widely!
(P.S., they’re hiring…)