MongoDB Schema Design at Scale

I had the recent opportunity to present a talk at MongoDB Seattle on Schema Design at Scale. It's basically a short case study on what steps the MongoDB Monitoring Service (MMS) folks did to evolve their schema, along with some quantitative performance comparisons between different schemas. Given that one of my most widely read blog posts is still MongoDB's Write Lock, I decided that my blog readers would also be interested in the quantitative comparison as well.

Python and MongoDB Travel Plans

It's been a little longer than normal between articles, so I wanted to let everyone know what I've been up to lately, and hope that I'll be able to see some of you at some upcoming events. So here's a summary of the things I'll be involved with over the next couple of months:

MongoDB Seattle

I have the privilege of speaking at MongoDB Seattle tomorrow, September 14, 2012. My talk is on "Schema Design at Scale", and wll have some insights based on experience 10gen gained when developing the MongoDB Monitoring Service. I'll even have some cool graphs based on experiments I did to illustrate the schema design tips I'll be giving.

MongoDB for Developers Online Training

I'm offering an 4-week online training class starting this Monday at 8pm EDT. I'll be delivering the training in 2 parts each week: a "lecture" portion on Mondays and an unstructured "office hours" on Tuesdays where I'll be available for questions. If you're interested, you can sign up by clicking the following link: Register for MongoDB for Developers. Right now the class is quite small, so I'll be able to spend a lot of time answering questions and interacting with students.

Python Training in Beijing

I'll be in Beijing, China to deliver an onsite "Fast Track to Python" class from October 5-10. The actual training is only on the 8-10, so if anyone's interested in getting together before that, I'd love to hang out in the People's Republic.

Big Dive and MongoTorino

I'm also happy to announce that I'm one of the featured instructors for the Big Dive training event in Turin, Italy this October. I'll be teaching a 2-day class on MongoDB and a 1-day class on Gevent, so if you're in northern Italy, I'd love to see you there! I'll also be at MongoTorino on October 19th, but the webpage for that doesn't seem to be updated yet. I'll be in Turin from October 13-20 if you want to come to either event, or just hang out over a bottle of wine.

PyCarolinas

To cap off my world tour I'm happy to be able to speak on Gevent and Socket.io at PyCarolinas on October 21st, so I'd love to see any of you there. Sadly I'll have to miss the 1st day of PyCarolinas on my way back from Italy.

And that's about it (and that's enough!). I'd love to see any of you who are in any of the cities mentioned above when I'm passing through, so feel free to contact me on twitter (@rick446), email (rick at arborian.com), or in the comments below!

Using ZeroMQ devices to support complex network topologies

Continuing in my ZeroMQ series, today I'd like to look at ZeroMQ "devices" and how you can integrate ZeroMQ with Gevent so you can combine the easy networking topologies of ZeroMQ with the cooperative multitasking of ZeroMQ. If you're just getting started with ZeroMQ, you might want to check out the following articles:

• Introduction to ZeroMQ
• ZeroMQ flow control and other options

And if you want some background on Gevent, you might want to check out that series at the following links:

• Introduction to Gevent
• Gevent, Threads, and Benchmarks
• Gevent and Greenlets
• Greening the Python Standard Library with Gevent
• Building TCP Servers with Gevent
• Building Web Applications with Gevent's WSGI Server

Once you're caught up, let's get started...

ZeroMQ flow control and other options

In a previous post, I provided an introduction to ZeroMQ. Continuing along with ZeroMQ, today I'd like to take a look at how you manage various "socket options" in ZeroMQ, particularly when it comes to flow control. If you've never used ZeroMQ, I recommend reading my previous post first. Once you're caught up, let's get started...

Python and MongoDB Online Training

One thing that I've been planning on offering as part of Arborian's services is online training. For the inaugural class, I've decided to offer a MongoDB and Python developer training class. Read on to hear more about this class...

Distributed Systems with ZeroMQ

Departing a bit from my current series on gevent and Python, today I want to take a look at a different networking technology that's been gaining traction: ZeroMQ. So without further ado, let's jump right in...

Building Web Applications with Gevent's WSGI Server

Continuing on in my series on gevent and Python, this article discusses how to use gevent to power your Python WSGI web applications. If you're just getting started with Gevent, you might want to read the previous articles in this series first:

• Introduction to Gevent
• Gevent, Threads, and Benchmarks
• Gevent and Greenlets
• Greening the Python Standard Library with Gevent
• Building TCP Servers with Gevent

And now that you're all caught up, let's jump into gevent's WSGI support...

Building TCP Servers with Gevent

Continuing on in my series on gevent and Python, this article discusses how to build TCP servers using the infrastructure provided with gevent. If you're just getting started with Gevent, you might want to read the previous articles in this series first:

• Introduction to Gevent
• Gevent, Threads, and Benchmarks
• Gevent and Greenlets
• Greening the Python Standard Library with Gevent

And now that you're all caught up, let's jump into gevent servers...

Greening the Python standard library with gevent

Continuing on in my series on gevent and Python, this article deals with what you need to do when want to use the Python standard library with gevent, showing how gevent provides a way to monkey-patch the standard library to make it compatible with gevent. If you're just getting started with Gevent, you might want to read the previous articles in this series first:

• Introduction to Gevent
• Gevent, Threads, and Benchmarks
• Gevent and Greenlets

And now that you're all caught up, let's get started with gevent...

Last Chance for Early Bird Atlanta MongoDB Training

As announced earlier, I'm offering some MongoDB training classes in the first week of August in Atlanta. So why am I cluttering up your inbox today? Mainly because the early bird price of $400 per day ends at 11:59 pm (EDT) tonight, after which the price goes up to $500.

Gevent and Greenlets

Continuing on in my series on gevent and Python, this article gets down into practical details, showing you how to install gevent and get started with basic greenlet operations. If you're just getting started with Gevent, you might want to read the previous articles in this series first:

• Introduction to Gevent
• Gevent, Threads, and Benchmarks

And now that you're all caught up, let's get started with gevent...

Two MongoDB books (one now, one early 2013)

Today I'm happy to make two book-related announcements.

MongoDB with Python and Ming Book

First off, I've collected, edited, and expanded upon the the MongoDB and Python series I've been working on at Just a Little Python and released it as an eBook. MongoDB with Python and Ming is now available on Lulu as an epub and on Amazon (at MongoDB with Python and Ming). In the book, I cover the following topics:

• Using PyMongo, from installing & basic queries to the new aggregation framework
• Overview of tuning your application for the best performance under MongoDB including discussions of replication and sharding
• Using the Ming toolkit to enforce your schema in MongoDB
• Using Ming's object-document mapper (ODM) to raise the abstraction level of your MongoDB programming
• Various tips, tricks, and goodies with Ming including Mongo-in-Memory, extending the ODM, and schema migrations

MongoDB Applied Design Patterns

The second announcement is that I've agreed to write another book for O'Reilly, tentatively titled MongoDB Design Patterns. That one's not done yet, and it's still early in the authoring / publication process, but it should be available early 2013. Sadly I don't have a nice product page link to share with you yet, but I will leave you with a summary of the topics I plan to cover:

• Part 1: MongoDB Design Patterns
  • Embedding versus referencing documents
  • Using polymorphic schemas
  • Using complex atomic updates
  • Optimistic updates with compensation (in lieu of multi-document transactions)
• Part 2: Use Cases
  • Operational intelligence / real-time analytics
  • E-Commerce
  • Content management systems
  • Online advertising networks
  • Social Networking
  • Online Gaming

I hope that's enough to whet your appetite. Of course, the book's not finished, so if you have any other topics you'd love to see covered in such a book, I'd love to hear about it in the comments!

Gevent, Threads, and Benchmarks

In a previous post, I gave an introduction to gevent to show some of the benefits your application might get from using gevent greenlets instead of threads. Some people, however, took issue with my benchmark code, saying that the threaded example was contrived. In this post, I'll try to answer some of the objections.

(It actually turns out that there was a bug in the version of ab I was using to test, as well, so I re-ran the tests from the previous post, too.)

Introduction to Gevent

In a previous post, I described how to build a web chat server with socket.io and gevent. If you're trying to actually learn gevent, socket.io, however, it's probably not the best place to start. So I figured I'd write this post and provide an overview of gevent.

[Update 2012-07-24] In response to some criticisms of the micro-benchmarks in this post, I reworked the benchmarks and wrote an updated gevent and threads post. Make sure you read that one for more perspective on greenlets vs. threads.

MongoDB Training Classes in Atlanta

As promised earlier, we have now officially launched MongoDB and Python training classes. The first batch of classes will be offered between August 7-10, 2012, with early bird registration closing on July 31. We'd love to have you attend one or more classes, so read on for more info!

Realtime Web Chat with Socket.io and Gevent

In a previous post, I described how to build a realtime analytics graph with socket.io. That was nearly a year ago, however, and both socket.io and the Python library gevent-socketio have both undergone significant changes. In this post, then, I'll use the latest versions of each to build a web chat server.

Rick's Resources Blog

Since starting my own consulting business (and actually, quite a while before that), I've run across various resources, from hosting, to web apps, to actual, tangible products I've bought, that others might find interesting. In order let those who want to hear about such things hear about them, while keeping this blog more technically focused, I've created a new blog, Rick's Resources, that will cover the various things I've found interesting over the years.

Hopefully this blog will be useful to those of you who are interested, and as I mentioned, for those of you who aren't, just stick around here and we'll get back to Python momentarily. For those who are interested, my inaugural post on the resources blog is The Five Minute DreamHost Blog, where I walk you through setting up a blog on DreamHost. (It's not technically five minutes since you have to wait for some emails, but it's a reasonable facsimile thereof, I hope.)

And now, back to your regularly scheduled Pythonning....

Multi-Master Replication in MongoDB

Since starting my own consulting business, I've had the opportunity to work with lots of interesting technologies. Today I wanted to tell you about some interesting technology I got to develop for MongoDB: multi-master replication.

Relational Mapping with MongoDB, Ming, and Python

Continuing on in my series on MongoDB and Python, this article delves into the object-document mapper (ODM) included with the Python MongoDB toolkit Ming. If you're just getting started with MongoDB, or with Ming, you might want to read the previous articles in the series first:

• Getting Started with MongoDB and Python
• Moving Along With PyMongo
• GridFS: The MongoDB Filesystem
• Aggregation in MongoDB (Part 1)
• MongoDB's New Aggregation Framework
• Schema Maintenance with Ming and MongoDB
• Declarative Schemas for MongoDB in Python Using Ming

And now that you're all caught up, let's take a look at what makes up an object document mapper, anyway...

Declarative Schemas for MongoDB in Python using Ming

Continuing on in my series on MongoDB and Python, this article continues to explore the Python MongoDB toolkit Ming, this time looking at an alternative, declarative syntax for schema definition. If you're just getting started with MongoDB, you might want to read the previous articles in the series first:

• Getting Started with MongoDB and Python
• Moving Along With PyMongo
• GridFS: The MongoDB Filesystem
• Aggregation in MongoDB (Part 1)
• MongoDB's New Aggregation Framework
• Schema Maintenance with Ming and MongoDB

And now that you're all caught up, let's take a look at that declarative syntax....

Schema Maintenance with Ming and MongoDB

Continuing on in my series on MongoDB and Python, this article introduces the Python MongoDB toolkit Ming and what it can do to simplify your MongoDB code and ease maintenance. If you're just getting started with MongoDB, you might want to read the previous articles in the series first:

• Getting Started with MongoDB and Python
• Moving Along With PyMongo
• GridFS: The MongoDB Filesystem
• Aggregation in MongoDB (Part 1)
• MongoDB's New Aggregation Framework

And now that you're all caught up, let's jump right in with Ming....

Python and MongoDB Training Classes, Anyone?

Departing just a bit from my current series on how to effectively use MongoDB from Python, today I wanted to let you, my blog readers, know that as of this past February I left SourceForge to start a new python and mongodb consulting business, Arborian Consulting. Why is this (hopefully) relevant to you? At this past PyCon, I had the opportunity to lead a python and mongodb tutorial and had a great time, but realized that the three hour format is just not enough time. So I'm hoping to start some full- or multi-day training classes, but I need your input to know what and where to offer them.

Using MongoDB's New Aggregation Framework in Python (MongoDB Aggregation Part 2)

Continuing on in my series on MongoDB and Python, this article will explore the new aggregation framework introduced in MongoDB 2.1. If you're just getting started with MongoDB, you might want to read the previous articles in the series first:

• Getting Started with MongoDB and Python
• Moving Along With PyMongo
• GridFS: The MongoDB Filesystem
• Aggregation in MongoDB (Part 1)

And now that you're all caught up, let's jump right in....

Aggregation in MongoDB (Part 1)

In some previous posts on mongodb and python, pymongo, and gridfs, I introduced the NoSQL database MongoDB how to use it from Python, and how to use it to store large (more than 16 MB) files in it. Here, I'll be showing you a few of the features that the current (2.0) version of MongoDB includes for performing aggregation. In a future post, I'll give you a peek into the new aggregation framework included in MongoDB version 2.1.

Review: Web2Py Application Development Cookbook

Changing things up a bit, today I have a special treat, as Jay Kelkar, an avid Pythonista and fellow member of the Python Atlanta user group, has agreed to write a guest post reviewing the web2py Application Development Cookbook. If you give him half a chance, Jay will be happy to expound upon the benefits of Web2Py, so this review seemed a perfect fit for him. So without further introduction, here's Jay's review (full disclosure: the Amazon links below are affiliate links):

GridFS: The MongoDB Filesystem

In some previous posts on mongodb and python and pymongo, I introduced the NoSQL database MongoDB and how you can use it from Python. This post goes beyond the basics of MongoDB and pymongo to give you a taste for MongoDB's take on filesystems, GridFS.

Moving along with PyMongo

In a previous post, I introduced MongoDB and its Python driver pymongo. Here, I continue in the same vein, describing in more detail how you can become productive using pymongo.

Getting Started with MongoDB and Python

If you've been following this blog for a while, you've seen me mention MongoDB more than once. One exciting thing for me is that I'll be co-teaching a tutorial at PyCon this year on Python and MongoDB that will cover MongoDB, PyMongo, and Ming. So to hopefully whet your appetite for learning more at the tutorial, I thought I'd write a few posts covering MongoDB, PyMongo, and Ming from a beginner's perspective.