Comment Load (0.000391) SELECT count(DISTINCT `comments`.id) AS count_all FROM `comments` WHERE (`comments`.thingy_id = 545)
User Load (0.000441) SELECT * FROM `users` WHERE (`users`.`id` = 188)
Comment Load (0.000364) SELECT count(DISTINCT `comments`.id) AS count_all FROM `comments` WHERE (`comments`.thingy_id = 99)
User Load (0.000424) SELECT * FROM `users` WHERE (`users`.id' = 6)
Comment Load (0.000358) SELECT count(DISTINCT `comments`.id) AS count_all FROM `comments` WHERE (`comments`.thingy_id = 85)
User Load (0.000408) SELECT * FROM `users` WHERE (`users`.`id` = 6)
Comment Load (0.000269) SELECT count(DISTINCT `comments`.id) AS count_all FROM `comments` WHERE (`comments`.thingy_id = 18)
The first line is reasonable - that’s just getting the list of stories. But look what happens when rendering the view: for each story displayed, there are two queries sent to the database. It’s no wonder that generating a long list of stories takes a long time when you are issuing two queries for each story.
A quick look at the code tells us what is happening: the expression “s.submitter.display_name” gets the submitter of the story (which is in the Story model as a belongs_to attribute) and extracts the display_name from it. But that requires a database retrieval (the query against the users table). And the expression “s.comments.count” requires the query against the comments table.
Happily, there’s a number of techniques that can be used to eliminate these extra queries. These include:
1. Use of the “include:” directive when fetching the data. For example, change the above controller code as follows:
@stories = Story.find(:all, :conditions => "some condition", :include => [:submitter])
and all of the users table queries above are replaced with a single query against the users table.
(Note: once upon a time, Rails could be inefficient in its implementation of the :include directive. Happily, this is fixed in Rails 2.0, so there is no longer any reason to avoid :include.)
2. Store frequently used data in the parent table. Admittedly, this is a violation of database normalization principles. Under these principles, we never want to duplicate data in the database. But in some cases, the performance gains that we get from denormalization are worth the potential problems with having data get out of sync. Especially when the data in question is not crucial.
In this case, we added a comment_count field to the stories table. We use after_create and before_destroy actions on the Comment model to keep the comment_count field updated - when we create a new comment, we increment the comment_count value for its parent Story. (We made a conscious decision that, while we could be clever and manage to keep comment_count accurate, we can live if it is incorrect for some stories.) After that, we can use the comment_count field in Story instead of comments.count, thus avoiding the query required to get the count of comments for a story. The resulting view is as follows:
- <%= s.title %> by <%= s.submitter.display_name %>
with <%= s.comment_count %> comments
<% @stories.each do |s| %>
<% end %>
In conjunction with the controller change noted in #1, the query log now looks like this:
Story Load (0.006564) SELECT * FROM `stories` WHERE ( some condition )
User Load (0.000943) SELECT * FROM `users` WHERE (`users`.id IN ('127','193','6','249','216','239','91','240','196','37','93','176', '188','244','235','136'))
Rendering stories/show
What used to take nine queries (and could easily grow to more as the number of stories increased) now takes two.
3. In some cases, you can find the data you need in the model’s attributes. If you can thus avoid retrieving via a belongs_to or has_X relationship, then do so
There isn’t an example of this in the sample code. But suppose that we did not want to display the actual submitter of a story in the above example, but we want to display the story differently if it is submitted by the person viewing the page. (In our case, the User model of the viewer of the page is always contained in the @user variable.) We could use any of the following methods to make this test:
if @user == story.submitter # bad! Requires database retrieval of submitter
if @user.id == story.submitter.id # bad! Also requires retrieval.
if @user.id == story.submitter_id # good! No retrieval required.
Even if you use the :include directive to get the submitter, the first two methods require at least one database retrieval for the page.
Another common case involves a test for the existence of data related to a model via a belongs_to. For example, suppose we want to test whether a story has a submitter at all. Here are two ways of doing it:
if story.submitter # bad - retrieves submitter if present
if story.submitter_id # good - no retrieval required
In summary, even though Rails allows you to abstract away database accesses, if you want your application to perform, you need to be aware of how it is using the database. An excellent method to be aware is to review the Rails log while you are developing your application, paying particular attention to repeating instances of the same query (as in the example above, where the queries against the users and comments tables repeat). In fact, you should frequently review the Rails log whenever you are building an application that has to perform well.
Add to Mixx!
* Filed under: Uncategorized |
* 4 comments
*
OpenID for the Rest of the World
Posted by Jason on October 6th, 2008
Through the history of the Internet, there are inflection points where a new technology hits “the masses” and really takes off. Almost universally, the common thread in a quick expansion and adoption of a technology is a major breakthrough—not of pure technology—but a breakthrough in design. Specifically, a major breakthrough in ease of use.
Before Google created a dead simple search interface, the vast majority of people used browsing as their primary method of navigating the Web. Before Mosaic made visual browsing simple, text-based navigation through Archie and Veronica were the norm. Apple made consuming music online easy with iTunes and the iPod. AOL made “getting online” easy for Mom and Dad. Most recently, YouTube has made Internet video sharing easy for every high schooler in the world.
The history of the Internet is littered with technologies that were fabulous, but never quite made it mainstream. At Mixx, we’re so in love with the idea of OpenID that we want to do our part to make sure OpenID isn’t relegated to the history books in the “might have been” chapter.
For those new to OpenID, the idea is simple. After creating an account with an OpenID provider, you are given a unique identifier (a custom URL) that you in turn use to log in to and register with sites that support OpenID (”consumers,” as they call them). The goal of OpenID is two-fold. First, you, dear user, can create your OpenID with an account provider you trust and, in turn, extend that trust to sites and services of your choosing. No longer do you need to hand over an email address and a password to every new (potentially fly-by-night) web application that comes along. Second, since you typically have a single OpenID, you won’t have to try and remember which username and password you used on this site or that site. You simply remember that you used OpenID!
OpenID, since it’s creation, has made great inroads across the web. Some of the largest service providers out there (AOL, Yahoo!, etc.) have become OpenID providers, giving OpenIDs to millions and millions of people. Every week, at least one site launches with OpenID support or an existing site adds OpenID functionality to their login or registration process. The trouble, as we’ve observed, is that while OpenID login and registration is being rapidly added to sites, its presentation lacks the design necessary for Mom and Dad to grasp OpenID’s power.
Today, we’re proud to launch our take on OpenID registration and login. If you swing by the Login or Registration pages and you’ll see something new. For existing Mixxers, the login screen will be contextual to your current method of login (either username/password or OpenID).
For new Mixxers, you can register with an AOL, Yahoo!, or Facebook account. Standard OpenID registration is also still available. If, by some twist of fate, you don’t have an account with any of the third-party services we currently support, you can still register with your email address. While Facebook isn’t OpenID per-se, both AOL and Yahoo! registration and login utilizes OpenID under the covers without asking users for their OpenID URL. In AOL’s case, we ask for your AOL or AIM account name and shuffle you off to the appropriate login page. Yahoo! login and registration is even simpler—click the big honkin’ “Login/Register with your Yahoo! ID” button and we take care of the rest.
We’ve gone one step further and added a great feature to login: we keep track of the last method of login you use and redesign the page based on that. So, if you use Facebook to login, you’ll be presented with a large Facebook icon and button. No need to hunt and click through our login options!
The last piece to the puzzle is managing your accounts. Navigate to your Account Settings and click the new “Accounts” tab. From there, you can add and remove your third-party accounts at will. By linking up your various accounts across the web with your Mixx account, you can use any of those methods to login to Mixx. This is a great first step toward the goal of interoperability between Mixx and your favorite web sites and applications.
We put a good deal of work into the new registration and login experience and we hope you find the improvements useful and, above all else, easy. As always, we appreciate your feedback and look forward to hearing what you have to say!
~ Jason (and the rest of the Mixx team)
Add to Mixx!
* Filed under: User Experience |
* 5 comments
*
Hiding content, accessibility, and the onload problem
Posted by Jason on September 15th, 2008
Since I joined the Mixx team (a year and some change ago) and began cranking out the HTML, CSS, and JavaScript that you see and use every day, I made it a point to build features out with accessibility in mind. Mixx has a great variety of users comprising all races, colors, creeds, and capabilities. It was (and remains to this day) important to us that we provide a great experience for our users while not leaving anyone out in the cold.
As most of you know, there’s a ton of interactivity on Mixx—voting, reporting, submitting, interacting with YourMixx—the list goes on. Each of these elements involves a different interaction with the application and every one of them involves manipulating content on-screen using a combination of JavaScript and CSS.
In an effort to accommodate users (or, more directly, their browser of choice) who have JavaScript turned off, we take the approach of displaying all content by default and then using JavaScript to hide the appropriate elements. The easiest to observe example of this is on the permalink pages. With recent redux of the permalinks, we added some new “tabs” (for lack of a better descriptor) below the entry information and above the comments: Activity, About this site, and Related.
When browsing to a permalink page, depending on how zippy your Internet connection is, you may notice that the three tabs are expaned and stacked on top of one another initially. After a brief pause, they’ll snap away and they may then be accessed by clicking on their respective button. This is, at the most basic, the situation described above. Page content loads. JavaScript does its job and hides the appropriate pieces.
In a perfect world, this would happen instantaneously and no one would notice. In the real world, Internet connections are variable, web servers hiccup, and external resources (like Google Analytics) have the side-effect of stalling firing of local scripts. The last point there is the one of concern to our discussion. Steve Souders, in his excellent book High Performance Web Sites, goes into great detail about why this stalling happens. Check out Chapter 6 on “Problems with Scripts.”
So what do we do? Let’s first take a look at how Mixx currently works.
As we’ve observed, our HTML and CSS style everything on the page and display elements by default. Once everything is loaded up, the Mixx JavaScript (using jQuery, a subject for another post) uses $().ready() to fire off our onload events. The appropriate bits hide away and we’re done. This is great * Posting
* Pengaturan
* Tata Letak
* Lihat Blog
Mixx’s Engine Room - Stoking the fires at Mixx
Return to Mixx
Performance and the Rails log
Posted by Joe on December 8th, 2008
One of the great strengths of Ruby on Rails is that it abstracts away database access so that you don’t have to worry about SQL when writing your application. Unfortunately, this hides database accesses from developers, which can lead to serious performance problems. The solution to this problem is no further away then the Rails log.
Consider this simple Mixx-ish application, which displays a list of stories:
The controller:
@stories = Story.find(:all, :conditions => "some condition")
The view:
- "<%= s.title %>" by <%= s.submitter.display_name %>
with <%= s.comments.count %> comments
<% @stories.each do |s| %>
<% end %>
(I should note that Jason and Doug create much better markup than this. This is what you get when a backend guy like me writes code for the purpose of demonstrating something - we wouldn’t use anything this ugly in Mixx production code.)
Seems pretty reasonable, right? But as the number of stories on the list grows, performance quickly goes bad. Let’s find out why.
First, sample output from a run of this code:
* “Obama wins!” by joe with 3 comments
* “Storms rage everywhere” by julie with 0 comments
* “Mixx launches” by chris with 3 comments
* “Eggplant farming” by chris with 1 comments
Next, take a look at the Rails log when this list is generated. This can be found in the application directory in logs/development.log:
Story Load (0.000911) SELECT stories.* FROM stories
Rendering stories/show
User Load (0.000601) SELECT * FROM `users` WHERE (`users`.`id` = 176)
Comment Load (0.000391) SELECT count(DISTINCT `comments`.id) AS count_all FROM `comments` WHERE (`comments`.thingy_id = 545)
User Load (0.000441) SELECT * FROM `users` WHERE (`users`.`id` = 188)
Comment Load (0.000364) SELECT count(DISTINCT `comments`.id) AS count_all FROM `comments` WHERE (`comments`.thingy_id = 99)
User Load (0.000424) SELECT * FROM `users` WHERE (`users`.id' = 6)
Comment Load (0.000358) SELECT count(DISTINCT `comments`.id) AS count_all FROM `comments` WHERE (`comments`.thingy_id = 85)
User Load (0.000408) SELECT * FROM `users` WHERE (`users`.`id` = 6)
Comment Load (0.000269) SELECT count(DISTINCT `comments`.id) AS count_all FROM `comments` WHERE (`comments`.thingy_id = 18)
The first line is reasonable - that’s just getting the list of stories. But look what happens when rendering the view: for each story displayed, there are two queries sent to the database. It’s no wonder that generating a long list of stories takes a long time when you are issuing two queries for each story.
A quick look at the code tells us what is happening: the expression “s.submitter.display_name” gets the submitter of the story (which is in the Story model as a belongs_to attribute) and extracts the display_name from it. But that requires a database retrieval (the query against the users table). And the expression “s.comments.count” requires the query against the comments table.
Happily, there’s a number of techniques that can be used to eliminate these extra queries. These include:
1. Use of the “include:” directive when fetching the data. For example, change the above controller code as follows:
@stories = Story.find(:all, :conditions => "some condition", :include => [:submitter])
and all of the users table queries above are replaced with a single query against the users table.
(Note: once upon a time, Rails could be inefficient in its implementation of the :include directive. Happily, this is fixed in Rails 2.0, so there is no longer any reason to avoid :include.)
2. Store frequently used data in the parent table. Admittedly, this is a violation of database normalization principles. Under these principles, we never want to duplicate data in the database. But in some cases, the performance gains that we get from denormalization are worth the potential problems with having data get out of sync. Especially when the data in question is not crucial.
In this case, we added a comment_count field to the stories table. We use after_create and before_destroy actions on the Comment model to keep the comment_count field updated - when we create a new comment, we increment the comment_count value for its parent Story. (We made a conscious decision that, while we could be clever and manage to keep comment_count accurate, we can live if it is incorrect for some stories.) After that, we can use the comment_count field in Story instead of comments.count, thus avoiding the query required to get the count of comments for a story. The resulting view is as follows:
- <%= s.title %> by <%= s.submitter.display_name %>
with <%= s.comment_count %> comments
<% @stories.each do |s| %>
<% end %>
In conjunction with the controller change noted in #1, the query log now looks like this:
Story Load (0.006564) SELECT * FROM `stories` WHERE ( some condition )
User Load (0.000943) SELECT * FROM `users` WHERE (`users`.id IN ('127','193','6','249','216','239','91','240','196','37','93','176', '188','244','235','136'))
Rendering stories/show
What used to take nine queries (and could easily grow to more as the number of stories increased) now takes two.
3. In some cases, you can find the data you need in the model’s attributes. If you can thus avoid retrieving via a belongs_to or has_X relationship, then do so
There isn’t an example of this in the sample code. But suppose that we did not want to display the actual submitter of a story in the above example, but we want to display the story differently if it is submitted by the person viewing the page. (In our case, the User model of the viewer of the page is always contained in the @user variable.) We could use any of the following methods to make this test:
if @user == story.submitter # bad! Requires database retrieval of submitter
if @user.id == story.submitter.id # bad! Also requires retrieval.
if @user.id == story.submitter_id # good! No retrieval required.
Even if you use the :include directive to get the submitter, the first two methods require at least one database retrieval for the page.
Another common case involves a test for the existence of data related to a model via a belongs_to. For example, suppose we want to test whether a story has a submitter at all. Here are two ways of doing it:
if story.submitter # bad - retrieves submitter if present
if story.submitter_id # good - no retrieval required
In summary, even though Rails allows you to abstract away database accesses, if you want your application to perform, you need to be aware of how it is using the database. An excellent method to be aware is to review the Rails log while you are developing your application, paying particular attention to repeating instances of the same query (as in the example above, where the queries against the users and comments tables repeat). In fact, you should frequently review the Rails log whenever you are building an application that has to perform well.
Add to Mixx!
* Filed under: Uncategorized |
* 4 comments
*
OpenID for the Rest of the World
Posted by Jason on October 6th, 2008
Through the history of the Internet, there are inflection points where a new technology hits “the masses” and really takes off. Almost universally, the common thread in a quick expansion and adoption of a technology is a major breakthrough—not of pure technology—but a breakthrough in design. Specifically, a major breakthrough in ease of use.
Before Google created a dead simple search interface, the vast majority of people used browsing as their primary method of navigating the Web. Before Mosaic made visual browsing simple, text-based navigation through Archie and Veronica were the norm. Apple made consuming music online easy with iTunes and the iPod. AOL made “getting online” easy for Mom and Dad. Most recently, YouTube has made Internet video sharing easy for every high schooler in the world.
The history of the Internet is littered with technologies that were fabulous, but never quite made it mainstream. At Mixx, we’re so in love with the idea of OpenID that we want to do our part to make sure OpenID isn’t relegated to the history books in the “might have been” chapter.
For those new to OpenID, the idea is simple. After creating an account with an OpenID provider, you are given a unique identifier (a custom URL) that you in turn use to log in to and register with sites that support OpenID (”consumers,” as they call them). The goal of OpenID is two-fold. First, you, dear user, can create your OpenID with an account provider you trust and, in turn, extend that trust to sites and services of your choosing. No longer do you need to hand over an email address and a password to every new (potentially fly-by-night) web application that comes along. Second, since you typically have a single OpenID, you won’t have to try and remember which username and password you used on this site or that site. You simply remember that you used OpenID!
OpenID, since it’s creation, has made great inroads across the web. Some of the largest service providers out there (AOL, Yahoo!, etc.) have become OpenID providers, giving OpenIDs to millions and millions of people. Every week, at least one site launches with OpenID support or an existing site adds OpenID functionality to their login or registration process. The trouble, as we’ve observed, is that while OpenID login and registration is being rapidly added to sites, its presentation lacks the design necessary for Mom and Dad to grasp OpenID’s power.
Today, we’re proud to launch our take on OpenID registration and login. If you swing by the Login or Registration pages and you’ll see something new. For existing Mixxers, the login screen will be contextual to your current method of login (either username/password or OpenID).
For new Mixxers, you can register with an AOL, Yahoo!, or Facebook account. Standard OpenID registration is also still available. If, by some twist of fate, you don’t have an account with any of the third-party services we currently support, you can still register with your email address. While Facebook isn’t OpenID per-se, both AOL and Yahoo! registration and login utilizes OpenID under the covers without asking users for their OpenID URL. In AOL’s case, we ask for your AOL or AIM account name and shuffle you off to the appropriate login page. Yahoo! login and registration is even simpler—click the big honkin’ “Login/Register with your Yahoo! ID” button and we take care of the rest.
We’ve gone one step further and added a great feature to login: we keep track of the last method of login you use and redesign the page based on that. So, if you use Facebook to login, you’ll be presented with a large Facebook icon and button. No need to hunt and click through our login options!
The last piece to the puzzle is managing your accounts. Navigate to your Account Settings and click the new “Accounts” tab. From there, you can add and remove your third-party accounts at will. By linking up your various accounts across the web with your Mixx account, you can use any of those methods to login to Mixx. This is a great first step toward the goal of interoperability between Mixx and your favorite web sites and applications.
We put a good deal of work into the new registration and login experience and we hope you find the improvements useful and, above all else, easy. As always, we appreciate your feedback and look forward to hearing what you have to say!
~ Jason (and the rest of the Mixx team)
Add to Mixx!
* Filed under: User Experience |
* 5 comments
*
Hiding content, accessibility, and the onload problem
Posted by Jason on September 15th, 2008
Since I joined the Mixx team (a year and some change ago) and began cranking out the HTML, CSS, and JavaScript that you see and use every day, I made it a point to build features out with accessibility in mind. Mixx has a great variety of users comprising all races, colors, creeds, and capabilities. It was (and remains to this day) important to us that we provide a great experience for our users while not leaving anyone out in the cold.
As most of you know, there’s a ton of interactivity on Mixx—voting, reporting, submitting, interacting with YourMixx—the list goes on. Each of these elements involves a different interaction with the application and every one of them involves manipulating content on-screen using a combination of JavaScript and CSS.
In an effort to accommodate users (or, more directly, their browser of choice) who have JavaScript turned off, we take the approach of displaying all content by default and then using JavaScript to hide the appropriate elements. The easiest to observe example of this is on the permalink pages. With recent redux of the permalinks, we added some new “tabs” (for lack of a better descriptor) below the entry information and above the comments: Activity, About this site, and Related.
When browsing to a permalink page, depending on how zippy your Internet connection is, you may notice that the three tabs are expaned and stacked on top of one another initially. After a brief pause, they’ll snap away and they may then be accessed by clicking on their respective button. This is, at the most basic, the situation described above. Page content loads. JavaScript does its job and hides the appropriate pieces.
In a perfect world, this would happen instantaneously and no one would notice. In the real world, Internet connections are variable, web servers hiccup, and external resources (like Google Analytics) have the side-effect of stalling firing of local scripts. The last point there is the one of concern to our discussion. Steve Souders, in his excellent book High Performance Web Sites, goes into great detail about why this stalling happens. Check out Chapter 6 on “Problems with Scripts.”
So what do we do? Let’s first take a look at how Mixx currently works.
As we’ve observed, our HTML and CSS style everything on the page and display elements by default. Once everything is loaded up, the Mixx JavaScript (using jQuery, a subject for another post) uses $().ready() to fire off our onload events. The appropriate bits hide away and we’re done. This is great as far as “best practices” and all that are concerned, but less-than-great from a perceptual viewpoint.
Robert Nyman, in his post How to hide and show initial content, depending on whether JavaScript support is available, outlines a technique where you add a