Speed up the publish time of your store

The following post comes courtesy of Istvan Siposs, Yahoo! Store developer, RTML guru, and author of several invaluable guides for merchants such as the Yahoo! Store Tips & Tricks 2nd edition, and RTML 101: The Unofficial Guide to Yahoo! Store Templates. The post is recommended for merchants with some understanding and experience with RTML.

Recently I did a complete redesign in a store. As part of the redesign, I added a hierarchical, DHTML menu navigation bar to the site. In a typical setup, such a navigation bar works like this: use WITH-OBJECT :index to reference the home page, then cycle through the contents field, create a menu label for each page you find there, and for any such page, create a sub-menu if the page also has contents.

If you didn’t know until now, whenever you reference another object in RTML, such as using WITH-OBJECT or FOR-EACH-OBJECT, such a reference causes a disk lookup on the server. This is what we call an “expensive” operator; it is expensive in terms of processing power and time, so a navigation bar in general, and a hierarchical navigation bar in particular is an “expensive” template.

In the store I was working in, they had over 4,000 pages, and of course, each of those pages had the navigation bar on them. So not surprisingly, the time it took to publish this site went from a couple of minutes to 3-4 hours!

ONCE to the rescue

There are many ways to write good, efficient, and bad inefficient RTML code, but regardless of your coding style, there is one operator you should consider every time you write a piece of code that won’t change from page to page, and it is the ONCE operator.

ONCE takes one parameter, which can be either :page or :publish. What it means is that whatever you paste within ONCE will be evaluated only once per page or per publish. Now, per page is not that interesting (I think at least, I haven’t really found any use for that,) but per publish is great.

When you use ONCE :publish, the expressions pasted inside it are evaluated only once during publishing. So how can you speed up your navigation bar? The immediate idea is to simply paste your original navigation code inside a ONCE :publish. Ok, try it, and you’ll be quite surprised: your navigation bar will appear only on a single page, the first one that was generated during your publish. Not exactly what you want, but if you think about what ONCE does, it does make sense: you asked it to generate your navigation bar once per publish. To get the speed benefits of ONCE :publish and also end up with a navigation bar on every page, do the following small variation:


TEXT ONCE :publish

Now this will do the trick nicely! We still generate the navigation bar once per publish, but now the result is saved (GRABbed) and output on each page!

With this simple trick, my store’s publish time went back to about 5-10 minutes.

I use this trick all the time for pieces of the page that doesn’t have to change throughout the site. It works nicely for banners, footers, random testimonials, etc. You can use the above code snippet as a boiler plate, just put the code that needs to be evaluated once inside the GRAB operator and you’ll be all set. Be careful though, don’t do this to code that changes from page to page (for example, you cannot use this trick if your navigation bar changes based on which main category you are in.)

Editor still slow?

Ok, you’ve tried it, publish time is great, but your editor is slow. Chances are you have some inefficient RTML code in your templates, and if that’s the case, ONCE will not save you anything there. Why? Because when you are in the editor, you are essentially publishing each page every time you view a page! Not the entire store (of course), but viewing any page in the editor is a “mini publish” of that page. So if you take my DHTML menu example above, that menu needs to be re-generated in the editor every time you view a page. One way to speed up the editor is to turn off slow pieces of code while in the editor. For example, you can program your template so that it does not show a navigation bar while in the editor (in the editor, you can still navigate in other means, like a bottom of page text navigation, breadcrumbs, or clicking the “Contents” button.)

It is easy enough to turn off any piece of RTML code by using the WHEN operator. You can, for example, create a custom yes-no variable called “in-editor”, and then take your existing site navigation and stuff it inside a WHEN @in-editor operator. This will do the trick, simply set in-editor to Yes while you are working in the editor, and set it to No before publishing.

There are two problems with this approach though: one, you have to manually do this all the time you are working in the editor, and two, you or someone at your organization might accidentally publish the store with that setting set to Yes. Then, your live site will end up with no navigation. So what to do?

With a little trick, instead of using a custom variable, let the editor tell you whether your code is running in the editor or not! Create the following template:


is-editor

Once you have this template, use it instead of a custom variable to disable pieces of code in the editor like this:


WHEN NOT CALL :is-editor

… and disabled inside the editor

But how do I navigate?

There are many other ways to navigate your store besides using the store’s navigation bar. You can use the “Contents” page in the Advanced Editor, or use the footer text links (if you have those), or breadcrumbs. Or, if you know the ID of the page you want to edit, you can erase the last piece (after the last / character) of the address in the browser’s address box, and replace it with the page Id plus the .html extension. For example, if you want to go to the site map page, and the address in your editor’s address bar looks like this (more or less):

http://us-f4-edit.store.yahoo.com/RT/NEWEDIT.storeid/d49f35bb5d6b/C7FwkAAB

then replace the bolded part with ind.html and hit Enter like this:

http://us-f4-edit.store.yahoo.com/RT/NEWEDIT.storeid/d49f35bb5d/ind.html

Istvan Siposs,
Guest blogger for Yahoo! Small Business

Sordid Details, of a sort

The following post from Sheridan Rawlins, Yahoo! Small Business engineer, outlines two often misunderstood RTML operators, SORT, and INDEXED-SORT. The post is intended for Yahoo! Store Developers or merchants with an advanced understanding of RTML. Learn more about RTML.

Some time ago we released the SORT operator in response to the difficulty and tedium of writing a sorting template directly in the RTML language. I would like to shed some light on how it works, describe the differences in the two flavors of sort (SORT & INDEXED-SORT), and describe more advanced sorting, such as sorting on a secondary property when the first property is equivalent.

Both the SORT and INDEXED-SORT operators expect 2 variable names (var1 & var2), a sequence, and a body. The elements of sequence are examined 2 elements at a time by setting the two variables and evaluating the body to determine whether var1 is less than the var2. A copy of the sequence in sorted order is returned.

The body allows full customization of the sort order subject to some conventions which should be followed: return-style consistency, and comparison consistency.

Return-style consistency

We support two styles of body’s return value – one of the following styles should be chosen and stay consistent for a given instance of SORT or INDEXED-SORT:

1. strcmp-like: a number, which is
   1. -1 if var1 < var2
   2. 0 if var1 = var2
   3. +1 if var1 > var2
2. Binary Predicate: a symbol which is
   1. T if A < B
   2. NIL if A >= B

Comparison Consistency

The comparison must obey the following “StrictWeakOrdering” properties:

• Irreflexive: when (var1=A, var2=A), body must return 0 or NIL
• Antisemetric: when (var1=A, var2=B) returns -1 or T, then (var1=B, var2=A) must return +1 or NIL
• Transitive: when (var1=A, var2=B) returns -1 or T, and (var1=B, var2=C) returns -1 or T, then (var1=A, var2=C) must return -1 or T as well.

When to use SORT vs. INDEXED-SORT

SORT is useful when sequence directly describes the elements to be sorted.

INDEXED-SORT is useful when “expensive” lookups must be performed against each element in sequence to determine the index to sort on. The best example of this is when you have a sequence of ids (i.e. @contents) and you wish to sort on the @name property of each element.

INDEXED-SORT’s extra parameter: getindex

INDEXED-SORT first iterates across sequence setting var1 for each element and evaluating getindex. Then, it evaluates body much like sort but sets var1, and var2 to these indices. A copy of the sorted sequence is returned (indices are discarded).

I hope this helps describe the basic principles of the sort and comparison operators. I’ll provide examples and other sorting tidbits in a companion post.

Sheridan Rawlins
Yahoo! Small Business

Edit 01/11/07

In order to reverse the sort order you need to tweak the body of the INDEXED-SORT.

The current implementation of sort-item-by-price has something like the following:

sort-items-by-price (items nilgreater)

INDEXED-SORT var1 var1
             var2 var2
             getindex WITH-OBJECT var1
                        ELEMENT position 0
                                sequence @price
             sequence items
  CALL :<=>-nil.
    var1
    var2
    nilgreater

To reverse the sort you would would switch the arguments and the boolean in the call to <=>-nil. — Something like:

sort-items-by-price (items nilgreater)

INDEXED-SORT var1 var1
             var2 var2
             getindex WITH-OBJECT var1
                        ELEMENT position 0
                                sequence @price
             sequence items
  CALL :<=>-nil.
    var2
    var1
    NOT nilgreater

I hope this helps!–SCR

RTML Tips# 1—Optimizing Publishing Times

The following post comes from Sheridan Rawlins, Yahoo! Small Business engineer, covers the use of RTML Operators and how certain custom template configurations can lead to long publishing times. The post is intended for Yahoo! Store Developers or merchants with an advanced understanding of RTML. Learn more about RTML.

In my role as lead engineer for the Editor & RTML language, I have come across questions regarding “slow publishing”. Almost invariably, the store or developer requesting assistance has customized their templates in some way. I would like to share some insights by describing answers to some frequently asked questions, along with their resolutions. With this knowledge, developers should be able to spread the best-practices for reducing publishing times for templates.

What causes Editor publishing to take a long time?

The most common root cause of slowness is due to having to fetch objects from disk. We do our best to cache as much as we can in memory, but as complex RTML templates are run on larger stores, or stores with complex object relationships, objects may be written back to disk and flushed from the cache to make room for other objects.

What operators cause objects to be read from disk?

• WITH-OBJECT
• FOR-EACH-OBJECT
• GET-PATH-TO (single “breadcrumb”)
• GET-ALL-PATHS-TO (multi “breadcrumbs”)

Contrary to popular belief, the WHOLE-CONTENTS operator does not cause cache-misses. However, it is usually used in conjunction with a FOR-EACH-OBJECT loop, which causes every object to be read into memory sequentially, almost guaranteeing that the cache will overflow and cause disk accesses.

What can I do to increase the speed of publishing my store?

There are several techniques to use to help improve the speed of publishing:

• Don’t repeat yourself – use WITH= to save the results of expensive operations where possible.
• Try not to nest FOR-EACH-OBJECT loops. Sometimes it’s inevitable, due to the nature of what you’re trying to accomplish, just be aware of this, and use trick #1 when you need to get object information (such as @name, or @price) multiple times.
• Try to keep your store hierarchy simple (see below)

I use the same templates for several stores. Why does one store publish faster than another?

Sometimes algorithms can be affected by the data. Take for instance multi-breadcrumbs. We’ve provided the operator GET-ALL-PATHS-TO in order to provide the fastest means of getting paths typically from INDEX -> id. The number of paths returned depends upon the data set.

Figure 1 – single inheritance (1 path per item) 

If all items and sections are only contained by one section or index, then the answer set is small. (See Figure 1)

Figure 2 – multiple inheritance (3 paths)

If, however, a store has items listed in many sections, then the problem grows to be as many sections as each item is listed in. (See Figure 2)

Figure 3 – multiple inheritance at multiple levels (9 paths)

Progressing further, if sections are contained in other sections, and yet all paths lead upwards, the problem multiplies to be all combinations of sections leading up to the top element. (See Figure 3)

Figure 4 – multiple inheritance with loops (585 paths)

The highest level of complexity is when there are “loops” in the hierarchy coupled with multiple sections leading to INDEX. If section A is contained in B, and B is contained in A, then the problem blows into the most complex yet, as it is each combination with 2 elements * each combination with 3 elements, etc up to the number of elements which are cross-referential. (See Figure 4)

This complexity also changes drastically with additional sections:

Hopefully these tips will help you make changes to your templates to make them more efficient and quicker to publish. Stay tuned for more RTML tips in later posts.

Sheridan Rawlins
Yahoo! Small Business