RTML Tips# 1—Optimizing Publishing Times

The following post comes from Sheridan Rawlins, Yahoo! Small Business engineer, covers the use of RTML Operators and how certain custom template configurations can lead to long publishing times. The post is intended for Yahoo! Store Developers or merchants with an advanced understanding of RTML. Learn more about RTML.

In my role as lead engineer for the Editor & RTML language, I have come across questions regarding “slow publishing”. Almost invariably, the store or developer requesting assistance has customized their templates in some way. I would like to share some insights by describing answers to some frequently asked questions, along with their resolutions. With this knowledge, developers should be able to spread the best-practices for reducing publishing times for templates.

What causes Editor publishing to take a long time?

The most common root cause of slowness is due to having to fetch objects from disk. We do our best to cache as much as we can in memory, but as complex RTML templates are run on larger stores, or stores with complex object relationships, objects may be written back to disk and flushed from the cache to make room for other objects.

What operators cause objects to be read from disk?

• WITH-OBJECT
• FOR-EACH-OBJECT
• GET-PATH-TO (single “breadcrumb”)
• GET-ALL-PATHS-TO (multi “breadcrumbs”)

Contrary to popular belief, the WHOLE-CONTENTS operator does not cause cache-misses. However, it is usually used in conjunction with a FOR-EACH-OBJECT loop, which causes every object to be read into memory sequentially, almost guaranteeing that the cache will overflow and cause disk accesses.

What can I do to increase the speed of publishing my store?

There are several techniques to use to help improve the speed of publishing:

• Don’t repeat yourself – use WITH= to save the results of expensive operations where possible.
• Try not to nest FOR-EACH-OBJECT loops. Sometimes it’s inevitable, due to the nature of what you’re trying to accomplish, just be aware of this, and use trick #1 when you need to get object information (such as @name, or @price) multiple times.
• Try to keep your store hierarchy simple (see below)

I use the same templates for several stores. Why does one store publish faster than another?

Sometimes algorithms can be affected by the data. Take for instance multi-breadcrumbs. We’ve provided the operator GET-ALL-PATHS-TO in order to provide the fastest means of getting paths typically from INDEX -> id. The number of paths returned depends upon the data set.

Figure 1 – single inheritance (1 path per item) 

If all items and sections are only contained by one section or index, then the answer set is small. (See Figure 1)

Figure 2 – multiple inheritance (3 paths)

If, however, a store has items listed in many sections, then the problem grows to be as many sections as each item is listed in. (See Figure 2)

Figure 3 – multiple inheritance at multiple levels (9 paths)

Progressing further, if sections are contained in other sections, and yet all paths lead upwards, the problem multiplies to be all combinations of sections leading up to the top element. (See Figure 3)

Figure 4 – multiple inheritance with loops (585 paths)

The highest level of complexity is when there are “loops” in the hierarchy coupled with multiple sections leading to INDEX. If section A is contained in B, and B is contained in A, then the problem blows into the most complex yet, as it is each combination with 2 elements * each combination with 3 elements, etc up to the number of elements which are cross-referential. (See Figure 4)

This complexity also changes drastically with additional sections:

Hopefully these tips will help you make changes to your templates to make them more efficient and quicker to publish. Stay tuned for more RTML tips in later posts.

Sheridan Rawlins
Yahoo! Small Business