Saturday, December 3, 2011

Feed "pagination" hobbles Yahoo Pipes

The quirky but useful Pipes web service has added a new kink, with implications for bloggers that use it to reverse the order of their posts.

Running your blog feed though the pipe I built will flip the order, and until recently you could then just port the pipe's feed to feed2js to show your blog in the order you want.

Frustratingly, Yahoo Pipes has started to paginate its feeds in blocks of 100 posts. So the feed url you get from Pipes only displays the first 100 posts in the series. To see the next 100 posts, you have to add the following to the feed url:
You then run both feed urls, separately, through feed2js, and mash the results together so they look like a continuous feed.

Pain in the neck, but it's actually worse than that. You have to do that for each block of 100 items in your feed, so if you have 600 posts on your blog that means six separate feed urls, up to &page=6.

Heck, it is worse than that. If you keep blogging (and I hope you do), you must periodically add 100-post segments as you grow. You can't just set it up and leave it to do its magic as you could previously. There is potentially no end to the process.

Bad Yahoo!

Before I walk you through how to do this (and it's not so bad, its just that it should not be set up this way at all), lets review some other options.

First of all, you might want to paginate your own blog journey by year, as I do on my other blog. A script with hundreds of blog posts can take a long time to load and breaking it up makes it more manageable.

If you are going to do that anyway, and if you do not have more than 100 posts in any year, then you don't need to worry about this new curve ball from Yahoo at all. Plop your feed into my pipe, segment the feed by date for each year, and go. If there are less than 100 posts, the issue does not arise.

Second, Yahoo does not paginate its JSON object--it give you the entire feed in that format. If you have facility with javascript you can whip up your own script using the JSON. Unfortunately, feed2js does not work with that format.

I think the JSON-javascript option may turn out to be the best solution. But so far my Blog Journey hack, for all its faults, has not required anyone to be a programmer. (Granted there is a script, generated for you by feed2js, but all you need to do is copy and paste it into your blog. Feed2js does all the heavy lifting and that, along with its great user interface, is why we love it.)

One great accomplishment of all these cloud-based web services, such as Pipes and Blogger itself, is that they put a friendly user interface between you and the underlying code. I try to honor and preserve that. I feel I am writing for some pretty smart people here, but I draw the line at actually programming.

So here's how to work around Yahoo's unfortunate pagination problem.
  1. Run the feed through my pipe and get the feed of the pipe output, as described here.
  2. Using a service such as feed2js, turn that into a script and paste it into into your blog.
  3. For each additional set of 100 posts in your feed, add &page=n to the feed url, where n is the page containing the set (starting with 2 for posts nos. 101–200).
  4. Run that though feed2js and add the code to the script you've already pasted into your blog in way outlined below.
  5. Repeat that for all the pages in your feed.

If you just paste the scripts one after the other, there will be a space between them in the output they generate on your blog. To avoid this, eliminate the spaces after the <script>...</script> and <noscript>...</noscript> tags. (Lots of stuff between those tags; don't mess with it.)

Those spaces make it a lot easier for the human eye to parse, but the script does not require them to run.

This whole episode is another reminder, not that we needed one, of the drawbacks of using third-party web services.

I still think that doing so avoids analogous drawbacks of trying to hack Blogger with scripts, but this sure has not been a good year for the third-party approach.


  1. Hi adam,

    This post was of very useful. I was facing similar issue and couldn't find how to get the full count nodes. This thrown me a light.


    1. Santhosh, glad to be of service.

      If you are working in Javascript, note that, apparently, Yahoo does not paginate the JSON object for these feeds so maybe you could use that. (No, I do not know how that works.)

  2. Adam,
    Your blog is so informative. Thanks. But, can this be done only at the point when the blog expands after 100? For example, when the blog is small, keep the feed as it is. If it goes above 100, add script for page 1, if it goes above 200, then page 2 and so on.

    1. @anon: That's what I do. Add 100 as needed.

      It is not very elegant though, it it?

    2. Yes. It is not, that is not how scripts should be working. But thanks for the solution.