Feed "pagination" hobbles Yahoo Pipes

The quirky but useful Pipes web service has added a new kink, with implications for bloggers that use it to reverse the order of their posts.

Running your blog feed though the pipe I built will flip the order, and until recently you could then just port the pipe's feed to feed2js to show your blog in the order you want.

Frustratingly, Yahoo Pipes has started to paginate its feeds in blocks of 100 posts. So the feed url you get from Pipes only displays the first 100 posts in the series. To see the next 100 posts, you have to add the following to the feed url:

&page=2

You then run both feed urls, separately, through feed2js, and mash the results together so they look like a continuous feed.

Pain in the neck, but it's actually worse than that. You have to do that for each block of 100 items in your feed, so if you have 600 posts on your blog that means six separate feed urls, up to &page=6.

Heck, it is worse than that. If you keep blogging (and I hope you do), you must periodically add 100-post segments as you grow. You can't just set it up and leave it to do its magic as you could previously. There is potentially no end to the process.

Bad Yahoo!

Before I walk you through how to do this (and it's not so bad, its just that it should not be set up this way at all), lets review some other options.

First of all, you might want to paginate your own blog journey by year, as I do on my other blog. A script with hundreds of blog posts can take a long time to load and breaking it up makes it more manageable.

If you are going to do that anyway, and if you do not have more than 100 posts in any year, then you don't need to worry about this new curve ball from Yahoo at all. Plop your feed into my pipe, segment the feed by date for each year, and go. If there are less than 100 posts, the issue does not arise.

Second, Yahoo does not paginate its JSON object--it give you the entire feed in that format. If you have facility with javascript you can whip up your own script using the JSON. Unfortunately, feed2js does not work with that format.

I think the JSON-javascript option may turn out to be the best solution. But so far my Blog Journey hack, for all its faults, has not required anyone to be a programmer. (Granted there is a script, generated for you by feed2js, but all you need to do is copy and paste it into your blog. Feed2js does all the heavy lifting and that, along with its great user interface, is why we love it.)

One great accomplishment of all these cloud-based web services, such as Pipes and Blogger itself, is that they put a friendly user interface between you and the underlying code. I try to honor and preserve that. I feel I am writing for some pretty smart people here, but I draw the line at actually programming.

So here's how to work around Yahoo's unfortunate pagination problem.

1. Run the feed through my pipe and get the feed of the pipe output, as described here.
2. Using a service such as feed2js, turn that into a script and paste it into into your blog.
3. For each additional set of 100 posts in your feed, add &page=n to the feed url, where n is the page containing the set (starting with 2 for posts nos. 101–200).
4. Run that though feed2js and add the code to the script you've already pasted into your blog in way outlined below.
5. Repeat that for all the pages in your feed.

If you just paste the scripts one after the other, there will be a space between them in the output they generate on your blog. To avoid this, eliminate the spaces after the <script>...</script> and <noscript>...</noscript> tags. (Lots of stuff between those tags; don't mess with it.)

Those spaces make it a lot easier for the human eye to parse, but the script does not require them to run.

This whole episode is another reminder, not that we needed one, of the drawbacks of using third-party web services.

I still think that doing so avoids analogous drawbacks of trying to hack Blogger with scripts, but this sure has not been a good year for the third-party approach.