Sometimes, it looks like it is not possible to avoid using an accumulating array, a pattern that feels unnatural in Ruby.
Recently, I’ve needed to chase down and unroll pagination links over a JSON / REST api. I don’t know how many pages there will be, and it’s probable (but not guaranteed) that I need to retrieve and use all of the content. Since each page is dependent on results from the previous page, there is no obvious Enumerable parallel. Here, I’ll demonstrate a quick refactoring that will provide in a clean, lazy enumerable object.
This being a HATEOAS API, the next page link is embedded in the response JSON, tucked under a [“links”][“next”] key. To fetch all of the data, I end up with code that looks something like:
1 2 3 4 5 6 7 8 9 10 11 12 13 14
As ruby goes, this is pretty ugly. When there’s something distinct to enumerate over, it’s recommend to replace the accumulating widgets array with a #map call, or working with some other Enumerable method. Unfortunately, there isn’t a clear parallel for this case.
Another issue is that this loop fetches every single page, before returning control, and regardless of how many results I actually end up using.
Fortunately, there is Enumerator, Ruby’s answer to producing generators. The enumerator class produces an enumerable, backed with any arbitrary generation logic.
I can refactor the while loop to look more like this:
1 2 3 4 5 6 7 8 9 10 11 12 13
We’ve gotten rid of the accumulator array, and instead have something that looks much closer to idiomatic ruby. Additionally, the method begins yielding immediately after fetching the first page, only retrieving additional pages when needed, without client code needing to understand the mechanics of the underlying pagination.
It is important to note that instead of returning an Array, retrieve_all_pages now returns an Enumerable, but generally quacks the same — it’s rather unlikely that any client of the original implementation was using direct array semantics; if so, a simple to_a converts an Enumerable to a normal Array.
Overall though, I find the resulting enumerator to be far easier to work with in standard ruby, providing a more flexible and versatile encapsulation of the remote interface.
cross-posted from Chris Flipse’s Blog