# XPath Scraping with FreshRSS
Canonical: https://social-archive.org/arc/3wwpdPDmEX
Original URL: https://danq.me/2022/09/27/freshrss-xpath/
Author: Dan Q
Platform: web
Share mode: full
## Content
I’ve been spending a while [running on reduced brain capacity](https://danq.me/2022/09/23/covid-brain/) lately so, to ease myself back into thinking like a programmer, I upgraded my preferred feed reader [FreshRSS](https://www.freshrss.org/) to version 1.20.0 – which was [released a couple of weeks ago](https://github.com/FreshRSS/FreshRSS/releases/tag/1.20.0) – and tried out what I believe is [its killer new feature](https://github.com/FreshRSS/FreshRSS/pull/4220): **HTML + XPath scraping**. ![Screenshot showing Beverley Newing](https://bcdn.danq.me/_q23u/2022/09/webdevbev-blog.png) I’ve been using RSS for about 20 years and [I *love* it](https://twobithistory.org/2018/09/16/the-rise-and-demise-of-rss.html). It [feels great](https://gilest.org/rss-feels.html) to be able to curate my updates [based on “what I care about”](https://lucybellwood.com/stumbling/), and [not on “what some social network thinks I should care about”](https://danq.me/2019/05/25/youtube/), to keep things to read later, to prioritise effectively based on my own categorisation, to consume content offline and have my to-read list synchronise later, etc. RSS never went away, of course ([what do you think a podcast is?](https://inessential.com/2013/03/14/why_i_love_rss_and_you_do_too)), but it got [steamrollered out of the public eye by big companies](https://www.techdirt.com/2022/06/03/the-internet-can-still-be-small-and-nice-but-its-on-all-of-us-to-make-that-work/) who make their money out of keeping your eyes on *their* platforms and off the open Web. But it feels like [it’s slowly coming back](https://www.wired.com/story/rss-readers-feedly-inoreader-old-reader/): even [Substack](https://substack.com/) – whose entire *thing* is that an email client is more-convenient than a feed reader for most people – [launched an RSS reader this week](https://on.substack.com/p/new-web-reader)! ![A smartphone on a wooden surface. The screen shows the FeedMe app, showing the most-recent blog post from Beverley](https://bcdn.danq.me/_q23u/2022/09/webdevbev-blog-on-phone-1024x659.jpg) I love RSS so much that [I routinely retrofit other people’s websites](https://danq.me/2018/07/05/comic-chameleon/) with feeds just so I can subscribe to them: I even published [the tool I use to do so](https://github.com/Dan-Q/rssey)! Whether [filtering sports headlines out of BBC News](https://danq.me/2019/05/14/bbc-news-without-the-sport/), turning [retro webcomics into “reading lists”](https://danq.me/2019/07/25/labs-comic-rss-archive/) so I can track my progress, or just [working around sites that really *should* have feeds](https://danq.me/2020/01/08/far-side-rss/) but refuse to, I just love sidestepping these “missing feeds”. [My friend Beverley](https://webdevbev.co.uk/) has a blog without any kind of feed, so [I added one](https://github.com/Dan-Q/rssey/blob/main/feeds/webdevbevblog.js) so I could subscribe to it. Magic. But with FreshRSS 1.20.0, I no longer have to maintain my own tool to get this brilliant functionality, and I’m overjoyed. Let’s look at how it works by re-subscribing to Beverley’s blog but *without* a middleware tool. ![Screenshot showing FetchRSS being used to graphically create a feed from Beverley](https://bcdn.danq.me/_q23u/2022/09/fetchrss-screenshot-1024x628.png) In the latest version of FreshRSS, when you add a new feed to your reader, a new section “Type of feed source” is available. Unfold it, and you can change from the default (“ RSS / Atom”) to the new option “ HTML + XPath (Web scraping)”. Put a human-readable page address rather than a feed address into the “Feed URL ” field and fill these fields to tell FreshRSS how to parse the page to get the content you want. Note that it doesn’t matter if the web page isn’t valid XML (e.g. missing closing tags) because [it’s going to get run through PHP’s DOMDocument anyway](https://github.com/Alkarex/FreshRSS/blob/871c1142a24b14a5ac621c290c2092f6b10361b3/app/Models/Feed.php#L587-L589) which will “correct” for some really sloppy code if needed. ![Browser debugger running document.evaluate(](https://bcdn.danq.me/_q23u/2022/09/debugger-select-from-xpath-1024x256.png) You’ll need to use XPath to express how to find a “feed item” on the page. Here’s the rules I used for [https://webdevbev.co.uk/blog.html](https://webdevbev.co.uk/blog.html) (many of these fields were optional – I didn’t have to do this much work): - **Feed title:** `//h1` I override this anyway in FreshRSS, so I could just have used the a string, but I wanted the XPath practice. There’s only one `` on the page, and it can be considered the “title” of the feed. - **Finding items:** `//li[@class="blog__post-preview"]` Each “post” on the page is an ``. - **Item titles:** `descendant::h2` Each post has a `` which is the post title. The descendant:: selector scopes the search to each post as found above. - **Item content:** `descendant::p[3]` Beverley’s static site generator template puts the post summary in the third paragraph of the ``, which we can select like this. - **Item link:** `descendant::h2/a/@href` This expects a URL, so we need the /@href to make sure we get the *value* of the ``, rather than its *contents*. - **Item thumbnail:** `descendant::img[@class="blog__image--preview"]/@src` Again, this expects a URL, which we get from the ``. - **Item author:** `"Beverley Newing"` Beverley’s blog doesn’t host any guest posts, so I just use a string literal here. - **Item date:** `substring-after(descendant::p[@class="blog__date-posted"], "Date posted: ")` This is the only complicated one: the published dates on Beverley’s blog aren’t explicitly marked-up, but part of a string that begins with the words “Date posted: “, so I use XPath’s [`substring-after`](https://developer.mozilla.org/en-US/docs/Web/XPath/Functions/substring-after) function to strtip this. The result gets passed to PHP’s [`strtotime()`](https://www.php.net/strtotime), which is pretty tolerant of different date formats (although not of the words “Date posted:” it turns out!). ![Screenshot: Adding a "HTML + XPath (Web scraping)" feed via FreshRSS.](https://bcdn.danq.me/_q23u/2022/09/freshrss-add-xpath-feed.png) I hope that this is just the beginning for this new killer feature in FreshRSS: there’s so much more it can be and do. But for now, I’m still mighty impressed that I can begin to phase-out my use of my relatively resource-intensive feed-building middleware and use my feed reader to do more and more of the heavy lifting for which I love it so much. I also love that this functionally adds h-feed support in by the back door. I’d still *prefer* there to be a “h-feed” option in the “Type of feed source” drop-down, but at least I can add such support manually, now! ![Beverley](https://bcdn.danq.me/_q23u/2022/09/webdevbev-blog-in-freshrss.png) ## Footnotes [1](#footnote-ref-20520-1) When I say RSS, I mean feed. Most of the feeds I subscribe to are RSS feeds, but some are Atom feeds, [h-feed](http://microformats.org/wiki/h-feed), etc. But I can’t get over the old-fashioned name, and I don’t care to try.
## Media
1. image: https://social-archiver-api.social-archive.org/media/archives/arc/kzA4s8FB2n/media/0.png