# Implement Web scraping "HTML + XPath" by Alkarex 路 Pull Request #4220 路 FreshRSS/FreshRSS 路 GitHub
Canonical: https://social-archive.org/arc/hjBUMCH9KU
Original URL: https://github.com/FreshRSS/FreshRSS/pull/4220
Author: Alkarex
Platform: web
Share mode: full
## Content
This PR adds a (killer? 馃ぉ) functionality, namely the ability to consume any Web site / HTML source, also when an RSS / Atom feed is not available. It is based on XPath 1.0, which is natively available in PHP: - [https://www.w3.org/TR/xpath-10/](https://www.w3.org/TR/xpath-10/) - [https://php.net/class.domxpath](https://php.net/class.domxpath) - [https://developer.mozilla.org/docs/Web/XPath/Axes](https://developer.mozilla.org/docs/Web/XPath/Axes) This is a light version of what RSS Bridge offers, but natively inside FreshRSS and much easier to use: - [https://github.com/RSS-Bridge/rss-bridge/wiki/The-collectData-function](https://github.com/RSS-Bridge/rss-bridge/wiki/The-collectData-function) - [https://github.com/RSS-Bridge/rss-bridge/wiki/XPathAbstract](https://github.com/RSS-Bridge/rss-bridge/wiki/XPathAbstract) Using a third-party tool did not scale enough anymore for me, when I just need to quickly add another (simple) source. FreshRSS can then also republish as RSS. This can be combined with our existing ability to follow article links to retrieve full article content (use with parsimony). Contributes indirectly to other issues: - Better handling of enclosures - Better RSS outputs for [Add in FreshRSS output rss to be compatible with podcast software #1796](https://github.com/FreshRSS/FreshRSS/issues/1796) - Cache HTTP requests when getting full article content - Better handling of encoding when getting full article content - Purge old feed caches regularly - Keep the original order of articles when they all have the same date (before, we got a random order) - More PHP type hints - Typos in French What remains to be done (other PRs): - Preview to help writing the XPath expressions just like for our full article content retrieval (help welcome as I will not be able to do much in the coming two weeks) - Consume "JSON +XPath" sources (only very little needs to be changed to support that) - A few new TODOs added in the code (in particular an easy performance improvement related to iterating over SimplePie entries) [Faster $simplePie->get_items() #4263](https://github.com/FreshRSS/FreshRSS/pull/4263) - Documentation (help welcome) Technical note: all XPath evaluations (i.e. XPath supposed to return some text and not the queries returning a collection) are automatically wrapped into a [`normalize-space()`](https://developer.mozilla.org/docs/Web/XPath/Functions/normalize-space) function to cast the result to string if not already done by the user and also remove superfluous white spaces. Include information in OPML export / import? [OPML export/import of some proprietary FreshRSS attributes #4342](https://github.com/FreshRSS/FreshRSS/pull/4342) Custom date format, e.g. with [https://php.net/datetime.createfromformat](https://php.net/datetime.createfromformat) Make corresponding upstream PR, reverting [simplepie/simplepie@e49c578#commitcomment-67585150](https://github.com/simplepie/simplepie/commit/e49c578817aa504d8d05cd7f33857aeda9d41908#commitcomment-67585150) : [Re-enable xml:base for all supported RSS formats simplepie/simplepie#723](https://github.com/simplepie/simplepie/pull/723) Summary of the tasks from comments below: - Implement support of `` --- ## Comments > **Alkarex** 路 2022-02-13 Example for `https://www.france.tv/france-2/les-petits-meurtres-d-agatha-christie/toutes-les-videos/` [](https://user-images.githubusercontent.com/1008324/153773031-68ddf8e9-7c29-4f1c-bd91-f30c84c584a1.png) [](https://user-images.githubusercontent.com/1008324/153772998-541c1d4b-1865-4631-9027-9f289b04c709.png) Result: [](https://user-images.githubusercontent.com/1008324/153772891-13d6a75f-2936-46d1-968b-f6e138518f84.png) > **Alkarex** 路 2022-02-13 This function is the only true new functional code for this feature, the rest of the PR being mainly related improvements and UI code > **Alkarex** 路 2022-02-13 [#1796](https://github.com/FreshRSS/FreshRSS/issues/1796) > **Alkarex** 路 2022-02-13 We sort ourselves. This created a bug randomising the order of entries when several have the same date. Also improves performances slightly by avoiding a sort. > **InterferencePattern** 路 2022-08-19 When will this feature be included in a release? > **Alkarex** 路 2022-08-19 When will this feature be included in a release? @jimbudarz You are welcome to already use our edge branch, which is a relatively stable rolling release, available by git, Docker, ZIP. Feedback welcome, and we need more testers. Next stable version 1.20.0 is coming in I guess ~2-3 weeks. > **math-GH** 路 2022-08-20 I can confirm that the edge (V1.20.0-dev) works very well. I use it on my prod instance. The Web scraping works great there. > **Alkarex** 路 2022-10-01 Nice article on the subject: [https://danq.me/2022/09/27/freshrss-xpath/](https://danq.me/2022/09/27/freshrss-xpath/) *Originally posted by [@marienfressinaud](https://github.com/marienfressinaud) in [#4647 (comment)](https://github.com/FreshRSS/FreshRSS/discussions/4647#discussioncomment-3777171)* > **math-GH** 路 2022-10-01 More amazing feedback: [https://forum.cloudron.io/topic/7651/freshrss-1-2-0-released-with-killer-feature-track-any-website](https://forum.cloudron.io/topic/7651/freshrss-1-2-0-released-with-killer-feature-track-any-website) > **Alkarex** 路 2022-10-21 Bug fix `$attributes` fixed in [#4759](https://github.com/FreshRSS/FreshRSS/pull/4759)
## Media
1. image: https://social-archiver-api.social-archive.org/media/archives/arc/dyqanH32b3/media/0.png