Back to @arc's timeline
arc shared this post 路 2h ago
Web
Alkarex
Alkarex

Implement Web scraping "HTML + XPath" by Alkarex 路 Pull Request #4220 路 FreshRSS/FreshRSS 路 GitHub

This PR adds a (killer? 馃ぉ) functionality, namely the ability to consume any Web site / HTML source, also when an RSS / Atom feed is not available.

It is based on XPath 1.0, which is natively available in PHP:

FreshRSS can then also republish as RSS.

This can be combined with our existing ability to follow article links to retrieve full article content (use with parsimony).

Contributes indirectly to other issues:

  • Better handling of enclosures
  • Better RSS outputs for Add in FreshRSS output rss to be compatible with podcast software #1796
  • Cache HTTP requests when getting full article content
  • Better handling of encoding when getting full article content
  • Purge old feed caches regularly
  • Keep the original order of articles when they all have the same date (before, we got a random order)
  • More PHP type hints
  • Typos in French
    What remains to be done (other PRs):
  • Preview to help writing the XPath expressions just like for our full article content retrieval (help welcome as I will not be able to do much in the coming two weeks)
  • Consume "JSON +XPath" sources (only very little needs to be changed to support that)
  • A few new TODOs added in the code (in particular an easy performance improvement related to iterating over SimplePie entries) Faster $simplePie->get_items() #4263
  • Documentation (help welcome) Technical note: all XPath evaluations (i.e. XPath supposed to return some text and not the queries returning a collection) are automatically wrapped into a normalize-space() function to cast the result to string if not already done by the user and also remove superfluous white spaces.

Include information in OPML export / import? OPML export/import of some proprietary FreshRSS attributes #4342
Custom date format, e.g. with https://php.net/datetime.createfromformat
Make corresponding upstream PR, reverting [simplepie/simplepie@e49c578#commitcomment-67585150](https://github.com/simplepie/simplepie/commit/e49c578817aa504d8d05cd7f33857aeda9d41908#commitcomment-67585150) : Re-enable xml:base for all supported RSS formats simplepie/simplepie#723

Summary of the tasks from comments below:

  • Implement support of ``

Comments

Alkarex 路 2022-02-13 Example for https://www.france.tv/france-2/les-petits-meurtres-d-agatha-christie/toutes-les-videos/ Result:

Alkarex 路 2022-02-13 This function is the only true new functional code for this feature, the rest of the PR being mainly related improvements and UI code

Alkarex 路 2022-02-13 #1796

Alkarex 路 2022-02-13 We sort ourselves. This created a bug randomising the order of entries when several have the same date. Also improves performances slightly by avoiding a sort.

InterferencePattern 路 2022-08-19 When will this feature be included in a release?

Alkarex 路 2022-08-19 When will this feature be included in a release?

@jimbudarz You are welcome to already use our edge branch, which is a relatively stable rolling release, available by git, Docker, ZIP. Feedback welcome, and we need more testers. Next stable version 1.20.0 is coming in I guess ~2-3 weeks.

math-GH 路 2022-08-20 I can confirm that the edge (V1.20.0-dev) works very well. I use it on my prod instance. The Web scraping works great there.

Alkarex 路 2022-10-01 Nice article on the subject: https://danq.me/2022/09/27/freshrss-xpath/ Originally posted by @marienfressinaud in [#4647 (comment)](https://github.com/FreshRSS/FreshRSS/discussions/4647#discussioncomment-3777171)

math-GH 路 2022-10-01 More amazing feedback: https://forum.cloudron.io/topic/7651/freshrss-1-2-0-released-with-killer-feature-track-any-website

Alkarex 路 2022-10-21 Bug fix $attributes fixed in #4759