container.yml will run after integration.yml COMPLETES successfully and in master branch.
Style changes, cleanup and improved integration with CI by leveraging the use of
shared cache between all workflows.
* Podman is now supported to build the container images (Docker also received a refactor, merging both build and buildx)
* Container images are being built by Buildah instead of Docker BuildKit.
* Container images are tested before release.
* Splitting "modern" (amd64 & arm64) and "legacy" (armv7) arches on different Dockerfiles allowing future optimizations.
l10n.yml will run after integration.yml finishes successfully (will defer anything depending on integration.yml until heavy loads like container building are moved to separate workflows) and in master branch.
* After every integration.yml workflow completes successfully, only the `update` job runs.
* Dispatch and Crontab triggers only the `pr` job.
Style changes, cleanup and improved integration with CI by leveraging the use of shared cache between all workflows (not functional until all workflows have been refactored).
Instead of executing the workflow after integration.yml completes correctly, let's run this workflow parallel to integration.yml restoring the original behaviour.
Starting with ionicons-8.0.8 the SVG already contains a class attribute and
instaed of using SVGO plugin ``addAttributesToSVGElement`` we habve to use
``addClassesToSVGElement`` to add out ``__jinja_class_placeholder__``.
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
We actually don't need to keep the token on checkout because `peter-evans/create-pull-request` will read from `github.token`. The obvious `pull-requests` write permission wasn't set in the last fix, so I added it now.
documentation.yml will run after integration.yml COMPLETES successfully (will
defer anything depending on integration.yml until heavy loads like container
building are moved to separate workflows) and in master branch.
Style changes, cleanup and improved integration with CI by leveraging the use of
shared cache between all workflows (not functional until all workflows have been
refactored).
Style changes, cleanup and improved integration with CI by leveraging the use of
shared cache between all workflows (not functional until all workflows have been
refactored).
Style changes, cleanup and improved integration with CI by leveraging the use of
shared cache between all workflows (not functional until all workflows have been
refactored).
This is one of various PR to refactor the entire SearXNG Docker workflow.
Switches to Python glibc based images, all dependencies are installed via pip and not from system repositories, and several minor changes.
This PR will increase the image size from 194.9 MB to 345.47 MB (amd64), this is due to ARMv7 images (needs dependencies for wheels compilation and runtime (?)) and uWSGI webserver. Later PR will reduce the final image size.
The definition of a context name belongs in the abstract base class (was
previously only in the concrete implementation for the SQLite adapter).
Suggested-by: @dalf https://github.com/searxng/searxng/pull/4650#discussion_r2069873853
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Prophylactic encryption of the value currently makes no sense; on the contrary,
since the ``cryptography`` package is not available on armv7, it would cause
further problems.
Suggested-by: @dalf https://github.com/searxng/searxng/pull/4650#issuecomment-2830786661
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
The EngineCache class replaces all previously individual solutions for caches in
the context of the engines.
- demo_offline.py
- duckduckgo.py
- radio_browser.py
- soundcloud.py
- startpage.py
- wolframalpha_api.py
- wolframalpha_noapi.py
Search term to test most of the modified engines::
!ddg !rb !sc !sp !wa test
!ddg !rb !sc !sp !wa foo
For introspection of the DB, jump into developer environment and run command to
show cache state::
$ ./manage pyenv.cmd bash --norc --noprofile
(py3) python -m searx.enginelib cache state
cache tables and key/values
===========================
[demo_offline ] 2025-04-22 11:32:50 count --> (int) 4
[startpage ] 2025-04-22 12:32:30 SC_CODE --> (str) fSOBnhEMlDfE20
[duckduckgo ] 2025-04-22 12:32:31 4dff493e.... --> (str) 4-128634958369380006627592672385352473325
[duckduckgo ] 2025-04-22 12:40:06 3e2583e2.... --> (str) 4-263126175288871260472289814259666848451
[radio_browser ] 2025-04-23 11:33:08 servers --> (list) ['https://de2.api.radio-browser.info', ...]
[soundcloud ] 2025-04-29 11:40:06 guest_client_id --> (str) EjkRJG0BLNEZquRiPZYdNtJdyGtTuHdp
[wolframalpha ] 2025-04-22 12:40:06 code --> (str) 5aa79f86205ad26188e0e26e28fb7ae7
number of tables: 6
number of key/value pairs: 7
In the "cache tables and key/values" section, the table name (engine name) is at
first position on the second there is the calculated expire date and on the
third and fourth position the key/value is shown.
About duckduckgo: The *vqd coode* of ddg depends on the query term and therefore
the key is a hash value of the query term (to not to store the raw query term).
In the "properties of ENGINES_CACHE" section all properties of the SQLiteAppl /
ExpireCache and their last modification date are shown::
properties of ENGINES_CACHE
===========================
[last modified: 2025-04-22 11:32:27] DB_SCHEMA : 1
[last modified: 2025-04-22 11:32:27] LAST_MAINTENANCE :
[last modified: 2025-04-22 11:32:27] crypt_hash : ca612e3566fdfd7cf7efe2b1c9349f461158d07cb78a3750e5c5be686aa8ebdc
[last modified: 2025-04-22 11:32:30] CACHE-TABLE--demo_offline: demo_offline
[last modified: 2025-04-22 11:32:30] CACHE-TABLE--startpage: startpage
[last modified: 2025-04-22 11:32:31] CACHE-TABLE--duckduckgo: duckduckgo
[last modified: 2025-04-22 11:33:08] CACHE-TABLE--radio_browser: radio_browser
[last modified: 2025-04-22 11:40:06] CACHE-TABLE--soundcloud: soundcloud
[last modified: 2025-04-22 11:40:06] CACHE-TABLE--wolframalpha: wolframalpha
These properties provide information about the state of the ExpireCache and
control the behavior. For example, the maintenance intervals are controlled by
the last modification date of the LAST_MAINTENANCE property and the hash value
of the password can be used to detect whether the password has been changed (in
this case the DB entries can no longer be decrypted and the entire cache must be
discarded).
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
Fixes the semantic scholar engine by extracting a ui version token.
BTW: remove html tags from the content.
Author's checklist:
- they are ratelimiting very fast, if you do approx more than 2 requests per
minute, you have to wait some time again...
- they also have an official api at api.semanticscholar.org, but it's ratelimits
are even harder
Closes: https://github.com/searxng/searxng/issues/4685
The language settings page in the documentation reference languages.py which was
removed in commit c9cd376
Closes: https://github.com/searxng/searxng/issues/4690
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
* filtering ChinaSo-News results by source, option ``chinaso_news_source``
* add ChinaSo engine to the online docs https://docs.searxng.org/dev/engines/online/chinaso.html
* fix SearXNG categories in the settings.yml
* deactivate ChinaSo engines ``inactive: true`` until [1] is fixed
* configure network of the ChinaSo engines
[1] https://github.com/searxng/searxng/issues/4694
Signed-off-by: @BrandonStudio
Co-authored-by: Markus Heiser <markus.heiser@darmarit.de>
