consult-omni

#+include: ~/OrgFiles/armin/org-macros.setup #+OPTIONS: h:1 num:nil toc:nil d:nil

#+TITLE: consult-omni - a powerful versatile omni search inside Emacs #+AUTHOR: Armin Darvish #+LANGUAGE: en

#+html: <a href="https://github.com/armindarvish"><img alt="Armin Darvish" src="https://img.shields.io/static/v1?label=Armin%20Darvish&message=consult-omni&color=00A8B0&logo=github"/></a> #+html: <a href="https://www.gnu.org/software/emacs/"><img alt="GNU Emacs" src="https://img.shields.io/static/v1?label=Made%20For&message=GNU%20Emacs&color=7a5aba&logo=gnuemacs&logoColor=white"/></a>

About =consult-omni=

consult-omni is a package for getting search results from one or several custom sources (web search engines, AI assistants, elfeed database, org notes, local files, desktop applications, mail servers, ...) directly in Emacs minibuffer. It is a successor of [[https://github.com/armindarvish/consult-web][consult-web]], with expanded features and functionalities.

consult-omni provides wrappers and macros around [[https://github.com/minad/consult][consult]], to make it easier for users to get results from different sources and combine local and web sources in an omni-style search. In other words, consult-omni enables getting consult-style multi-source or dynamically completed results in minibuffer for a wide range of sources including Emacs functions/packages (e.g. Emacs buffers, org files, elfeed,...), command-line programs (grep, find, gh, ...), or web search engines (Google, Brave, Bing, ...).

consult-omni can be an open-source free alternative to other omni-search tools such as Alfred or MacOS spotlight. It provides a range of default sources as examples, but the main idea here is to remain agnostic of the source and provide the toolset for the users to define their own sources/workflows (a.k.a plugins).

Here is the mandatory screenshot:

#+ATTR_ORG: :width 800px #+ATTR_LATEX: :width 800px #+ATTR_HTML: :width 800px [[https://github.com/armindarvish/consult-web/blob/screenshots/screenshots/consult-omni.gif]]

In the screenshot above, I use a key shortcut to open an Emacs frame and call =consult-omni= (similar to calling MacOS spotlight) and search for the term “emacs”, then move around and look at previews from different sources including application, gptel, Brave, Google, YouTube, elfeed, mu4e, ....

Here is another screenshot demonstrating using consult-omni as an application launcher: #+ATTR_ORG: :width 800px #+ATTR_LATEX: :width 800px #+ATTR_HTML: :width 800px [[https://github.com/armindarvish/consult-web/blob/screenshots/screenshots/consult-omni-launcher.gif]]

Usage and Features For explanation of features and comparison to some other packages, you can watch my YouTube videos, both for consult-web and for consult-omni below:

(video 1) in-depth tutorial for consult-omni: [[https://www.youtube.com/watch?v=wNH2E7iT__c][https://www.youtube.com/watch?v=wNH2E7iT__c]]
(video 2) in-depth tutorial of consult-web: https://www.youtube.com/watch?v=7pDfyqBZwvo

** Interactive Commands *** single source commands For each source, you may have static or dynamic commands. Static commands query the use for an input and then fetch the results. Dynamic commands, do dynamic completion as the user types (show results as user is typing). Dynamic commands feel a bit more intuitive and modern in 2024, but on the other hand have the disadvantage of sending the query to the servers multiple times especially if you type slowly! Depending on the service provider and the API model, you may want to avoid hitting the server too frequently (for example for services that you pay per query), therefore for certain services a static command might be a better choice than the dynamic command. Using the macro =consult-omni-define-source=, you can chose to create static, dynamic or both by passing =nil=, =t=, or ='both= to the keyword =:static=. Here is an example from the source code, for creating both static and dynamic commands for Brave search:

#+begin_src emacs-lisp (consult-omni-define-source "Brave" :narrow-char ?b :type 'dynamic :require-match t :face 'consult-omni-engine-title-face :request #'consult-omni--brave-fetch-results :preview-key consult-omni-preview-key :search-hist 'consult-omni--search-history :select-hist 'consult-omni--selection-history :enabled (lambda () (bound-and-true-p consult-omni-brave-api-key)) :group #'consult-omni--group-function :sort t :static 'both) #+end_src

Here is a screenshot of STATIC interactive command for Wikipedia: #+ATTR_ORG: :width 800px #+ATTR_LATEX: :width 800px #+ATTR_HTML: :width 800px [[https://github.com/armindarvish/consult-web/blob/screenshots/screenshots/consult-omni-wikipedia-static.gif]]

Here is a screenshot of DYNAMIC interactive command for Wikipedia: #+ATTR_ORG: :width 800px #+ATTR_LATEX: :width 800px #+ATTR_HTML: :width 800px [[https://github.com/armindarvish/consult-web/blob/screenshots/screenshots/consult-omni-wikipedia.gif]]

*** multi source commands By default consult-omni provides two main multi-source interactive commands.

=consult-omni-multi=: This is an interactive DYNAMIC command that uses multiple sources, as defined by =consult-omni-multi-sources=, and shows the results in minibuffer completion with dynamic completion (results are fetched as the user types). Here is a screenshot:

#+ATTR_ORG: :width 800px #+ATTR_LATEX: :width 800px #+ATTR_HTML: :width 800px [[https://github.com/armindarvish/consult-web/blob/screenshots/screenshots/consult-omni-multi.gif]]

=consult-omni-multi-static=: This is an interactive STATIC command that uses multiple sources, as defined by =consult-omni-multi-sources=, and shows the results in minibuffer completion. Note that consult-omni-multi-static does not provide dynamic completion but some might find using this more intuitive for narrowing down the results. The user provides one search term, and once the results are retrieved, typing in the minibuffer will narrow down the candidates.

Note that unlike the predecessor consult-web, consult-omni does not provide other multi-source interactive commands (e.g. scholar search) anymore. It's now up to the user to define such commands for their custom workflow. For example one user may define separate interactive commands to query web sources, v.s. local sources, while another user may want to define interactive commands for knowledge sources (academic references, notes, ...) v.s. utilities (apps, buffers, files,...). Examples for how to define new interactive commands are provided below in the advanced config [[id:DFBCF797-44D6-4265-B737-77916700C43D][here]].

** Dynamic Completion: Passing Arguments and Narrow Down

Arguments can be passed to the dynamic interactive commands and further narrowing down the results can be done using a syntax similar to the “Perl splitting” style in [[https://github.com/minad/consult?tab=readme-ov-file#asynchronous-search][consult asynchronous search]].

For narrowing down the results you need adding =#= (or another character defined in =consult-async-split-style=) after the search query. For example typing the following in the minibuffer: #+begin_example #emacs web search#github #+end_example

First searches for “emacs web search”, and then uses “github” for narrow down.

Furthermore, arguments can be passed to dynamic commands using similar syntax as =consult-grep=, too. For example typing the following in the minibuffer:

#+begin_example #how to browse a url in emacs -- --model gpt-3.5-turbo #+end_example

passes =gpt-3.5-turbo= as the value for the keyword argument =:model= to the back-end functions of all the sources that fetch results. If any of those sources accept the keyword argument =:model=, the value =gpt-3.5-turbo= gets passed to them. For this reason it is recommended to always use functions that accept any keyword arguments (a.k.a. add =&allow-other-keys=) to avoid errors when non-existing keywords are passed to them.

instead of using =--= , you can also use a keyword with colon =:=. The following would be similar to the example above:

#+begin_example #how to browse a url in emacs -- :model gpt-3.5-turbo #+end_example

** Understanding different Type of Sources

When using consult, we deal with different types of sources, either elisp functions (e.g. =buffer-list=, =re-search-forward=) or command-line programs (e.g. =grep=, =notmuch=, =gh=, =mu=, ...). They return a list of candidates, which in turn are passed to =completing--read=. While consult provides a way to combine multiple sources with =consult--multi=, the ability to combine sources of different types (command-line program with elisp for example) is limited. In consult-omni we solve this by creating some wrappers and machinery around built-in consult functions and by having each source declare its type. This type tells consult-omni how to collect candidates from that source and combine with other sources. The type can be ='sync=, ='dynamic=, or ='async=.

='sync=: This is a synchronous source, meaning that when consult-omni calls the request function to collect candidates from this source, it calls it synchronously (blocks the Emacs process) and waits for a returned value (expecting a list of candidates or nil). This is for example suitable for sources that simply run an elisp function and return a list quickly like =buffer-list=. There is almost no gain in calling this function asynchronously because the overhead for async control (timers and watchers,...) is probably going to cost more time than just calling the function directly and synchronously.
='dynamic=: Dynamic sources are those that would benefit or require calling the collecting elisp function asynchronously. For example, if collecting items from a source requires to send an HTTP request to a server and waiting for the response, then a dynamic type should be used, so that Emacs process is not blocked while waiting for the response. In this case the request function for the source (the function that returns the items) should take a callback function that will be called when the response arrives. This is for example is suitable for getting response from web search APIs like Google's, Brave's or OpenAI's APIs.
='async=: asynchronous sources, are those that use an external command line program and therefore require to create process, get the output of the command line program and parse it to collect candidates. This is for example suitable for getting candidates from =grep=, or =notmuch=, ....

consult-omni machinery allows combining multiple sources of different types in one command by combining sync and async calls/processes so that the user can efficiently get search results from various sources in almost real-time (no waiting needed, the candidate are shown as they arrive).

** Embark Actions You can load the default embark actions by; #+begin_src emacs-lisp (require 'consult-omni-embark) #+end_src The default actions allow you to open the links in the default or alternate browser and also to copy or insert, title and/or URL of the links. Other embark actions can be defined per your own specific work flow.

See the YouTube video on consult-web for an example, here: [[https://youtu.be/7pDfyqBZwvo?t=4962]].

** Other Important Features *** Minimal Code Base Without doc-strings and whitespaces the code is less than 1000 lines and it only depends on [[https://github.com/minad/consult][consult]] and built-in url-retrieve.

*** Modular You can only load the parts you need. For example if all you need is an autosuggestion utility similar to =helm-google-autosuggest=, then you can use minimal config like this:

#+begin_src emacs-lisp (use-package consult-omni :straight (consult-omni :type git :host github :repo "armindarvish/consult-omni" :branch "main" :files (:defaults "sources/*.el")) :after consult :config ;; Load Sources Core code (require 'consult-omni-sources) ;; Load Embark Actions (require 'consult-omni-embark) ;; Only load brave-auto-suggest source (require 'consult-omni-brave-autosuggest) ;;; Set your shorthand favorite interactive command (setq consult-omni-default-interactive-command #'consult-omni-brave-autosuggest)) #+end_src

Note that every module (a.k.a. every source) adds an extra 100-200 lines of code. This also means to add a new source, you only need to write a short piece of code following those examples!

*** Customizable and Extendable Lots of customization options both for sources and also for running actions on the results. New sources can be added as you wish with different format, different actions,...

*** Power User Capabilities Dynamic collection allows for complex workflows on the fly. Change query parameters on the fly by passing arguments. Select a random set of results ad-hoc using embark and run embark actions on them. This allows batch processing as well. For example to add a long list of results to an org-mode note for later review (as shown in this YouTube video: [[https://youtu.be/7pDfyqBZwvo?t=4774]]).

Getting Started

** Precautions Before you start, make sure you understand three points:

Important Note 1: This is work in progress in its early stage and bugs and issues are very much expected.
Important Note 2: You should consider the general risks of using Emacs to browse the web. By default, all codes are trusted inside Emacs and browsers are naturally the target of many attacks. Therefore, it is important to be aware of the risks and be intentional about what links you open (or do not open) inside Emacs. By default, consult-omni would only be opening web pages (or calling APIs of) the sources (e.g. search engines, ...) and not any other websites. It's up to the user then to decide how she or he wants to open the links and chose their own risk tolerance. consult-omni provides customization variables for different actions (e.g. opening links in external browser v.s. in Emacs), so make sure you know how to set everything up.
Important Note 3: The functions provided in =consult-omni-sources=, provide a basic demonstration for integrating different services (such as search providers), however since each service comes with its own terms and conditions (that may change over time and vary from location to location), it is difficult to provide all-encompassing solutions and maintain them over time. consult-omni is agnostic of how you connect and integrate other services in your setup (because neither consult-omni nor Emacs collect any information of the users or their usage), and therefore ultimately only you the user are responsible for setting up everything correctly and understand consequences of the usage (e.g. costs of using paid APIs) and ensure to stay within the bounds of relevant laws and regulations for your use case (i.e. follow software user agreements, etc.). Therefore, it is important for you to read and understand how to use each service, and also understand what happens under the hood when you integrate the service with consult-omni. I try my best to provide documentation here as well as on the [[https://github.com/armindarvish/consult-omni/wiki][wiki pages]], and will try to help when possible but before you proceed understand that you do everything at your own risk.

** Installation If you want an example config see [[https://github.com/armindarvish/consult-omni?tab=readme-ov-file#drop-in-example-config][Drop-in Example Config]]. Here is some detailed explanation;

*** Requirements In order to use consult-omni, you need Emacs >28.0 (I have not tested earlier versions) and you need [[https://github.com/minad/consult][consult]]. While this is the only requirements, I suggest you read the rest of this README. I do recommend some other packages and useful configurations for different settings. Some of those extra packages and settings can improve your experience of consult-omni, therefore you may want to consider installing them as well. For example, combining consult with other packages such as [[https://github.com/minad/vertico][vertico]], [[https://github.com/oantolin/orderless][orderless]], and [[https://github.com/oantolin/embark][embark]] can improve the functionality as well as user-experience.

*** Installing consult-omni Package consult-omni is not currently on [[https://elpa.gnu.org/packages/consult.html][ELPA]] or [[https://melpa.org/#/consult][MELPA]]. Therefore, you need to install it using an alternative non-standard package manager such as [[https://github.com/radian-software/straight.el][straight.el]], [[https://github.com/progfolio/elpaca][elpaca]], ... or use manual installation.

**** straight.el To install consult-omni with straight.el you can use the following command. Make sure you load consult-omni after loading consult (e.g. =require 'consult=).

#+begin_src emacs-lisp (straight-use-package '(consult-omni :type git :host github :repo "armindarvish/consult-omni" :files (:defaults "sources/*.el"))) #+end_src

or if you