There, hyperscan is generally the king, which means hypergrep numbers are likely accurate: https://github.com/p-ranav/hypergrep?tab=readme-ov-file#dire...

Disclaimer: I rarely use any *grep utilities, but often implement string libraries.

Your claim is true to a first approximation. But greps are line oriented, and that means there are optimizations that can be done that are hard to do in a general regex library. You can read more about that here: https://blog.burntsushi.net/ripgrep/#anatomy-of-a-grep (greps are more than simple CLI wrappers around a regex engine).

If you read my commentary in the ripgrep discussion above, you'll note that it isn't just about the benchmarks themselves being accurate, but the model they represent. Nevertheless, I linked the hypergrep benchmarks not because of Hyperscan, but because they were done by someone who isn't the author of either ripgrep or ugrep.

As for regex benchmarks, you'll want to check out rebar: https://github.com/BurntSushi/rebar

You can see my full thoughts around benchmark design and philosophy if you read the rebar documentation. Be warned though, you'll need some time.

There is a fork of ripgrep with Hyperscan support: https://sr.ht/~pierrenn/ripgrep/

Hyperscan also has some preculiarities on how it reports matches. You won't notice it in basic usage, but it will appear when using something like the -o/--only-matching flag. For example, Hyperscan will report matches of a, b and c for the regex \w+, where as a normal grep will just report a match of abc. (And this makes sense given the design and motivation for Hyperscan.) Hypergrep goes to some pain to paper over this, but IIRC the logic is not fully correct. I'm on mobile, otherwise I would link to the reddit thread where I had a convo about this with the hypergrep author.

From some searching I think you might mean this: https://www.reddit.com/r/cpp/comments/143d148/hypergrep_a_ne...

    $ echo 'foobar' | hg -o '\w{3}'
    1:foobar
    $ echo 'foobar' | grep -E -n -o '\w{3}'
    1:foo
    1:bar

I want to be clear that these are intended semantics as part of Hyperscan. It's not a bug with Hyperscan. But it is something you'll need to figure out how to deal with (whether that's papering over it somehow, although I'm not sure that's possible, or documenting it as a difference) if you're building a grep around Hyperscan.

I also wonder if there's a performance issue when matching a really long line because Hyperscan is not greedy and will ping back to Hypergrep for every sub match. I guessing this is the reason for those shenanigans in the callback [0].

  $ python -c 'print("foo" + "bar" * 3000)' | hg -o 'foo.*bar'

It's hard to learn too much from hypergrep. It still has some rough spots:

    $ hgrep -o 'foo.*bar' foobarbar.txt
    foobarbar.txt
    1:[Omitted long line with 1 matches]

    $ hgrep -M0 -o 'foo.*bar' foobarbar.txt
    Too few arguments

    For more information try --help
    $ hgrep -M 0 -o 'foo.*bar' foobarbar.txt
    foobarbar.txt
    1:[Omitted long line with 1 matches]

    $ hgrep -M 0 'foo.*bar' foobarbar.txt
    foobarbar.txt
    1:[Omitted long line with 1 matches]

    $ hgrep -M0 'foo.*bar' foobarbar.txt
    terminate called after throwing an instance of 'std::invalid_argument'
      what():  pattern not found
    zsh: IOT instruction (core dumped)  hgrep -M0 'foo.*bar' foobarbar.txt

> This flag instructs Hyperscan to treat the pattern as a sequence of UTF-8 characters. The results of scanning invalid UTF-8 sequences with a Hyperscan library that has been compiled with one or more patterns using this flag are undefined.

That's another issue you'll need to grapple with if you use Hyperscan. PCRE2 used to have this issue[4], but they've since defined the semantics of searching invalid UTF-8 with Unicode mode enabled. ripgrep 14 uses that new mode, but I haven't updated that FAQ answer yet.

Hyperscan isn't alone. Many regex engines do not support searching arbitrary byte sequences[5]. And this is why many/most regex engines are awkward to use in a fast grep implementation. Because you really do not want your grep to fall over when it comes across invalid UTF-8. And the overhead of doing UTF-8 checking in the first place (and perhaps let you just skip over lines that contain invalid UTF-8) would make it difficult to be competitive in performance. It also inhibits its usage in OSINT work.

[1]: https://intel.github.io/hyperscan/dev-reference/api_files.ht...

[2]: https://github.com/p-ranav/hypergrep/blob/ee85b713aa84e0050a...

[3]: https://github.com/p-ranav/hypergrep/blob/ee85b713aa84e0050a...

[4]: https://github.com/BurntSushi/ripgrep/blob/master/FAQ.md#why...

[5]: https://github.com/BurntSushi/rebar/blob/96c6779b7e1cdd850b8...

You could perhaps work around this by only applying it as an optimization when you know the pattern has identical semantics in both regex engines. But you would have to do the work to characterize them.

I would rather just make the regex crate faster. If you look at the rebar benchmarks, it's not that far behind and is sometimes even faster. The case where Hyperscan really destroys everything else is for searches for many patterns.

Hyperscan has other logistical issues. It is a beast to build. And its pattern compilation times can be large (again, see rebar). Hyperscan itself only supports x86-64, so one would probably want to actually use Vectorscan (a fork of Hyperscan that supports additional architectures).

Would be interesting to see comparisons where memory is limited, i.e., where the file being searched will not fit entirely into memory.

Personally I'm interested in "grep -o" alternatives. The files I'm searching are text but may have few newlines. For example I use ired instead of grep -o. ired will give the offsets of all matches, e.g.,

      echo /\"something\"|ired -n 1.htm

      #!/bin/sh
      test $# -gt 0||echo "usage: echo string|${0##*/} file [blocksize] [seek] [match-no]"
      {
      read x;
      x=$(echo /\""$x"\"|ired -n $1|sed -n ${4-1}p); 
      test "$x"||exit 1;
      echo
      printf s"$x"'\n's-${3-0}'\n'x$2'\n'|ired -n $1;
      echo;
      printf s"$x"'\n's-${3-0}'\n'X$2'\n'|ired -n $1;
      echo;
      echo w$(printf s"$x"'\n's-${3-0}'\n'X$2'\n'|ired -n $1)|ired -n /dev/stdout;
      echo;
      }

Which test exactly? That's just likely because of memory maps futzing with the RSS data. Not actually more heap memory. Try with --no-mmap.

I'm not sure I understand the rest of your comment about grep -o. Grep tools usually have a flag to print the offset of each match.

EDIT: Now that I have hands on a keyboard, I'll demonstrate the mmap thing. First, ugrep:

    $ time ugrep-4.4.1 -c '\w+\s+Sherlock\s+Holmes\s+\w+' sixteenth.txt
    72

    real    22.115
    user    22.015
    sys     0.093
    maxmem  30 MB
    faults  0
    $ time ugrep-4.4.1 -c '\w+\s+Sherlock\s+Holmes\s+\w+' sixteenth.txt --mmap
    72

    real    21.776
    user    21.749
    sys     0.020
    maxmem  802 MB
    faults  0

    $ time rg-14.0.3 -c '\w+\s+Sherlock\s+Holmes\s+\w+' opensubtitles/2018/en/sixteenth.txt
    72

    real    0.076
    user    0.046
    sys     0.030
    maxmem  779 MB
    faults  0
    $ time rg-14.0.3 -c '\w+\s+Sherlock\s+Holmes\s+\w+' opensubtitles/2018/en/sixteenth.txt --no-mmap
    72

    real    0.087
    user    0.033
    sys     0.053
    maxmem  15 MB
    faults  0

If the file were bigger than available memory, then the OS would automatically handle paging.

    # stat -c %s file
    6297285

    # file file
    file: ASCII text, with very long lines (1545), with CRLF, LF line terminators

1. Printing byte offsets.

    # time grep -ob string file

    0.03user 0.08system 0:00.22elapsed 52%CPU (0avgtext+0avgdata 1104maxresident)k
    0inputs+0outputs (0major+86minor)pagefaults 0swaps

    # rg -V
    ripgrep 13.0.0
   
    # time rg -ob string file

    0.10user 0.17system 0:01.11elapsed 25%CPU (0avgtext+0avgdata 7804maxresident)k
    0inputs+0outputs (0major+559minor)pagefaults 0swaps

    # time sh -c "echo -n string|od -An -tx1|sed 's>^>/>;s/ //g'|ired -n file"

    0.03user 0.09system 0:00.18elapsed 67%CPU (0avgtext+0avgdata 720maxresident)k
    0inputs+0outputs (0major+189minor)pagefaults 0swaps

Baseline.

     # time grep -o string file

     0.02user 0.07system 0:00.15elapsed 65%CPU (0avgtext+0avgdata 1068maxresident)k
     0inputs+0outputs (0major+84minor)pagefaults 0swaps

     # time grep -o .string file

     0.21user 0.08system 0:00.36elapsed 83%CPU (0avgtext+0avgdata 1088maxresident)k
     0inputs+0outputs (0major+87minor)pagefaults 0swaps

     # time grep -o ..string file

     0.29user 0.09system 0:00.46elapsed 82%CPU (0avgtext+0avgdata 1064maxresident)k
     0inputs+0outputs (0major+88minor)pagefaults 0swaps

     # time rg -o ..string file

     0.13user 0.13system 0:00.90elapsed 28%CPU (0avgtext+0avgdata 9012maxresident)k
     0inputs+0outputs (0major+574minor)pagefaults 0swaps

Now let's try ired. Another shell script. This one will print all occurences of string.

     cat > 1.sh 

     # echo string|time sh 1.sh 6

     0.11user 0.10system 0:00.17elapsed 127%CPU (0avgtext+0avgdata 772maxresident)k
     0inputs+0outputs (0major+466minor)pagefaults 0swaps

     # echo string|time sh 1.sh 7 1

     0.12user 0.09system 0:00.16elapsed 131%CPU (0avgtext+0avgdata 740maxresident)k
     0inputs+0outputs (0major+473minor)pagefaults 0swaps

     # echo string|time sh 1.sh 8 2

     0.12user 0.11system 0:00.20elapsed 112%CPU (0avgtext+0avgdata 744maxresident)k
     0inputs+0outputs (0major+461minor)pagefaults 0swaps

But ired is faster at printing matches with context. (NB. Context here means characters, not lines.)

Try using ripgrep to print offsets for ired.

    #!/bin/sh
    read x; 
    rg --no-mmap -ob $x $1 \
    |cut -d: -f1 \
    |sed "s/.*/s&@s-${3-0}@X$2/" \
    |tr @ '\12' \
    |ired -q -i /dev/stdin $1 \
    |sed 's/.*/w&0a/' \
    |ired -n /dev/stdout

    # time sh -c "echo string|1.sh file 8 2"

    0.11user 0.06system 0:00.18elapsed 101%CPU (0avgtext+0avgdata 5972maxresident)k
    0inputs+0outputs (0major+905minor)pagefaults 0swaps

    # stat -c %s /usr/bin/ired /usr/bin/grep /usr/bin/rg

    37544
    219248
    5074800

- https://github.com/acheronfail/repgrep

- https://github.com/konradsz/igrep

- https://github.com/seg-mx/grep_tui

- https://github.com/Robertleoj/grepedit

[1]: https://github.com/junegunn/fzf/blob/master/ADVANCED.md#usin...

Unfortunately... I just tried the indexer and it's extremely slow on my machine. It took 86 seconds to index a Linux kernel tree, while csearch's cindex tool took 8 seconds.

[1] https://github.com/BurntSushi/ripgrep/issues/1497

“Indexing adds a hidden index file ._UG#_Store to each directory indexed.”

“Re-indexing is incremental, so it will not take as much time as the initial indexing process.”

https://github.com/hound-search/hound

In zsh, the closest thing I've gotten to this was to bind Ctrl-\ to the `accept-and-hold` zle widget, which executes what is in the current buffer while still retaining it and the cursor position. That gets me close (no more ^P^B^B^B^B for editing), but I'd much rather see the result of the pipeline in real-time rather than having to manually hit a key whenever I want to see the result.

    $ grep -V | head -n1
    grep (GNU grep) 3.11
    $ alias ugrep-grep="ugrep-4.4.1 -G -U -Y -. --sort -Dread -dread"
    $ echo 'pokémon' | LC_ALL=en_US.UTF-8 grep 'pok[[=e=]]mon'
    pokémon
    $ echo 'pokémon' | LC_ALL=en_US.UTF-8 ugrep-grep 'pok[[=e=]]mon'
    $ echo 'γ' | LC_ALL=en_US.UTF-8 grep -i 'Γ'
    γ
    $ echo 'γ' | LC_ALL=en_US.UTF-8 ugrep-grep -i 'Γ'

    $ grep -V
    grep (BSD grep, GNU compatible) 2.6.0-FreeBSD
    $ echo 'pokémon' | LC_ALL=en_US.UTF-8 grep 'pok[[=e=]]mon'
    pokémon
    $ echo 'γ' | LC_ALL=en_US.UTF-8 grep -i 'Γ'
    γ

I didn't do XDG because this route seemed simpler, and XDG isn't something that is used everywhere.

Assuming you aren't doing weird things with paths, I can work around 'dumb lazy' developers releasing half-assed tools with symlinks/junctions, but I really don't want to spend a ton of time configuring your tool or fighting its presumptions.

Good luck. You're already off to a rough start with XDG, since that isn't what is used on Windows. And it's unclear whether it ought to be used on macOS.

Simpler how? This requires custom config, instead of following what I set system-wide.

> and XDG isn't something that is used everywhere.

Yeah, that‘s why it defines defaults to fall back on.

No, you don't understand. I'm not saying The XDG variables might not be defined. Give me a little credit here lol. I have more than a passing familiarity with XDG. I've implemented it before. I'm saying the XDG convention itself may not apply. For example, Windows. And its controversial whether to use them on macOS when I last looked into it.

I don't see any significant problem with defining an environment variable. You likely already have dozens defined. I know I do.

I'm not trying to convince you of anything. Someone asked why. This is why for ripgrep at least.

Bottom line is that while ripgrep doesn't follow XDG, it also doesn't force you to litter your HOME directory. That's what most people care about in my experience.

I would encourage you to search the ripgrep issue tracker for XDG. This has all been discussed.

Really most people want it in their path and it just to work in as many scenarios as possible. Config almost shouldn't be the responsibility of the tool at all... (Options passed to the tool via env variables, perhaps)...

Ripgrep is also quite a large project (judging on both star count and contribution cout), so people probably volunteer to support their platform/package manager of choice.

https://github.com/Genivia/ugrep#fuzzy

(I've been using ripgrep for quite some time now, how does this otherwise compare to it? would I be able to just replace rg with ug?)

For regular use, I use ugrep’s %u option with its format feature to only get one match per line same as other grep tools.

Overall, I’m a happy user of ugrep. ugrep works as well as ripgrep for me. It’s VERY fast and has built-in option to search archives within archives recursively.

I currently use ripgrep-all (which can search into anything, video captions or pdfs) and fzf.

They are more or less equivalent. One has obscure feature X other has obscure feature Y, one is a bit faster on A, other is a bit faster on B, the defaults are a bit different, and one is written in Rust, the other in C++.

Pick the one you like, or both. I have both on my machine, and tend to use the one that does what I want with the least options. I also use GNU grep when I don't need the speed or features of either ug and rg.

Ugrep not only has a pager built in, but it also allows searching the results which is super nice! And that feature works on all supported platforms!

Then again I'm on Linux and can always pipe to less if I need to. I'm also not the target audience for ugrep because I've never noticed that grep would be slow. :shrug:

Git obeys that value, and I would hope that most other UNIXy terminal apps do too.

    $ cat ~/bin/rgp
    #!/bin/sh
    exec rg -p "$@" | less -RFX

   fan of your tool. like it's speed and defaults.

I use cygwin and WSL for my unix needs. I have more and less in cygwin for use in windows.

bat is fancy pager written in Rust. It's on GitHub: https://github.com/sharkdp/bat

Two pallets that should work on Windows are https://github.com/walles/moar (golang) and https://github.com/markbt/streampager (Rust). There might also be a newer one that uses rust, I'm unsure.

https://github.com/noborus/ov

bat on Windows does page, but I believe it's only available on Choco and not winget.

Cargo is one of the main reasons to use Rust of C++. I am pretty sure there is more involved with C++ than this:

   rustup target add x86_64-unknown-linux-musl 
   cargo build --target=x86_64-unknown-linux-musl

For an up-to-date performance comparison of the latest ugrep, please see the ugrep performance benchmarks [at https://github.com/Genivia/ugrep-benchmarks]. Ugrep is faster than GNU grep, Silver Searcher, ack, sift. Ugrep's speed beats ripgrep in most benchmarks.

Either way, ripgrep is awesome and I’m staying with it.

To me the regex langage is very much a reason to not use grep.

If you consider it "the weirdest ever", I'm guessing that I'm probably older than you. I've certainly been using regex long before PCRE became common.

As a vim user I compose 10s if not 100s of regexes a day. It does not use PCRE. Nor does sed, a tool I've been using for decades. Do you also recommend not using these?

ripgrep, Python, JavaScript, and practically every other non-C language: \b

vim: \>

BSD sed: [[:>:]]

GNU sed, GNU grep: \> or \b

BSD grep: \>, \b, or [[:>:]]

less: depends on the OS it's running on

And then there's \p{Letter}. It can be spelled in a lot of ways: \pL, \p{L}, \p{Letter}, \p{gc=Letter}, \p{gc:Letter}, \p{LeTtEr}. All equivalent. Very few regex engines support all of them. Several support \p{L} but not \pL. See: https://github.com/BurntSushi/rebar/blob/f9a4f5c9efda069e798...

Also, \ are in ripgrep 14. Although you usually just want to use the -w/--word-regexp flag.

Isn't that inconsistent with the way Perl's regex syntax was designed? In Perl's syntax an escaped non-ASCII character is always a literal [^1], and that is guaranteed not to change.

That's nice for beginners because it saves you from having to memorize all the metacharacters. If you are in doubt you on whether something has a special meaning, you just escape it.

[^1]: https://perldoc.perl.org/perlrebackslash#The-backslash

With that said, I do like Perl's philosophy here. And it was my philosophy too up until recently. I decided to make an exception for \ given their prevalence.

It was also only relatively recently that I made it possible for superfluous escapes to exist. Prior to ripgrep 14, unrecognized escapes were forbidden:

    $ echo '@' | rg-13.0.0 '\@'
    regex parse error:
        \@
        ^^
    error: unrecognized escape sequence
    $ echo '@' | rg '\@'
    @

`\` are forever and always the lone exceptions to this. It is perhaps a trap for beginners, but there are many traps in regexes, and this seemed worth it.

Note that `\b{start}` and `\b{end}` also exist and are aliases for `\`. The more niche `\b{start-half}` and `\b{end-half}` also exist, and those are what are used to implement the -w/--word-regexp flag. (Their semantics match GNU grep's -w/--word-regexp.) For example, `\b-2\b` will not match in `foo -2 bar` since `-` is not a word character and `\b` demands `\w` on one side and `\W` on the other. However, `rg -w -e -2` will match `-2` in `foo -2 bar`:

    $ echo 'foo -2 bar' | rg -w -e '\b-2\b'
    $ echo 'foo -2 bar' | rg -w -e -2
    foo -2 bar

I recall using hyperscan but isn't it discontinued.

Specifically -P / --perl-regexp support on MacOS and FreeBSD

It really would reduce the WTF moments for the students.

Insert jokes about standards below... =)

It's easier IMO to just use the same tool on all platforms. Which you can of course do.

Cheers, =)

POSIX also forbids greps from searching UTF-16 because it mandates that certain characters always use a single byte. ripgrep, for example, doesn't have this constraint and thus can transparently search UTF-16 correctly via BOM sniffing.

I will not even contemplate using this tool.

The reason is very simple: I can trust 'grep' to be on any system I ever touch. Learning ugrep doesn't make any sense as I can't trust it to be available.

I could still use it on my own systems, but I work on customer systems which won't have this tool installed.

And I'm proficient enough with grep that it's 'good enough', I'm not focussing on a better grep. I'm focussing on fixing a problem, or trying something new.

I'd rather invest my time into something that will benefit me across all environments I work with.

Because a tool may be 'better' (whatever that means) doesn't mean it will see adoption.

This is not about being closeminded, but it's about focus on what's really important.

On the other hand for a non-work environment where security isn't in question this is cool.

Why did you build a new grep?

No snark here, but is the subtitle not enough to start? "a more powerful, ultra fast, user-friendly, compatible grep"

* a more powerful -- This is meaningless without some sort of examples. Powerful how? What does it do that's better than grep?

* ultra fast -- This at least means something, but it should be quantified in some way. "50%+ faster for most uses cases" or something like that.

* user-friendly -- not even sure what this means. Seems kind of subjective anyway. I find grep plenty user friendly, for a command line tool.

* compatible grep -- I mean, they all are pretty much, but I guess it's good to know this?

That would be begging for nerd rage posts, just like so many disputing the benchmarks. >:D

> * user-friendly -- not even sure what this means. Seems kind of subjective anyway. I find grep plenty user friendly, for a command line tool.

Just below is a huge, captioned screenshot of the TUI?

> * compatible grep -- I mean, they all are pretty much, but I guess it's good to know this?

One would think so... but I have so many scars concerning incompatibilities with different versions of grep (as do others in the comments). If you don't know, then that feature isn't listed for you. :)

regular expressions are not friendly, but the user friendly way for a cli filter to behave is to return retvals appropriately, output to stdout, error messages to stderr... does user friendly mean copious output to stderr? what else could it possibly mean? do I want copious output to stderr?

Granted, it is far from a thing of beauty, but there is a large, captioned screenshot of the included text user interface just beneath. Then again, it is a website for a command line tool. "Many Bothans died to bring us this information."