Commit graph

27 commits

Author SHA1 Message Date
jade
9896d309cb Revert "libexpr: Replace regex engine with boost::regex"
This reverts commit 447212fa65.

Reason for revert: Regression in eval behaviour bug-compatibility.

Expected behaviour (Nix 2.18.5, macOS and Linux [libstdc++/libc++]):

```
nix-repl> builtins.match "\\.*(.*)" ".keep"
[ "keep" ]

nix-repl> builtins.match "(\\.*)(.*)" ".keep"
[ "." "keep" ]
```

Actual behaviour (boost::regex):

```
nix-repl> builtins.match "\\.*(.*)" ".keep"
[ ".keep" ]

nix-repl> builtins.match "(\\.*)(.*)" ".keep"
[
  "."
  "keep"
]
```

Bug: https://git.lix.systems/lix-project/lix/issues/483
Change-Id: Id462eb8586dcd54856cf095f09b3e3a216955b60
2024-08-22 18:35:11 +00:00
sugar
447212fa65 libexpr: Replace regex engine with boost::regex
This avoids C++'s standard library regexes, which aren't the same
across platforms, and have many other issues, like using stack
so much that they stack overflow when processing a lot of data.

To avoid backwards and forward compatibility issues, regexes are
processed using a function converting libstdc++ regexes into Boost
regexes, escaping characters that Boost needs to have escaped, and
rejecting features that Boost has and libstdc++ doesn't.

Related context:

- Original failed attempt to use `boost::regex` in CppNix, failed due to
  boost icu dependency being large (disabling ICU is no longer necessary
  because linking ICU requires using a different header file,
  `boost/regex/icu.hpp`): https://github.com/NixOS/nix/pull/3826

- An attempt to use PCRE, rejected due to providing less backwards
  compatibility with `std::regex` than `boost::regex`:
  https://github.com/NixOS/nix/pull/7336

- Second attempt to use `boost::regex`, failed due to `}` regex failing
  to compile (dealt with by writing a wrapper that parses a regular
  expression and escapes `}` characters):
  https://github.com/NixOS/nix/pull/7762

Closes #34. Closes #476.

Change-Id: Ieb0eb9e270a93e4c7eed412ba4f9f96cb00a5fa4
2024-08-22 03:17:55 +02:00
piegames
278fddc317 libexpr: Deprecate URL literals
Closes #437.

Change-Id: I9f67fc965bb4a7e7fd849e5067ac1cb3bab064cd
2024-08-17 20:31:57 +02:00
piegames
49d61b2e4b libexpr: Introduce Deprecated features
They are like experimental features, but opt-in instead of opt-out. They
will allow us to gracefully remove language features. See #437

Change-Id: I9ca04cc48e6926750c4d622c2b229b25cc142c42
2024-08-17 19:47:51 +02:00
piegames
28ae24f3f7 libexpr: Add experimental pipe operator
The |> operator is a reverse function operator with low binding strength
to replace lib.pipe. Implements RFC 148, see the RFC text for more
details. Closes #438.

Change-Id: I21df66e8014e0d4dd9753dd038560a2b0b7fd805
2024-08-08 11:13:53 +02:00
piegames
ec7552ff74 libexpr/parser: Test experimental features
Currently, the parser relies on the global experimental feature flags.
In order to properly test conditional language features, we instead need
to pass it around in the parser::State.

This means that the parser cannot cache the result of isEnabled anymore,
which wouldn't necessarily hurt performance if the function didn't
perform a linear search on the list of enabled features on every single
call. While we could simply evaluate once at the start of parsing and
cache the result in the parser state, the more sustainable solution
would be to fix `isEnabled` such that all callers may profit from the
performance improvement.

Change-Id: Ic9b9c5d882b6270e1114988b63e6064d36c25cf2
2024-08-07 13:07:50 +00:00
Alois Wohlschlager
40c39aa5d2
libexpr/print: do not show elided nested items when there are none
When the configured maximum depth has been reached, attribute sets and lists
are printed with ellipsis to indicate the elision of nested items. Previously,
this happened even in case the structure being printed is empty, so that such
items do not in fact exist. This is confusing, so stop doing it.

Change-Id: I0016970dad3e42625e085dc896e6f476b21226c9
2024-07-18 18:41:34 +02:00
Alois Wohlschlager
b5da823138
libexpr/print: never show empty attrsets or derivations as «repeated»
The repeated value detection logic exists so that the occurrence of large
common substructures does not fill up the screen or the computer's memory.
However, empty attribute sets and derivations (when their detection is enabled)
are always cheap to print, and in practice I have observed them to make up a
significant majority of the cases where I was annoyed by the repeated value
detection kicking in. Furthermore, `nix-instantiate --eval` already disables
this logic for empty attribute sets, and empty lists are already exempted
everywhere. For these reasons, always print empty attribute sets and
derivations as what they are.

Change-Id: I5dac8e7739f9d726b76fd0521ec46f38af94463f
2024-07-18 18:41:34 +02:00
Qyriad
59bf6825ef add an impl of Expr::show for ExprInheritFrom that doesn't crash
ExprVar::show() assumes it has a name. dynamic inherits do not
necessarily (ever?) have a name.

Change-Id: If10893188e307431da17f0c1bd0787adc74f7141
2024-07-04 15:55:38 -06:00
eldritch horrors
bcb774688f libexpr: add expr memory management
with the prepatory work done this mostly means turning plain pointers
into unique_ptrs, with all the associated churn that necessitates. we
might want to change some of these to box_ptrs at some point as well,
but that would be a semantic change that isn't fully appropriate yet.

Change-Id: I0c238c118617420650432f4ed45569baa3e3f413
2024-06-17 19:46:44 +00:00
eldritch horrors
ad5366c2ad libexpr: pass Exprs as references, not pointers
almost all places where Exprs are passed as pointers expect the pointers
to be non-null. pass them as references to encode this constraint in the
type system as well (and also communicate that Exprs must not be freed).

Change-Id: Ia98f166fec3c23151f906e13acb4a0954a5980a2
2024-06-17 19:46:44 +00:00
Qyriad
b9be46fb31 remove the autoconf+Make buildsystem
We're not using it anymore. Any leftover bugs in the Meson buildsystem
are now just bugs.

Closes #249.

Change-Id: I0465a0c37ae819f94d40e7829f5bff046aa63d73
2024-05-07 17:04:30 -06:00
eldritch horrors
86a1121d16 use byte indexed locations for PosIdx
we now keep not a table of all positions, but a table of all origins and
their sizes. position indices are now direct pointers into the virtual
concatenation of all parsed contents. this slightly reduces memory usage
and time spent in the parser, at the cost of not being able to report
positions if the total input size exceeds 4GiB. this limit is not unique
to nix though, rustc and clang also limit their input to 4GiB (although
at least clang refuses to process inputs that are larger, we will not).

this new 4GiB limit probably will not cause any problems for quite a
while, all of nixpkgs together is less than 100MiB in size and already
needs over 700MiB of memory and multiple seconds just to parse. 4GiB
worth of input will easily take multiple minutes and over 30GiB of
memory without even evaluating anything. if problems *do* arise we can
probably recover the old table-based system by adding some tracking to
Pos::Origin (or increasing the size of PosIdx outright), but for time
being this looks like more complexity than it's worth.

since we now need to read the entire input again to determine the
line/column of a position we'll make unsafeGetAttrPos slightly lazy:
mostly the set it returns is only used to determine the file of origin
of an attribute, not its exact location. the thunks do not add
measurable runtime overhead.

notably this change is necessary to allow changing the parser since
apparently nothing supports nix's very idiosyncratic line ending choice
of "anything goes", making it very hard to calculate line/column
positions in the parser (while byte offsets are very easy).

(cherry picked from commit 5d9fdab3de0ee17c71369ad05806b9ea06dfceda)
Change-Id: Ie0b2430cb120c09097afa8c0101884d94f4bbf34
2024-03-18 16:12:46 +01:00
Jade Lovelace
886a418d23 builtins.nixVersion: return fixed fake version
This builtin is only going to cause us problems because we are not Nix,
so let's just falsify being in the 2.18 series, since that is the
closest target that has any meaning.

In future we might want to have a better feature detection mechanism,
for when we actually add stuff to some builtin's attr set argument. But
builtins.nixVersion is just going to be hopelessly broken and it should
be stubbed out.

Fixes https://git.lix.systems/lix-project/lix/issues/144

Change-Id: Id7390b32a29c6147f2977737d81846320de5d67e
2024-03-17 00:32:19 -07:00
eldritch horrors
1958152d14 Pretty-print values in the REPL
Pretty-print values in the REPL by printing each item in a list or
attrset on a separate line. When possible, single-item lists and
attrsets are printed on one line, as long as they don't contain a nested
list, attrset, or thunk.

Before:
```
{ attrs = { a = { b = { c = { }; }; }; }; list = [ 1 ]; list' = [ 1 2 3 ]; }
```

After:
```
{
  attrs = {
    a = {
      b = {
        c = { };
      };
    };
  };
  list = [ 1 ];
  list' = [
    1
    2
    3
  ];
}
```

(cherry picked from commit c0a15fb7d03dfb8f53bc6726c414bc88aa362592)
Change-Id: Ia2b41849165a5ddb63f7a8c272a2476b3e4292df
2024-03-09 07:20:23 -07:00
eldritch horrors
b221a14f0a Merge pull request #9925 from 9999years/fmt-cleanup
Cleanup `fmt.hh`

(cherry picked from commit 47a1dbb4b8e7913cbb9b4d604728b912e76e4ca0)
Change-Id: Id076a45cb39652f437fe3f8bda10c310a9894777
2024-03-09 07:00:13 -07:00
eldritch horrors
08252967a8 libexpr: Support structured error classes
While preparing PRs like #9753, I've had to change error messages in
dozens of code paths. It would be nice if instead of

    EvalError("expected 'boolean' but found '%1%'", showType(v))

we could write

    TypeError(v, "boolean")

or similar. Then, changing the error message could be a mechanical
refactor with the compiler pointing out places the constructor needs to
be changed, rather than the error-prone process of grepping through the
codebase. Structured errors would also help prevent the "same" error
from having multiple slightly different messages, and could be a first
step towards error codes / an error index.

This PR reworks the exception infrastructure in `libexpr` to
support exception types with different constructor signatures than
`BaseError`. Actually refactoring the exceptions to use structured data
will come in a future PR (this one is big enough already, as it has to
touch every exception in `libexpr`).

The core design is in `eval-error.hh`. Generally, errors like this:

    state.error("'%s' is not a string", getAttrPathStr())
      .debugThrow<TypeError>()

are transformed like this:

    state.error<TypeError>("'%s' is not a string", getAttrPathStr())
      .debugThrow()

The type annotation has moved from `ErrorBuilder::debugThrow` to
`EvalState::error`.

(cherry picked from commit c6a89c1a1659b31694c0fbcd21d78a6dd521c732)
Change-Id: Iced91ba4e00ca9e801518071fb43798936cbd05a
2024-03-09 04:47:05 -07:00
eldritch horrors
7673312ccc Merge pull request #9928 from 9999years/error-messages-in-nix-repl
Improve error printing in `nix repl`

(cherry picked from commit a8050d9b83052e4b5c52bf2d116381aedec3a93e)
Change-Id: I588f92d1dd4c546c98788b71403cc034f5e7129a
2024-03-09 03:37:35 -07:00
eldritch horrors
c864923928 Merge pull request #9929 from 9999years/dont-print-values-in-magenta
Don't print the first bracket in values in magenta in error messages

(cherry picked from commit 46a0625a40aef6946a35f92fdacf0e6b4a14414f)
Change-Id: I8435565c87db182116140eaeea9df1243e67ea94
2024-03-09 03:37:35 -07:00
eldritch horrors
609a8e0d94 Merge pull request #9754 from 9999years/print-value-when-coercion-fails
Print the value in `error: cannot coerce` messages

(cherry picked from commit 5b7bfd2d6b89d7dd5f54c1ca6c8072358d31a84e)

===

test taken from 6e8d5983143ae576e3f4b1d2954a5267f2943a49; it was added
previously (and not backported because its pr was a mostly-revert), but
it's useful to have around.

Change-Id: Icbd14b55e3610ce7b774667bf14b82e6dc717982
2024-03-09 00:05:41 -07:00
eldritch horrors
2f7c3fa251 Merge pull request #9818 from Ma27/print-value-on-function-call-type-error
libexpr: print value of what is attempted to be called as function
(cherry picked from commit 50e5d7b883042852538371237e32a66bb22f0485)
Change-Id: I7cb6290bd8f244e83bfce3b2eed2a4c8b4f16a83
2024-03-09 00:05:41 -07:00
eldritch horrors
87e6ac5eb7 Merge pull request #9753 from 9999years/print-value-on-type-error
Print the value in `value is X while a Y is expected` error

(cherry picked from commit 5f72a97092da6af28a7d2b2a50d74e9d34fae7e1)
Change-Id: Idb4bc903ae59a0f5b6fb3b1da4d47970fe0a6efe
2024-03-09 00:05:41 -07:00
eldritch horrors
512c1f05c3 Unify and refactor value printing
Previously, there were two mostly-identical value printers -- one in
`libexpr/eval.cc` (which didn't force values) and one in
`libcmd/repl.cc` (which did force values and also printed ANSI color
codes).

This PR unifies both of these printers into `print.cc` and provides a
`PrintOptions` struct for controlling the output, which allows for
toggling whether values are forced, whether repeated values are tracked,
and whether ANSI color codes are displayed.

Additionally, `PrintOptions` allows tuning the maximum number of
attributes, list items, and bytes in a string that will be displayed;
this makes it ideal for contexts where printing too much output (e.g.
all of Nixpkgs) is distracting. (As requested by @roberth in
https://github.com/NixOS/nix/pull/9554#issuecomment-1845095735)

Please read the tests for example output.

Future work:
- It would be nice to provide this function as a builtin, perhaps
  `builtins.toStringDebug` -- a printing function that never fails would
  be useful when debugging Nix code.
- It would be nice to support customizing `PrintOptions` members on the
  command line, e.g. `--option to-string-max-attrs 1000`.

(cherry picked from commit 0fa08b451682fb3311fe58112ff05c4fe5bee3a4, )

===

Restore ambiguous value printer for `nix-instantiate`

The Nix team has requested that this output format remain unchanged.
I've added a warning to the man page explaining that `nix-instantiate
--eval` output will not parse correctly in many situations.

(cherry picked from commit df84dd4d8dd3fd6381ac2ca3064432ab31a16b79)

Change-Id: I7cca6b4b53cd0642f2d49af657d5676a8554c9f8
2024-03-09 03:50:06 +01:00
eldritch horrors
7614aa9797 Merge pull request #4093 from matthewbauer/eval-system
Add eval-system option

(cherry picked from commit 071dbbee33af9f27338c3e53e4ea067dbfa14010)
Change-Id: Ia81358c8cfb60241da07a4d0e84b9ee62a18a53f
2024-03-04 07:21:01 +01:00
eldritch horrors
3f3badffc9 Merge pull request #9395 from nbraud/buitlins
builtins.concatMap: Fix typo in error message

(cherry picked from commit 4292d997568eb30503e287f98e24821ff0bc2816)
Change-Id: Ia33d1b02e41f699ef0c8c2d6487c9f70b2cc8cf4
2024-03-04 05:56:04 +01:00
eldritch horrors
5e182235cb Merge pull request #7348 from thufschmitt/dont-use-vlas
Remove the usage of VLAs in the code

(cherry picked from commit ac4431e9d016e62fb5dc9ae36833bd0c6cdadeec)
Change-Id: Ifbf5fbfc2e27122362a2aaea4b62c7cf3ca46b1a
2024-03-04 05:51:23 +01:00
John Ericson
f7f37035c8 Move tests to separate directories, and document
Today, with the tests inside a `tests` intermingled with the
corresponding library's source code, we have a few problems:

- We have to be careful that wildcards don't end up with tests being
  built as part of Nix proper, or test headers being installed as part
  of Nix proper.

- Tests in libraries but not executables is not right:

  - It means each executable runs the previous unit tests again, because
    it needs the libraries.

  - It doesn't work right on Windows, which doesn't want you to load a
    DLL just for the side global variable . It could be made to work
    with the dlopen equivalent, but that's gross!

This reorg solves these problems.

There is a remaining problem which is that sibbling headers (like
`hash.hh` the test header vs `hash.hh` the main `libnixutil` header) end
up shadowing each other. This PR doesn't solve that. That is left as
future work for a future PR.

Co-authored-by: Valentin Gagarin <valentin.gagarin@tweag.io>

(cherry picked from commit 91b6833686a6a6d9eac7f3f66393ec89ef1d3b57)
(cherry picked from commit a61e42adb528b3d40ce43e07c79368d779a8b624)
2023-12-01 13:05:03 -05:00