Commit graph

37 commits

Author SHA1 Message Date
adisbladis
b7ec621e91
jquery & jquery-ui: Move to nixos/lib/testing/
These (outdated) derivations are only used by nixos/lib/testing.nix.

If we want to provide jquery & jquery-ui packages this is better done
in nodePackages.
2018-02-27 20:07:35 +08:00
Michael Fellinger
8c9a8d6805 Document the makeTest length restriction 2018-02-20 16:36:16 +02:00
Michael Fellinger
0d988d6735 Only allow test names of up to 50 characters 2018-02-20 16:36:16 +02:00
Franz Pletz
54c7ca34c4
nixos/testing: use the same qemu in the test driver
The qemu_test package is also used for running a NixOS qemu VM.
2018-02-11 23:40:54 +01:00
Shea Levy
943592f698
Add setFunctionArgs lib function.
Among other things, this will allow *2nix tools to output plain data
while still being composable with the traditional
callPackage/.override interfaces.
2018-01-31 14:02:19 -05:00
Orivej Desh
50d8720d2a nixos/testing: fix after --set ceased to evaluate the value 2017-11-19 14:15:19 +00:00
aszlig
20487112ed
nixos: Fix output path generation of runInMachine
Regression introduced by a02bb00156.

The fix is done by disabling writableStore, because the latter will set
up an overlayfs on the Nix store within the VM, which in turn will
discard all the outputs of the resulting output path.

However in runInMachine we actually *want* the contents of the generated
path and also don't want a writable store within the VM (except of
course for $out, which is writable anyway).

I've added a small regression test to verifify the output in
nixos/tests/run-in-machine.nix to make sure this won't break again in
the future.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2017-10-11 20:08:11 +02:00
aszlig
a443bdc0a6
nixos/testing: Improve quality of OCR
First of all, we're now using ImageMagick to improve the screenshot so
that Tesseract has an esier time to recognize the text. The resulting
image of this post-processing is a scaled up black-and-white version
with the backgrounds almost entirely removed and the text edges a bit
blurred, so the screen shots now more or less resemble an image from a
scanner rather. This is what Tesseract is trained for by default.

As mentioned in the previous commit we now also use Tesseract 4, which
further improves the quality of text recognition.

I've spent countless hours just to test different postprocessing
variants and testing what works best for our tests and this is the one
that worked best so far. It's certainly not perfect and I'd like to
avoid the scaling step but we're way better off than before.

In addition to this, the OCR process is now done without an intermediate
file, solely using pipes.

I've tested this using the following VM tests which have OCR enabled:

 * nixos/tests/chromium.nix -A stable
 * nixos/tests/emacs-daemon.nix
 * nixos/tests/installer.nix -A luksroot
 * nixos/tests/lightdm.nix
 * nixos/tests/plasma5.nix
 * nixos/tests/sddm.nix

All of the tests still succeed and comparing some of the recognition
results to the earlier results it now also detects a lot more text than
before this commit.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2017-04-11 03:21:53 +02:00
aszlig
288a79187c
tesseract: Reintroduce enableLanguages
I've removed that attribute in 68bc260ca2,
because the language files no longer were distributed as seperate files,
but if we for example only want to use the English training data, the
closure size of Tesseract gets quite large (around 1.2 GB), which is a
bit much just to be able to run NixOS VM tests.

For this reason I've also switched the VM tests back to using only the
English language.

Tested using the following VM tests (the ones that have OCR enabled) on
x86_64-linux:

 * nixos/tests/chromium.nix -A stable
 * nixos/tests/emacs-daemon.nix
 * nixos/tests/installer.nix -A luksroot
 * nixos/tests/lightdm.nix
 * nixos/tests/plasma5.nix
 * nixos/tests/sddm.nix

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2017-04-11 03:21:26 +02:00
Nikolay Amiantov
1f0ce0eaf8 makeTest: don't rely on makeWrapper not quoting newlines 2017-03-20 16:15:38 +03:00
Eelco Dolstra
f173da375d
Use only one build of qemu in VM tests
Previously we were using two or three (qemu_kvm, qemu_test, and
qemu_test with a different dbus when minimal.nix is included).

(cherry picked from commit 8bfa4ce82ea7d23a1d4c6073bcc044e6bf9c4dbe)
2016-12-20 10:52:46 +01:00
Eelco Dolstra
aad5d1f9a7
virtualisation.qemu.program: Remove
This option is defined in qemu-vm.nix, but that module is not always
imported.

http://hydra.nixos.org/build/44817443
(cherry picked from commit 03c55005dfd6fbcd5cf8e00128a3bb6336b3bc0f)
2016-12-20 10:52:46 +01:00
aszlig
68bc260ca2
tesseract: 3.02.02 -> 3.04.01
From the upstream changelog:

 * Tesseract development is now done with Git and hosted at github.com
   (Previously we used Subversion as a VCS and code.google.com for
   hosting).

So let's move over to the GitHub repository, where the organisation also
includes a full repository for tessdata, so we no longer need to fetch
it one-by-one.

The build also got significantly simpler, because we no longer need to
run autoconf, neither do we need to patch the configure script for
Leptonica headers.

This also has the advantage that we don't need to use the
enableLanguages attribute for the test runner anymore.

Full upstream changelog can be found at:

https://github.com/tesseract-ocr/tesseract/blob/c4d273d33cc36e/ChangeLog

Tested against all NixOS tests with enabled OCR (chromium, emacs-daemon,
installer.luksroot and lightdm).

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Cc: @viric
2016-12-19 22:25:38 +01:00
aszlig
b4e2b6bc6a
nixos/lib/testing: Fix unsetting $xchg
Regression introduced by 4dcb685af9.

Unsetting the environment variable shortly before using it is not going
to end up very well, so let's just filter out the variable from the
output of export and unset it shortly afterwards.

This fixes the runInMachine NixOS test.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2016-09-11 17:31:12 +02:00
Eelco Dolstra
4dcb685af9 runInMachine: Unset another variable
Thanks @cstrahan.
2016-09-09 20:02:56 +02:00
Eelco Dolstra
75baee8523 runInMachine: Support passAsFile
We need to rewrite attributes passed via files to their location in
/tmp/xchg in the VM. Otherwise functions like runCommand don't work.
2016-09-08 15:38:56 +02:00
Tuomas Tynkkynen
282277dbc8 treewide: Use more makeBinPath 2016-08-23 01:18:10 +03:00
Vladimír Čunát
61b8d27b4c make-wrapper: fixup #14753 changes to make tests work
The tests need to expand passed variable and very carefully.
I could see no other easy way than to change single-quoting in
makeWrapper to double-quoting.
The tests now fail with the same problem as on master...
2016-06-19 13:38:46 +02:00
Domen Kožar
c56c3b6596 nixos/lib/testing.nix: make 'config' a free variable
cc @edolstra
2016-03-29 14:16:54 +01:00
Eelco Dolstra
1e18b75170 Prefer local build for the test driver script 2015-08-26 15:04:47 +02:00
Eelco Dolstra
a5645c5896 Let's not have a package named "lib"
See fb24c7c95b.

The "_" suffix follows the convention of the "if" package.
2015-08-11 11:52:40 +02:00
aszlig
fb24c7c95b
nixos/testing: Don't shadow "lib" in makePerlPath.
Regression introduced by 032f0ffdd0.

The change doesn't look obvious at the first sight why it may cause
problems with lib.makePerlPath, but it introduces a Perl package called
"lib".

And using "with perlPackages;" uses the Perl library "lib" instead of
the lib attribute set from pkgs.

So let's use pkgs.lib.makePerlPath directly in hope that there won't be
a Perl package anytime soon which is called "pkgs".

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2015-08-11 03:14:09 +02:00
Joachim Schiele
0731489953 all tests: added meta.maintainers section 2015-07-12 12:29:51 +02:00
aszlig
f12208c5a4
nixos/testing: Use stripped down tesseract only.
Only include the English language for the VM tests, because we most
likely won't need other languages. At least for now.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2015-05-22 08:01:17 +02:00
aszlig
8be00dc71d
nixos/test-driver: Make tesseract OCR optional.
By default this is now enabled, and it has to be explicitely enabled
using "enableOCR = true". If it is set to false, any usage of
getScreenText or waitForText will fail with an error suggesting to pass
enableOCR.

This should get rid of the rather large dependency on tesseract which
we don't need for most tests.

Note, that I'm using system("type -P") here to check whether tesseract
is in PATH. I know it's a bashism but we already have other bashisms
within the test scripts and we also run it with bash, so IMHO it's not a
problem here.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2015-05-22 07:21:58 +02:00
aszlig
235c2228ca
nixos/test-driver: Add new getScreenText function.
Basically, this creates a screenshot and throws tesseract at it to
recognize the characters from the screenshot. In order to produce a
result that is well enough, we're using lanczos scaling and scale the
image up to 400% of its original size.

This provides the base functionality for a new Machine method which will
be called waitForText. I originally had that idea long ago when writing
the VM tests for VirtualBox and Chromium, but thought it would be
disproportionate to the case.

The downside however is that VM tests now depend on tesseract, but given
the average runtime of our tests it really shouldn't have a too big
impact and it's only a runtime dependency after all.

Another issue is that the OCR process takes quite some time to finish,
but IMHO it's better (as in more deterministic) than to rely on sleep().

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2015-05-21 19:55:21 +02:00
Eelco Dolstra
13dcb35233 VM tests: Don't produce output on failure
Producing output makes it hard to restart builds that failed for
intermittent reasons (like timeouts). So let's not do that.
2015-04-10 11:18:30 +02:00
Eelco Dolstra
51c349d0cc test-driver: Use netpbm instead of imagemagick
Imagemagick pulls in 100s of megabytes of dependencies.
2014-09-25 00:07:30 +02:00
Bjørn Forsman
8e8f95beda nixos/test-reports: use local copy of jquery
The current way test reports get jquery,

  src="//ajax.googleapis.com/ajax/libs/jquery/1.8.3/jquery.min.js"

only works when getting reports over http:// or https://, not file://.
Change it so that it works for all protocols by using a local copy of
jquery.

This fixes the issue where locally created and browsed test reports
cannot be navigated properly; clicking the '+' symbol to expand
sub-sections doesn't work.
2014-09-07 15:47:59 +02:00
Eelco Dolstra
b599deedb4 makeTest: Pass nodes attribute
This allows doing things like:

  nix-build nixos/tests/misc.nix -A nodes.machine.config.system.build.initialRamdisk
2014-07-30 14:30:25 +02:00
Florian Friesdorf
6a3e6164bd support for nixos test names 2014-06-28 16:05:03 +02:00
Eelco Dolstra
019e9d8a3d Fix simpleTest function 2014-04-24 15:46:10 +02:00
Eelco Dolstra
36c05d5e5b Simplify running tests even further
Now you can just say:

  $ nix-build '<nixos/tests/login.nix>'

You can still get the driver script for interactive testing:

  $ nix-build '<nixos/tests/login.nix>' -A driver
  $ ./result/bin/nixos-test-driver
2014-04-14 14:23:38 +02:00
Eelco Dolstra
abe218950c Make it easier to run the tests
You can now run a test in the nixos/tests directory directly using
nix-build, e.g.

  $ nix-build '<nixos/tests/login.nix>' -A test

This gets rid of having to add the test to nixos/tests/default.nix.
(Of course, you still need to add it to nixos/release.nix if you want
Hydra to run the test.)
2014-04-14 14:02:44 +02:00
Eelco Dolstra
497997cc38 Move generation of coverage reports from nixos/lib/testing to releaseTools
Also, turn some stdenv adapters into setup hooks.
2014-03-03 13:57:08 +01:00
Eelco Dolstra
6dafee8d67 Fix runInMachine
It requires a writable /nix/store to store the build result.  Also,
wait until we've reached multi-user.target before doing the build, and
do a sync at the end to ensure all data to $out is properly written.

http://hydra.nixos.org/build/6496716
2013-10-16 11:37:38 +02:00
Eelco Dolstra
5c1f8cbc70 Move all of NixOS to nixos/ in preparation of the repository merge 2013-10-10 13:28:20 +02:00
Renamed from lib/testing.nix (Browse further)