Commit graph

77 commits

Author SHA1 Message Date
Jacek Galowicz
e05ae69904 nixos/test: Use retry() in all looping functions that need timeouts 2019-12-03 16:59:43 +01:00
Michael Raskin
ad38a08ecb test-driver.py: add Machine.copy_from_vm 2019-11-25 11:35:48 +01:00
Michael Raskin
ef5bc381f9 test-driver.py: Fix the shared directory reference 2019-11-25 11:33:47 +01:00
Robin Gloster
ba03a1b731
nixos/tests: fail on failing subtests
This was the behaviour in the perl test driver too and a regression.

Also cleaned up unused imports
2019-11-25 01:08:06 +01:00
Florian Klink
758efb9348
Merge pull request #73190 from flokli/ceph-tests-python
WIP: nixos/ceph: port tests to python
2019-11-21 20:13:41 +01:00
Jacek Galowicz
4dba4db1db nixos/test: let python driver evaluate test code in global scope so the
user can define and use functions
2019-11-20 19:08:18 +01:00
Jacek Galowicz
72dd1c6a74 nixos/test: If qemu monitor returns 0, simply return (like perl script does) 2019-11-20 17:38:20 +01:00
Jacek Galowicz
67e2042861 nixos/test: Fix child environment of python driver 2019-11-20 17:37:12 +01:00
Jörg Thalheim
03e6ca15e2
test-driver: add mypy support
It's a good idea to expand this in future to test code as well,
so we get type checking there as well.
2019-11-11 13:49:48 +00:00
Florian Klink
a8e9a93493 nixos/tests: fix succeed() with multiple commands
we previously immediately returned the first commands output, and didn't
execute any of the other commands.

Now, return the last commands output.
This should be documented in the method docstring.
2019-11-11 13:45:11 +01:00
Leonhard Markert
c7becac738
nixos/tests: correct arguments to get_unit_info 2019-11-08 15:01:42 +01:00
Marijan Petricevic
5854aaf680 nixos/tests: improve get_unit_info expressiveness on failure 2019-11-08 11:20:54 +01:00
worldofpeace
1e7ddf233a
Merge pull request #72943 from marijanp/port-test-driver-python
Port remaining test-driver functions to python
2019-11-07 20:23:27 +00:00
worldofpeace
6ed6d1af6a
Merge pull request #72947 from tfc/boot-test-port-fix
Boot test port fix
2019-11-07 17:32:43 +00:00
Jacek Galowicz
52ee1026b0 nixos/test: Handle undecodable bytes
This threw exceptions in boot menus
2019-11-07 10:00:39 +01:00
worldofpeace
3780b9e69c
Merge pull request #72835 from tfc/nixos-integration-test-ports
Nixos integration test ports
2019-11-07 01:05:36 +00:00
Marijan
9915a8ca45 nixos/tests: implemented python test-driver version of forwardPort 2019-11-06 22:43:37 +01:00
Marijan
01f79dfc9e nixos/tests: implemented python test-driver version of dumpTTYContents 2019-11-06 22:43:37 +01:00
Jacek Galowicz
cf138b4e6b nixos/tests: Fix subprocess launch call for VM monitor
This crashed in the create-script case
2019-11-06 21:48:08 +01:00
Daniel Schaefer
afbfef93a7 nixos/tests: Ignore shutdown/crash if not booted
Condition seems to be inverted. Crash and shutdown only make sense, when
the machine is booted; i.e. we return immediately otherwise.
In the Perl test driver this is:

    return unless $self->{booted};
2019-11-06 16:06:43 +01:00
Jacek Galowicz
45b339b736 nixos/tests: Implement python test-driver version of getWindowNames and waitForWindow 2019-11-05 18:07:05 +01:00
Jacek Galowicz
9c54658226 nixos/tests: Implement python test-driver version of requireActiveUnit 2019-11-05 18:07:00 +01:00
Florian Klink
ac97edf013 nixos/test: use ptpython as repl 2019-11-04 23:50:27 +01:00
Jacek Galowicz
3a28fefe7d nixos/test: Port test driver to python
Thanks @blitz and @jtraue for help with implementing machine methods
2019-11-04 23:50:27 +01:00
Andrew Childs
d2144755a4 nixos-test-driver: allow configuration of net frontend and backend
When IPXE tests were added, an option was added for configuring only
the frontend, and the backend configuration was dropped entirely. This
caused most installer tests to fail.
2019-07-22 13:44:27 +03:00
Nikolay Amiantov
81d35a9d7e nixos-test-driver: support netRomFile
Needed for UEFI PXE netboot testing.
2019-07-15 19:33:26 +03:00
Samuel Dionne-Riel
3aab228d09 Revert "Add ssh backdoor to VM tests infrastructure."
This reverts commit d6e3db44cf.

See #53935 for explanations. In short, it may be causing issues with
tests on the build infrastructure.
2019-01-19 13:24:39 -05:00
Samuel Dionne-Riel
2646a64fbc
Merge pull request #53827 from samueldr/feature/data-in-logs
tests: Logs timing in tests
2019-01-12 12:54:56 -05:00
Samuel Dionne-Riel
b28b37eb00 tests: Wait for shell for twice as long (10m)
See #49441 for an earlier attempt, which was subsequently reverted. I am
assuming that doubling the time will be sufficient if the machine is
overloaded since so many of the tests already pass at 5 minutes, while
still not holding back failures for needlessly long.
2019-01-11 22:40:19 -05:00
Samuel Dionne-Riel
5d93e2c01c test-driver: Logs time taken for nests 2019-01-11 22:36:31 -05:00
Samuel Dionne-Riel
1fe0018df8 test-driver: Adds time it took to connect to guest in logs
This will make it possible to track whether the time is generous or not
when ran on hydra.
2019-01-11 22:36:31 -05:00
Sarah Brofeldt
95486ca306 Revert "NixOS tests: Wait for shell for 10x longer (50m)"
This reverts commit 9bc10e1291.
2018-11-05 08:58:08 +00:00
Sarah Brofeldt
9bc10e1291 NixOS tests: Wait for shell for 10x longer (50m) 2018-10-30 09:22:42 +01:00
Tuomas Tynkkynen
d6e3db44cf Add ssh backdoor to VM tests infrastructure.
Thanks to @dezgeg for prototype implementation, I've
cleaned it up and added documentation.
2018-09-28 10:53:08 +01:00
aszlig
f148c5c4a1
nixos/tests: Fix QEMU flags for SCSI disks
The ability to specify "-drive if=scsi" has been removed in QEMU version
2.12 (introduced in 3e3b39f173).

Quote from https://wiki.qemu.org/ChangeLog/2.12#Incompatible_changes:

> The deprecated way of configuring SCSI devices with "-drive if=scsi"
> on x86 has been removed. Use an appropriate SCSI controller together
> "-device scsi-hd" or "-device scsi-cd" and a corresponding "-blockdev"
> parameter instead.

So whenever the diskInterface is "scsi" we use the new way to specify
the drive and fall back to the deprecated way for the time being. The
reason why I'm not using the new way for "virtio" and "ide" as well is
because there is no simple generic way anymore to specify these.

This also turns the type of the virtualisation.qemu.diskInterface option
to be an enum, so the user knows which values are allowed but we can
also make sure the right value is provided to prevent typos.

I've tested this against a few non-disk-related NixOS VM tests but also
the installer.grub1 test (because it uses "ide" as its drive interface),
the installer.simple test (just to be sure it still works with
"virtio") and all the tests in nixos/tests/boot.nix.

In order to be able to run the grub1 test I had to go back to
8b1cf100cd (which is a known commit where
that test still works) and apply the QEMU update and this very commit,
because right now the test is broken.

Apart from the tests here in nixpkgs, I also ran another[1] test in
another repository which uses the "scsi" disk interface as well (in
comparison to most of the installer tests, this one actually failed
prior to this commit).

All of them now succeed.

[1]: 9b5a119972/tests/system/kernel/bfq.nix

Signed-off-by: aszlig <aszlig@nix.build>
Cc: @edostra, @grahamc, @dezgeg, @abbradar, @ts468
2018-04-30 03:02:59 +02:00
Will Dietz
0086036141 Machine.pm: tweak to look for graphical target in waitForX
Current message may not be printed
(nothing in systemd contains that string,
 it comes from elsewhere)
2018-04-03 13:59:16 -05:00
Maximilian Bosch
e538e00404
test-driver: support testing user units
It is quite complicated to test services using the test-driver when
declaring user services with `systemd.user.services` such as many
X11-based services like `xautolock.service`.

This change adds an optional `$user` parameter to each systemd-related
function in the test-driver and runs `systemctl --user` commands using
`su -l $user -c ...` and sets the `XDG_RUNTIME_DIR` variable
accordingly and a new function named `systemctl` which is able to run a
systemd command with or without a specified user.

The change can be confirmed with a simple VM declaration like this:

```
import ./nixos/tests/make-test.nix ({ pkgs, lib }:

with lib;

{
  name = "systemd-user-test";

  nodes.machine = {
    imports = [ ./nixos/tests/common/user-account.nix ];

    services.xserver.enable = true;
    services.xserver.displayManager.auto.enable = true;
    services.xserver.displayManager.auto.user = "bob";
    services.xserver.xautolock.enable = true;
  };

  testScript = ''
    $machine->start;
    $machine->waitForX;

    $machine->waitForUnit("xautolock.service", "bob");
    $machine->stopJob("xautolock.service", "bob");
    $machine->startJob("xautolock.service", "bob");
    $machine->systemctl("list-jobs --no-pager", "bob");
    $machine->systemctl("show 'xautolock.service' --no-pager", "bob");
  '';
})
```
2018-01-02 20:14:27 +01:00
Tuomas Tynkkynen
8762490337 nixos/qemu: Add virtio-rng-pci to test-driver
Otherwise AArch64 KVM guests don't have enough entropy at boot.
2017-11-26 11:13:21 +02:00
Joerg Thalheim
00344162a7 Revert "nixos/test-instrumentation: softcode DISPLAY"
This reverts commit 4531ec5955.

This broke waitForX: b11ac5d29d (commitcomment-25486347)

cc @Lassulus
2017-11-08 17:04:39 +00:00
lassulus
4531ec5955 nixos/test-instrumentation: softcode DISPLAY 2017-11-07 23:01:59 +01:00
Bas van Dijk
5b8ff5ed49 graphite: 0.9.15 -> 1.0.2
Fixes: #29961

Also added the option:

  services.graphite.web.extraConfig

for configuring graphite_web.
2017-10-08 03:03:22 +02:00
Graham Christensen
3f40fcabbf
nixos tests: waitForWindow: output a list of windows we see prior to the final check
machine: must succeed: xwininfo -root -tree | sed 's/.*0x[0-9a-f]* \"\([^\"]*\)\".*/\1/; t; d'
    machine: exit status 0
    machine: Last chance to match /(?^:dfiirst configuration)/ on the the window list, which currently contains:
    machine: [i3 con] container around 0xf8a5f0, i3: first configuration, [i3 con] floatingcon around 0xf8c260, [i3 con] container around 0xf8a380, i3bar for output Virtual-1, [i3 con] bottom dockarea Virtual-1, [i3 con] workspace 1, [i3 con] content Virtual-1, [i3 con] top dockarea Virtual-1, [i3 con] output Virtual-1, [i3 con] workspace __i3_scratch, [i3 con] content __i3, [i3 con] pseudo-output __i3, i3
2017-06-20 21:16:35 -04:00
Graham Christensen
1b833015b7
nixos tests: waitForText: output the detected screen content prior to the last attempt
machine: Last chance to match /(?^:BALICE)/ on the screen, which currently contains:
    machine: performing optical character recognition
    machine: sending monitor command: screendump /tmp/nix-build-vm-test-run-sddm.drv-0/ocrin.ppm
    machine: Session Layout

    O O

    0 1 : 0 9

    Wednesday, June 21, 2017

    |_ I

    Select your user and enter password
2017-06-20 21:10:34 -04:00
Graham Christensen
348785eec0
nixos tests: waitUntilTTYMatches: Log TTY contents on last try
If the test has not passed yet, on the last attempt it now outputs:

    machine: Last chance to match /logine: / on TTY2, which currently contains:
    machine: running command: fold -w$(stty -F /dev/tty2 size | awk '{print $2}') /dev/vcs2
    machine: exit status 0
    machine:

    <<< Welcome to NixOS 17.09.git.a804ef4 (x86_64) - tty2 >>>

    machine login:

to help debug the problem. Notice the "logine" typo in my check.
2017-06-20 20:57:39 -04:00
Graham Christensen
56435c1404
nixos tests: retry: Count down to 0, and pass remaining attempts to the sub
Allows test functions to output diagnostic information on failure.
2017-06-20 20:54:33 -04:00
Eelco Dolstra
0996ea8a68
NixOS VM tests: Don't create a setgid group in vde_switch
Nix no longer allows this for security reasons.

http://hydra.nixos.org/build/53993125
2017-06-07 11:57:36 +02:00
Volth
e5fbeb2f11 test-driver: sendKeys() support for capital letters 2017-05-19 13:56:24 +00:00
aszlig
a443bdc0a6
nixos/testing: Improve quality of OCR
First of all, we're now using ImageMagick to improve the screenshot so
that Tesseract has an esier time to recognize the text. The resulting
image of this post-processing is a scaled up black-and-white version
with the backgrounds almost entirely removed and the text edges a bit
blurred, so the screen shots now more or less resemble an image from a
scanner rather. This is what Tesseract is trained for by default.

As mentioned in the previous commit we now also use Tesseract 4, which
further improves the quality of text recognition.

I've spent countless hours just to test different postprocessing
variants and testing what works best for our tests and this is the one
that worked best so far. It's certainly not perfect and I'd like to
avoid the scaling step but we're way better off than before.

In addition to this, the OCR process is now done without an intermediate
file, solely using pipes.

I've tested this using the following VM tests which have OCR enabled:

 * nixos/tests/chromium.nix -A stable
 * nixos/tests/emacs-daemon.nix
 * nixos/tests/installer.nix -A luksroot
 * nixos/tests/lightdm.nix
 * nixos/tests/plasma5.nix
 * nixos/tests/sddm.nix

All of the tests still succeed and comparing some of the recognition
results to the earlier results it now also detects a lot more text than
before this commit.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
2017-04-11 03:21:53 +02:00
aszlig
a5e9668c5c
nixos/test: Fix escaping for copyFileFromHost
A long-time issue and one of the reasons I've never used that function
before. So let's remove that todo-comment and escape the contents
properly.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Cc: @edolstra
2017-02-17 19:45:49 +01:00
Graham Christensen
59e77daf5b
nixos tests: make getTTYText smart about tty size 2017-02-12 18:27:06 -05:00