Page MenuHomeLubuntu Development

Transfer testcases to the actual testcase branch
Open, NormalPublic

Description

We're the only flavor to use Calamares, therefore, we have some extra testing overhead compared to other flavors. Up to now, this has lived in Testing Checklist, but it really should be transferred to the ISO QA tracker. Instructions on doing that can be found here; once they're written and proposed for merging, I can merge them and complete the correct administration tasks.

Event Timeline

tsimonq2 triaged this task as Normal priority.Apr 13 2019, 12:37 PM
tsimonq2 created this task.
tsimonq2 added a parent task: T41: Lubuntu 19.10.
guiverc claimed this task.Jul 26 2019, 6:47 AM

Hey @guiverc, any progress on this one?

guiverc added a comment.EditedAug 3 2019, 12:48 AM

I ran the installer a few times & made notes (largely a comparison between 19.10's install and the manual), right now I have no idea where those notes are (pad.ubuntu.com as I recall, but i'll have to find the email or whatever breadcrumb I left for myself to locate it). I also downloaded existing testcases and then looked at alternate editors as I felt my usual editor would be error prone (I didn't like how it appeared).

My intention was to continue on it today, but i've been doing 18.04.3 instead...

https://pad.ubuntu.com/l9zgyhdZab (*minimal there except one image different to manual example*)

Our current (testcase 1701) is written for "Erase Disk" however it doesn't always show.

On a (vbox) VM on this (d980) box it shows between Replace.a.partition and Manual.partitioning. I got the same on another vbox session (hp 8200), or test on where I had two recent 18.04.3 installs (install & a install(auto-resize)) and Erase disk appears, as it did on my d780 (multiple ubuntu installs).

However on the machine besides (d755) me the current daily offers only, likewise another d755 (which contains only a single debian-testing installation) :-

  • Install.alongside
  • Replace a partition
  • Manual partitioning

Should I/we write text under the "Erase Disk" to allow for that situation (if so what?; it'll get complicated fast..) or leave it blank as it currently is letting the tester work around the problem in making the test work...

Is it because of a mounted partition? I looked and didn't see one, rebooted d755 & mount |grep sd shows only the install-media mounted...

wxl added a subscriber: wxl.Aug 13 2019, 12:08 AM

Weird. Let's see the ~/.cache/Calamares/session.log on that one.

https://paste.ubuntu.com/p/Cvvx77Gv5r/
has the ~/.cache/Calamares/session.log file.
I hadn't exited the installer at that stage, it was grabbed when it was at the

  • install alongside
  • replace partition
  • manual partitioning

question, ie. no erase disk & install option provided. Single 80gb disk; partitions for /, swap & /home only

kc2bez added a subscriber: kc2bez.Aug 13 2019, 12:54 PM
In T56#1982, @guiverc wrote:

https://paste.ubuntu.com/p/Cvvx77Gv5r/
has the ~/.cache/Calamares/session.log file.
I hadn't exited the installer at that stage, it was grabbed when it was at the

  • install alongside
  • replace partition
  • manual partitioning

question, ie. no erase disk & install option provided. Single 80gb disk; partitions for /, swap & /home only

I am guessing that the Swap is mounted as /dev/sda2?

2019-08-13 - 05:38:37 [6]: Checking if "/dev/sda2" can be resized. 
2019-08-13 - 05:38:37 [6]:  ..  NO, partition is mounted 
2019-08-13 - 05:38:37 [6]: Checking if "/dev/sda2" can be replaced. 
2019-08-13 - 05:38:37 [6]:  ..  NO, it is mounted.

:) I didn't think of swap, and that would apply to all systems I tested on.
and easy to deal with in a text line; I'm thinking we ask testers to leave a comment on that one

wxl added a comment.Aug 13 2019, 11:08 PM

Here's what I know (and this is covered in the manual, but feel free to suggest improvements; it might be good to have GUI instructions, especially as it relates to any mounted partition rather than just swaps):

  • If a device has any partitions mounted, you will not get the Erase Disk option
  • Swaps are automounted by the core Ubuntu system in live, for better or for worse
  • It is beneficial to have swaps mounted on the live system, at least for systems low on memory, so we don't want to make some sort of universal change where we necessarily
  • There's been plenty of discussion upstream about posting a warning or providing an option to unmount things but no real traction on doing anything about it
  • I don't have the hardware to test but I have reason to believe based on some support items and bugs I've seen that eMMC devices are automounted

Therefore, I believe the testcase should provide instructions to:

  • Check to see if the target drive has any mounted partitions
  • Unmount any partitions if found

…before launching the installer.

Thanks Walter & Dan too.

I'm disappointed I didn't think of swap (and yeah I did know the kernel mounts it if detected but it still escaped me).

I'll add messages re: your instructions when I get a chance. At least 3 of the diffs were added to https://pad.ubuntu.com/l9zgyhdZab though I've only made the (4) BIOS cases so far (2x4 uefi still to go). I've compared each to an install so i believe they're accurate (though I've yet to read the diffs to look for language/typos..)

wxl added a comment.Aug 14 2019, 9:08 PM

@guiverc looking good. I would suggest simply committing these as new test cases (we need to keep the old ones for 18.04 for now) as long as you have a complete test case. We can worry about proofreading and more minor edits later.

BTW remember we have notes.lubuntu.me for collaborative documentation. If you need a login, let me or @tsimonq2 know.

They will be new testcases; the files currently are all local (diffs online were a form of backup, though created at that point as I wanted two to speed creation of the fourth one via copy/paste); I haven't yet done the vm install to verify fourth is good. I don't yet understand the upload steps which is why they're all local - but I'll/we'll get there.

Thanks re: notes.lubuntu.me, a copy exists there too but I didn't click 'publish' in case it was publish to world (https://notes.lubuntu.me/3mjA6fgTS56sKUd1dW516g?edit) though it'll be identical to pad.ubuntu.com link I suspect; I aim to continue on this today (arvo/later)

wxl added a comment.Aug 14 2019, 9:46 PM

Sign ups to notes are closed, so if it's public, it doesn't matter ☺

I just booted [19.10] daily on vaiocrap & noticed an OEM option at grub; maybe we should have testcase(s) for that??
(i'm starting EFI set)

In T56#2000, @wxl wrote:

...

Therefore, I believe the testcase should provide instructions to:

  • Check to see if the target drive has any mounted partitions
  • Unmount any partitions if found

…before launching the installer.

I was going to handle it with a

<dd>If "Erase disk" does not show you have a mounted partition, please close installer, umount your partitions (it may be a swap partition that was mounted on boot) and return to the 'click the "Install FAMILY XX.YY" icon step'</dd>

@wxl Is this acceptable? (my first draft had a scream aloud that was just replaced by that message)

My Reasoning: On a lot of boots I've recently done, I did not have the trouble. Thus why have testers check for 'mount' if the lack of "Erase disk" will tell them.. They can then exit, umount & return a few steps... [yeah it's a "GOTO" - but I'm an ex-COBOL dev :) ]

I don't believe anything shows any differently for secure+EFI boot, (vs efi) so it'll use identical testcases as EFI.

The EFI ones differ slightly [to bios] ..

guiverc@d960-ubu2:/de2900/lubuntu/qa-testing/1910-image-tests$ diff full_disk_install_encypt_bios_noweb.txt full_disk_install_encypt_efi_noweb.txt
23c23

< <dd>Please ensure you see BIOS listed besides your drive. If you see EFI it's covered in another testcase</dd>

<dd>Please ensure you see EFI listed besides your drive. If you see BIOS it's covered in another testcase</dd>

I've updated the notes diffs (https://notes.lubuntu.me/3mjA6fgTS56sKUd1dW516g) but not pad.ubuntu.com

Files pass format checks..
guiverc@d960-ubu2:/de2900/lubuntu/qa-testing/1910-image-tests$ test-case-format full_disk_install_*.txt
full_disk_install_encypt_bios_noweb.txt
full_disk_install_encypt_bios_waitweb.txt
full_disk_install_encypt_efi_noweb.txt
full_disk_install_encypt_efi_waitweb.txt
full_disk_install_noencypt_bios_noweb.txt
full_disk_install_noencypt_bios_waitweb.txt
full_disk_install_noencypt_efi_noweb.txt
full_disk_install_noencypt_uefi_waitweb.txt

wxl added a comment.Aug 15 2019, 2:19 PM

I'd suggest this:

<dd>If "Erase disk" does not show, your target device has a mounted partition. Please close installer, umount your partitions (most likely a swap partition that was mounted on boot), and return to the 'click the "Install FAMILY XX.YY" icon step'</dd>

Slight difference, but it should make things more clear. Should we give specific instructions to unmount?

guiverc added a comment.EditedAug 29 2019, 11:08 PM

I can't currently login to notes.lubuntu.me so I'll flood-post here (delete or move if it's a problem)

Note a lot of comment is just for me; I start with a skeleton & then build/fill-it-in...

I've progressed no further; still need to upload them, but if there are problems with the testcases these diffs should be simpler to scan as they diff to another 'close' testcase instead of to the original each time..

Please if there should be a difference between SECURE(boot)-EFI and EFI - please let me know as I have no difference (which could be a plus anyway)

guiverc@d960-ubu2:/de2900/lubuntu/qa-testing/1910-image-tests$ pastebinit diffs_full-disk.txt
https://paste.ubuntu.com/p/NYzQ4vRynB/

guiverc added a comment.EditedAug 30 2019, 2:27 AM
  • Pushed [bazaar] & merge request done thru launchpad

guiverc@d960-ubu2:/de2900/lubuntu/qa-testing/ubuntu-manual-tests/testcases/image$ bzr push lp:~guiverc/ubuntu-manual-tests/lubuntu-calamares
Using default stacking branch /+branch-id/715983 at chroot-139717651353168:///~guiverc/ubuntu-manual-tests/
Created new stacked branch referring to /+branch-id/715983.

  • for info only - the following contains the commands used

guiverc@d960-ubu2:/de2900/lubuntu/qa-testing/ubuntu-manual-tests/testcases/image$ pastebinit /de2900/lubuntu/qa-testing/1910-image-tests/bzr_upload_2019-08-30.txt
https://paste.ubuntu.com/p/Rt3mqdfsVN/

  • Response - waiting Moderation

Your mail to 'ci-jenkins-ro@lists.canonical.com' with the subject

[Merge] lp:~guiverc/ubuntu-manual-tests/lubuntu-calamares into lp:ubuntu-manual-tests

Is being held until the list moderator can review it for approval.
The message is being held because:

Message has implicit destination
The message is larger than the 40 KB maximum size

*I just noted the phab.lubuntu testcase also has other install types... I hadn't scrolled down so missed these.. so are not currently included*

guiverc added a comment.EditedAug 30 2019, 11:00 PM

Diffs for other testcases
(test-case-format checks pass etc)
https://paste.ubuntu.com/p/CHPzh5sk4F/

Do you want TUI & GUI release-upgrade testcases also on iso.qa...?
I don't recall seeing them for any other flavor, though I have found http://iso.qa.ubuntu.com/qatracker/testcases/1310/info which IS used by Ubuntu Budgie and appears with their other tests as a 'run-once'.

guiverc@d960-ubu2:/de2900/lubuntu/qa-testing/ubuntu-manual-tests/testcases/image$ bzr status
added:

testcases/image/auto-login_after_install.txt
testcases/image/custom_partitioning_on_btrfs.txt
testcases/image/custom_partitioning_on_xfs.txt
testcases/image/custom_partitioning_with_seperate_home.txt
testcases/image/install_with_existing_partition.txt

guiverc@d960-ubu2:/de2900/lubuntu/qa-testing/ubuntu-manual-tests/testcases/image$ bzr commit -m "Lubuntu 19.10 Calamares installer variations"
Committing to: /de2900/lubuntu/qa-testing/ubuntu-manual-tests/
added testcases/image/auto-login_after_install.txt
added testcases/image/custom_partitioning_on_btrfs.txt
added testcases/image/custom_partitioning_on_xfs.txt
added testcases/image/custom_partitioning_with_seperate_home.txt
added testcases/image/install_with_existing_partition.txt
Committed revision 391.

@tsimonq2 I'm stuck with GUI upgrade..

Upgrade testcases (started from 1310_Upgrade as used by Ubuntu-Budgie)

I believe TUI is good, but given it'll be used for pre-release installs in most cases; adding -d to do-release-upgrade was easy, but I'm unsure of how to accomplish this with GUI upgrade testing...

On the assumption my TUI is okay, I'm stuck with how to initiate GUI upgrade.
Also note: What I did do was based on how it appears in 18.10; it doesn't match 19.04 where I can't find where to go
(it's missing from our manual; do we support it). Any advice is welcome.

(blank lines to signal where it needs work; plus line above which was correct for 18.10)
File: Upgrade_using_GUI.txt{F71114}

guiverc@d960-ubu2:/de2900/lubuntu/qa-testing/1910-image-tests$ diff Upgrade_using_TUI.txt Upgrade_using_GUI.txt
8a9,12

<dt>Navigate through menu to Settings -> Software.Updater. It should scan for upgrades and if a newer release is found, you'll be asked to Upgrade. If no release is found you can click Settings and ensure in the Updates tab, that "For any new version" is selected in the bottom new Ubuntu version option</dt>
<dt>If however you're testing a pre-release upgrade path it may not be found</dt>

My testcases can be seen using

guiverc@d960-ubu2:/de2900/lubuntu/qa-testing/1910-image-tests$ pastebinit Upgrade_using_TUI.txt
https://paste.ubuntu.com/p/8TmNfgwpBQ/
guiverc@d960-ubu2:/de2900/lubuntu/qa-testing/1910-image-tests$ pastebinit Upgrade_using_GUI.txt
https://paste.ubuntu.com/p/yFHQy4z66M/
guiverc@d960-ubu2:/de2900/lubuntu/qa-testing/1910-image-tests$ test-case-format Upgrade_using_TUI.txt
Upgrade_using_TUI.txt

Should you want to view diff between my 1310-source & TUI

guiverc@d960-ubu2:/de2900/lubuntu/qa-testing/1910-image-tests$ diff 1310_Upgrade Upgrade_using_TUI.txt |pastebinit
https://paste.ubuntu.com/p/fdqcSDkNKm/

test-case

@tsimonq2, (I was supposed to mention @ standup but i had my usual *blank/empty mind* at meeting)..

I'm not active in QA-testing testcases; full_disk were submitted awaiting review (08-30) along with other custom/existing/auto-login etc on 08-31. The TUI (release-upgrade) was done but I don't think submitted via bzr as I was waiting GUI upgrade-advice I think (prior entry here)

I have also committed the TUI (text user interface) upgrade
Please note: There have been no changes since 1-Sept-2019's comment & diff

guiverc@d960-ubu2:/de2900/lubuntu/qa-testing/ubuntu-manual-tests/testcases/image$ bzr add Upgrade_using_TUI.txt
adding testcases/image/Upgrade_using_TUI.txt
guiverc@d960-ubu2:/de2900/lubuntu/qa-testing/ubuntu-manual-tests/testcases/image$ bzr commit -m "Lubuntu 19.10 Upgrade using TUI"
Committing to: /de2900/lubuntu/qa-testing/ubuntu-manual-tests/
added testcases/image/Upgrade_using_TUI.txt
Committed revision 392.

I believe this is all cases (excluding the Upgrade_using_GUI.txt I've not committed hoping for instruction first; ie. how to write up a -d for GUI).
Thanks @wxl

wxl added a comment.Sep 20 2019, 9:58 AM

@guiverc thanks for that!!

Feel free to push those commits and they'll get added to your review.

Also, as per the testing checklist, you can instigate a GUI upgrade with do-release-upgrade -d -m desktop -f DistUpgradeViewKDE.

@tsimonq2 or @wxl - the existing test cases all have a number (eg. I've based mine on 1701 for new installs, and 1310 for upgrades). What numbers shall I give them? or where do I get the numbers? (or can it be any unused number?, isn't there a some document/procedure for claiming numbers?)

I've started making changes; I haven't yet found a ? @wxl but it could also be I've not to a file that contains it. I hoped to have done this Saturday, but whilst I may not get it done today, it should be by tomorrow (my plan anyway)

I'm adding a top line on each that gives the purpose of the file (allowing it to be renamed without loosing anything as the first line contains the detail I previously had in the filename (ie. the release-team/testing-checklist test detail data).

fyi: the primary reason for "Lubuntu QA-test" was to make it easy for me to grep & find them; but I think it's useful. First examples of top lines (from a grep; and it's outdated as case issue i detected has been fixed)

auto-login_after_install.txt:<em>Lubuntu QA-test: auto-login after install</em>
custom_partitioning_on_btrfs.txt:<em>Lubuntu QA-test: Custom Partitioning on BTRFS</em>
custom_partitioning_on_xfs.txt:<em>Lubuntu QA-test: Custom partitioning on XFS</em>
custom_partitioning_with_seperate_home.txt:<em>Lubuntu QA-test: Custom partitioning with seperate home</em>

@guiverc The number is assigned once the commit is merged; it is the same as the internal ISO QA tracker number referencing the testcase.

The testcases are not read by the tracker itself, they're copied over manually. It really makes it up to the person who claims responsibility for merging it.

'people' names committed - READY FOR REVIEW please. @wxl @tsimonq2

files unchanged today; 3x </em?> fixed yesterday
today was final name changes ie. "_" replaced by spaces, noweb->offline, encypt->encryption, secefi_>SecureEFI, etc with more consistency I hope (though longer)

// the following is cut/pasted & maybe missing some files
commit 395
added testcases/image/Auto login
added testcases/image/Custom partitioning on BTRFS
added testcases/image/Custom partitioning on XFS
added testcases/image/Custom partitioning with seperate home
added testcases/image/Install with existing partition
added testcases/image/Upgrade using GUI
added testcases/image/Upgrade using TUI

commit 396:
added testcases/image/Full disk install encryption BIOS internet
added testcases/image/Full disk install encryption BIOS offline
added testcases/image/Full disk install encryption EFI internet
added testcases/image/Full disk install encryption EFI offline
added testcases/image/Full disk install encryption SecureEFI internet
added testcases/image/Full disk install encryption SecureEFI offline
added testcases/image/Full disk install no-encryption BIOS internet
added testcases/image/Full disk install no-encryption BIOS offline
added testcases/image/Full disk install no-encryption EFI internet
added testcases/image/Full disk install no-encryption EFI offline
added testcases/image/Full disk install no-encryption SecureEFI internet
added testcases/image/Full disk install no-encryption SecureEFI offline

This really needs to be done by the end of this week so we can get the release properly tested.

@wxl @guiverc What's the next step?

To me they were ready Sept 24, however https://code.launchpad.net/~guiverc/ubuntu-manual-tests/lubuntu-calamares shows

"Ready for review for merging into lp:ubuntu-manual-tests
Walter Lapchynski: Pending requested 2019-09-20
Simon Quigley: Pending requested 2019-08-30
Diff: 467 lines (+408/-0)
12 files modified
testcases/image/full_disk_install_encypt_bios_noweb.txt (+38/-0)
testcases/image/full_disk_install_encypt_bios_waitweb.txt (+31/-0)
testcases/image/full_disk_install_encypt_efi_noweb.txt (+38/-0)
testcases/image/full_disk_install_encypt_efi_waitweb.txt (+31/-0)
testcases/image/full_disk_install_encypt_secefi_noweb.txt (+38/-0)
testcases/image/full_disk_install_encypt_secefi_waitweb.txt (+31/-0)
testcases/image/full_disk_install_noencypt_bios_noweb.txt (+36/-0)
testcases/image/full_disk_install_noencypt_bios_waitweb.txt (+31/-0)
testcases/image/full_disk_install_noencypt_efi_noweb.txt (+36/-0)
testcases/image/full_disk_install_noencypt_efi_waitweb.txt (+31/-0)
testcases/image/full_disk_install_noencypt_secefi_noweb.txt (+36/-0)
testcases/image/full_disk_install_noencypt_secefi_waitweb.txt (+31/-0)
Propose for merging"

which are the initial names, and not the final versions and filename used in commit 395 & 396 that are wanted (sept 24) which has me confused, and unwilling to do anything further. I expected the page to have changed as files were updated.

guiverc added a comment.EditedSat, Oct 5, 8:08 AM

Neyder noted (#lubuntu-devel) that the LIVE test case mentions Ubiquity & has references to picking timezone.... prior to "Try FAMILY"

I haven't amended "1303_Live\ Session" as it's used by other flavors, but a new case has been added for us to use if easily done. It really just drops some lines (one line added about 'Check disc for defects' as I think it's useful), ie.

guiverc@d960-ubu2:/de2900/lubuntu/qa-testing/ubuntu-manual-tests/testcases/image$ diff 1303_Live\ Session Modern\ Live
7,9c7
< <dt>When ubiquity starts select your language in the left column</dt>
< <dd>Language is selected, all labels are changed to translated versions</dd>
< <dt>Press "Try FAMILY" and wait for the Live session to start</dt>


> <dt>Select "Try FAMILY" and wait for the Live session to start</dt>
20a19
> If you get many strange errors, using the 'Check disc for defects' option is worth using to ensure a valid ISO was used

I also bzr commit & bzr push again (I'm surprised it sees modifications for some; but I guess I missed a step last time, sorry)

guiverc@d960-ubu2:/de2900/lubuntu/qa-testing/ubuntu-manual-tests/testcases/image$ bzr commit -m "Lubuntu 19.10 Calamares installer variations"
Committing to: /de2900/lubuntu/qa-testing/ubuntu-manual-tests/
modified testcases/image/Full disk install no-encryption BIOS internet
modified testcases/image/Full disk install no-encryption EFI internet
modified testcases/image/Full disk install no-encryption SecureEFI internet
added testcases/image/Modern Live
modified testcases/image/Upgrade using GUI
modified testcases/image/Upgrade using TUI
Committed revision 397.
guiverc@d960-ubu2:/de2900/lubuntu/qa-testing/ubuntu-manual-tests/testcases/image$ bzr push lp:~guiverc/ubuntu-manual-tests/lubuntu-calamares
Pushed up to revision 397.

(Much of this detail is for me; I remember little so need it..)

guiverc added a comment.EditedFri, Oct 11, 11:56 PM

@wxl mentioned testcases; so I queried "Replace a partition" using Calamares text; I've opted to add a test rather than modify the "Replace a partition" as it really tested for something different (in my mind anyway).

"Replace a partition" is the more important; being a major option provided in the Calamares installer options. "Install with existing" is more geared for a re-install with /home data surviving (as I saw it anyway), and could still be listed as 'optional' (or dropped).

"Replace a partition" has been added, committed & pushed. (it's just a modified 'Install with existing partition' so I compared diff with that one)

guiverc@d960-ubu2:/de2900/lubuntu/qa-testing/ubuntu-manual-tests/testcases/image$ bzr add Replace\ a\ partition
adding "testcases/image/Replace a partition
guiverc@d960-ubu2:/de2900/lubuntu/qa-testing/ubuntu-manual-tests/testcases/image$ bzr commit -m "Lubuntu 19.10 Calamares installer variations"
Committing to: /de2900/lubuntu/qa-testing/ubuntu-manual-tests/
added testcases/image/Replace a partition
Committed revision 398.
guiverc@d960-ubu2:/de2900/lubuntu/qa-testing/ubuntu-manual-tests/testcases/image$ bzr push lp:~guiverc/ubuntu-manual-tests/lubuntu-calamares
Pushed up to revision 398.