| *** yadnesh|away is now known as yadnesh | 05:28 | |
| *** yadnesh is now known as yadnesh|afk | 08:09 | |
| *** yadnesh|afk is now known as yadnesh | 08:53 | |
| *** dviroel_ is now known as dviroel | 09:40 | |
| *** slaweq_ is now known as slaweq | 14:09 | |
| *** dasm|off is now known as dasm | 14:10 | |
| *** dviroel is now known as dviroel|lunch | 15:50 | |
| clarkb | vishalmanchanda: re https://review.opendev.org/c/zuul/zuul-jobs/+/864119 I've left a -1 because I'm worried that this will break users on ubnutu bionic if they explicitly require chromium. Instead I think we need to do our best to fix the chromium install or only exclude the install on platforms where it is known to be broken. Can you share log links for where it is breaking? | 16:37 |
|---|---|---|
| vishalmanchanda | clarkb: error logs https://zuul.opendev.org/t/openstack/build/afdcbc4bad4f421bbc7fec5614966d9f/log/job-output.txt#538 | 16:39 |
| fungi | clarkb: see the failures linked from the earlier discussion too: https://meetings.opendev.org/irclogs/%23openstack-infra/%23openstack-infra.2022-11-02.log.html | 16:39 |
| clarkb | in the log all we are getting is dpkg returning 100 | 16:41 |
| clarkb | I see no indicating that the error is due to a /home symlink (which we don't do) and I'm not sure https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1951491 applies either since we don't do upgrades of ubuntu but fresh minimal installs using debootstrap | 16:41 |
| clarkb | do we have something with a bit more verbosity on what is failing or do we need to hold a jammy node and install chromium-browser then check dpkg logs? | 16:42 |
| *** yadnesh is now known as yadnesh|away | 16:42 | |
| clarkb | zuul-jobs is expected to be a library of roles that all zuul users can use. For this reason we need to be careful when making changes like that. In this particular case I think we need to better understand the failure so that we can make an appropriate decision on how to fix or workaround the problem | 16:43 |
| *** dviroel|lunch is now known as dviroel | 16:46 | |
| vishalmanchanda | yeah if we upgrade node to jammy, then we hit more issues which I discussed above with fungi | 16:47 |
| clarkb | vishalmanchanda: is the error happening on both focal and jammy then? did the behavior recently change on focal? | 16:49 |
| vishalmanchanda | clarkb: yeah we are seeing this issue on focal from today morning, otherwise it is working fine. | 16:51 |
| clarkb | got it. So there may be two different issues here | 16:51 |
| vishalmanchanda | clarkb: fro jammy, we hit different bug https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1951491 | 16:52 |
| vishalmanchanda | clarkb: yes. | 16:52 |
| vishalmanchanda | clarkb: let me recheck on my patch, if we hit https://bugs.launchpad.net/ubuntu/+source/chromium-browser/+bug/1927513 on jammy as well. | 16:53 |
| vishalmanchanda | recheck on https://review.opendev.org/c/openstack/horizon/+/861140 | 16:54 |
| clarkb | vishalmanchanda: I'm not convinced https://bugs.launchpad.net/ubuntu/+source/chromium-browser/+bug/1927513 is the problem you are hitting on focal | 16:54 |
| clarkb | we don't symlink homedirs | 16:54 |
| clarkb | and if the behavior changed today then I suspect something new | 16:54 |
| vishalmanchanda | clarkb: ok. | 16:55 |
| vishalmanchanda | clarkb: let me know if you find any solution or workaround. | 16:56 |
| vishalmanchanda | I am also investigating. | 16:56 |
| vishalmanchanda | clarkb: thanks | 16:56 |
| clarkb | vishalmanchanda: is there a focal change we can recheck too? I can put a job hold in place | 16:58 |
| clarkb | then we can inspect the dpkg logs and try to understand why it is failing | 16:58 |
| vishalmanchanda | clarkb: here https://review.opendev.org/c/openstack/horizon/+/864062 | 16:59 |
| clarkb | vishalmanchanda: thanks I've put a hold on horizon-nodejs16-run-test for that change and rechecked it | 17:02 |
| clarkb | fwiw I think ianw ran into similar jammy update problems but with firefox | 17:03 |
| clarkb | and the underlying issues are snaps being different. In the firefox case I think selenium didn't know how to talk to the snap? | 17:03 |
| clarkb | so it wasn't an install issue | 17:03 |
| clarkb | ya looks like the jammy recheck does manage to install firefox and chromium so it fails later. I think that matches what ianw say | 17:06 |
| clarkb | s/say/saw/ | 17:06 |
| clarkb | vishalmanchanda: looks like the chromium-browser install succeeded on that horizon job I rechecked. I wonder if they just temporarily broke the snap | 17:21 |
| vishalmanchanda | clarkb: yeah, it is passing now for focal. | 17:23 |
| clarkb | I'll leave the hold in place now in case the behavior comes back but will plan to delete it later today | 17:25 |
| vishalmanchanda | ok. | 17:25 |
| vishalmanchanda | clarkb: Please take a look at jammy node as well, that's a separate issue. | 17:27 |
| vishalmanchanda | and still happen | 17:27 |
| vishalmanchanda | See https://zuul.opendev.org/t/openstack/build/2143f3af585043f78917baa09a6c3f63/log/job-output.txt#2716 | 17:27 |
| vishalmanchanda | https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1951491 | 17:27 |
| vishalmanchanda | Here is test patch https://review.opendev.org/c/openstack/horizon/+/861140 | 17:28 |
| clarkb | vishalmanchanda: I think this is the same issue that ianw ran into on our side. I don't know if that has been addressed yet | 17:35 |
| clarkb | But it seems the issue is that xvfb and our user session are not running in a way that firefox expects. | 17:36 |
| vishalmanchanda | clarkb: ok. | 17:37 |
| vishalmanchanda | ianw: Did you find any solution to fix it? | 17:37 |
| vishalmanchanda | ^^ | 17:37 |
| clarkb | https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1951491/comments/31 might be something to try? | 17:42 |
| clarkb | skimming the whole thread it seems to be something related to dbus and logins not being fully configured for it. The assumption seems to be that your login manager will do all this setup for you. But in our case wer are ssh'd in and don't have gdm etc | 17:43 |
| vishalmanchanda | I tried here https://review.opendev.org/c/openstack/horizon/+/861140/9/package.json | 17:43 |
| clarkb | does unsetting the env var set the value? | 17:44 |
| clarkb | in the linked comment the set the value to the actual bus location aiui | 17:44 |
| vishalmanchanda | clarkb: I don't know how to check if value is set or not | 17:47 |
| clarkb | vishalmanchanda: the `env` command will dump everything set in the current env. In this case `env | grep DBUS` should do it. Why unset the var though if the comment says it needs to be set? | 17:48 |
| clarkb | quickly checking on a jammy node there is no XDG_RUNTIME_DIR either | 17:55 |
| clarkb | there is no bus socket in /run/user/$UID/ | 17:56 |
| vishalmanchanda | clarkb: because it is suggested here in bug discussion https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1951491/comments/66 | 17:56 |
| fungi | this sounds eerily like the suggestions i made last week about the problem | 17:57 |
| fungi | including suggesting where in the horizon repo it could be tested | 17:57 |
| clarkb | ya the var isn't set as far as I can tell and the bus doesn't exist | 17:57 |
| clarkb | I guess maybe in some situations unsetting things indicates to snap/firefox to not look for that dbus but in others you set it so it can find it successfully? | 17:58 |
| clarkb | I'm confused though why sometimes setting and sometimes unsetting it is fixing it for different people | 18:02 |
| clarkb | it seems like snap wants to be able to talk to dbus in order to manage user cgroup stuff so that it can further isolate process like firefox | 18:02 |
| clarkb | and it is doing that because it is running into cgroupv2 and needing to rely on dbus to manage cgroupv2 things (but doesn't under v1 cgroups?) | 18:02 |
| fungi | right, switching to cgroups v1 seemed to be the workaround some users went with | 18:04 |
| clarkb | and ya xvfb is just a framebuffer not an entire desktop session login | 18:04 |
| fungi | the xvfb package could be augmented with the necessary session setup steps though, maybe as a wrapper script | 18:05 |
| clarkb | I agree it seems the fundamental underlying issue here is that snap + firefox expect an entire desktop session with all of the bits that come along with that. We've not needed that in the past but maybe we need to do that somehow. Or stop using snap firefox or something else | 18:05 |
| fungi | it's possible jamespage or coreycb have some additional insights on it as well, since i'm sure we're not the only ones trying to run ubuntu packaged browsers in a headless environment for webapp testing | 18:06 |
| fungi | it seems like ubuntu openstack has some testing of its own, so presumably covers horizon as well | 18:07 |
| clarkb | xvfb-run exists, but I don't think it has the sessions management. But maybe updating it to include that would make sense | 18:08 |
| clarkb | digging around I think we may need dbus-user-session package installed? | 18:19 |
| clarkb | and possibly need to relogin after installing that? | 18:19 |
| clarkb | It looks like horizon manages xvfb directly in the test suite via xcfbwrapper though | 18:19 |
| vishalmanchanda | clarkb: maybe. | 18:25 |
| *** dviroel is now known as dviroel|biba | 19:51 | |
| *** dviroel|biba is now known as dviroel|biab | 19:51 | |
| ianw | (unhelpful but i did not find a solution to using firefox directly. in infra we use the selenium-firefox container, and talk to it using the remote api. this is not 100% ideal as you can't access some firefox specific things from that api, like the full page screenshot that only firefox provides) | 20:01 |
| fungi | that does sound like a possible alternative solution for horizon's tests though | 20:07 |
| opendevreview | Nikolay Vinogradov proposed openstack/project-config master: Add the cinder-infinidat charm to Openstack charms https://review.opendev.org/c/openstack/project-config/+/863954 | 23:00 |
| *** dasm is now known as dasm|offp | 23:03 | |
| *** dasm|offp is now known as dasm|off | 23:03 | |
| opendevreview | Ian Wienand proposed openstack/openstack-zuul-jobs master: rpm-openafs: bump to 1.8.9pre1 https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/864149 | 23:05 |
| opendevreview | Ian Wienand proposed openstack/openstack-zuul-jobs master: rpm-openafs: bump to 1.8.9pre1 https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/864149 | 23:26 |
| opendevreview | Nikolay Vinogradov proposed openstack/project-config master: Add the cinder-infinidat charm to Openstack charms https://review.opendev.org/c/openstack/project-config/+/863954 | 23:26 |
| opendevreview | Nikolay Vinogradov proposed openstack/project-config master: Add the infinidat-tools charm to Openstack charms https://review.opendev.org/c/openstack/project-config/+/863955 | 23:31 |
| opendevreview | Ian Wienand proposed openstack/openstack-zuul-jobs master: rpm-openafs: bump to 1.8.9pre1 https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/864149 | 23:32 |
| opendevreview | Nikolay Vinogradov proposed openstack/project-config master: Add manila-infinidat backend charm to OpenStack charms https://review.opendev.org/c/openstack/project-config/+/863957 | 23:33 |
| opendevreview | Ian Wienand proposed openstack/openstack-zuul-jobs master: rpm-openafs: bump to 1.8.9pre1 https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/864149 | 23:42 |
| opendevreview | Ian Wienand proposed openstack/openstack-zuul-jobs master: rpm-openafs: bump to 1.8.9pre1 https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/864149 | 23:55 |
| opendevreview | Ian Wienand proposed openstack/openstack-zuul-jobs master: rpm-openafs: bump to 1.8.9pre1 https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/864149 | 23:59 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!