| *** vhari_ is now known as vhari | 00:47 | |
| carthaca | clarkb : Do you know which highlight.js theme is used? I think some work better than others. E.g. atom-one-dark (https://kutt.it/cNPATE ) does a better job than tommorow-night-bright (https://kutt.it/p9WqdA) regarding that example | 10:34 |
|---|---|---|
| ykarel | Hi was there any changes done on infra during the weekend? asking as seeing many failures in periodic jobs when jobs ran on node with 1 cpu(provider rax) | 13:54 |
| ykarel | example failure https://526c8f807052e79b58db-bd4a7cb2b56ee03c292b720cd5fef857.ssl.cf1.rackcdn.com/openstack/0ce11a034a6d499a8a53f6645f91c63e/testr_results.html | 13:54 |
| fungi | ykarel: we have automated zuul upgrades that happen on saturdays, just pulling in the current master branch state, but no actual maintenance | 14:33 |
| fungi | and definitely nothing that i would expect to impact a handful of tempest tests | 14:34 |
| fungi | ykarel: could it be related to gmaan's post on openstack-discuss about https://review.opendev.org/c/openstack/tempest/+/975356 ? | 14:38 |
| fungi | https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.org/thread/JFTDTTTKDTCVP2I33CJUL4D3NTAQ5UQR/ | 14:38 |
| ykarel | fungi, no not related to ^, the issue seen when the target nodes have cpus=1(and rax provider), normally nodes i see have 8 cpu | 14:40 |
| fungi | rax only, not also raxflex? so could be something related to xen i guess? | 14:40 |
| ykarel | seen this issue before as well but was not seeing that frequent but with weekend seen many failures so suspecting some change on infra side | 14:40 |
| ykarel | yes i think seen in rax only | 14:41 |
| fungi | ykarel: oh! yes if it's the random single-cpu node problem, that was something to do with outdated xen in legacy rackspace. i thought we had introduced a node acceptance check to discard any like that before they got used for jobs though... | 14:45 |
| fungi | https://zuul.opendev.org/t/openstack/build/0ce11a034a6d499a8a53f6645f91c63e seems to be the build result page corresponding to your example | 14:47 |
| fungi | https://zuul.opendev.org/t/openstack/build/0ce11a034a6d499a8a53f6645f91c63e/log/zuul-info/host-info.controller.yaml#586-594 does seem to confirm that ansible only saw one cpu on the node | 14:49 |
| fungi | i wonder if our preflight test for that got lost in the switch from nodepool to zuul-launcher, but that's been many months now | 14:50 |
| fungi | i can try to hunt it down after my next meeting | 14:51 |
| ykarel | fungi, yes it's random single-cpu node problem, thx and take your time | 14:56 |
| eolivare | frickler, hi. Last Friday slaweq asked you about an issue with the tobiko-upload-git-mirror job and you advised to refresh our github keys. He's on PTO and I'm trying to fix the issue. I have added a new ssh key to our github repo and I've created an encrypted secret for our zuul job, but I'm facing this problem: https://review.opendev.org/c/x/tobiko/+/975402 | 14:57 |
| eolivare | the x/tobiko project has to branches: master and osp-16.2. Do I have to remove the tobiko-upload-git-mirror job from osp-16.2? | 14:58 |
| frickler | eolivare: well you can either drop the secret from the branch or you can use a new name for the secret and update the job definition accordingly. see https://review.opendev.org/c/openstack/kolla/+/972855 for an example how we did it in kolla | 15:30 |
| clarkb | ykarel: fungi I don't think we ever updated the job side of things for that. I think one of the concerns there was we weren't sure how safe it is (it is probably safe as I don't think we've booted anything with fewer than 4vcpus) | 15:31 |
| clarkb | ykarel: fungi: the other potential solution is adding the boot flag to the kernels on the images we boot. The problem with that is it is likely to be a performance impact so maybe the simpe check of do I have at least 2 vcpus if not then fail in a pre-run job would be best | 15:32 |
| fungi | oh right, i forgot i'd stumbled across that workaround back when we were first looking into the problem | 15:34 |
| clarkb | carthaca: I don't find any indication of Gerrit using any specific thing for highlight.js. It almost looks like they may be defining their own theme as part of the overall gerrit theming process? | 15:36 |
| clarkb | *Any specific theme | 15:37 |
| clarkb | ykarel: fungi https://opendev.org/opendev/system-config/src/branch/master/launch/src/opendev_launch/launch_node.py#L134-L144 this is how we check when launching our control plane nodes. So we'd want to port that into https://opendev.org/opendev/base-jobs/src/branch/master/playbooks/base-test/pre.yaml (maybe as a new role) and check it really quickly and if that works then | 15:43 |
| clarkb | apply it to base and base-minimal as well | 15:43 |
| eolivare | frickler, thanks! | 15:56 |
| eolivare | frickler, is there any way I can test my proposed changing before it's merged? https://review.opendev.org/c/x/tobiko/+/975402 | 16:03 |
| fungi | eolivare: secrets aren't allowed in pre-review pipeline jobs for safety reasons (because anyone could propose a change that exposes the decrypted plaintext value), but the gate pipeline is post-review so you could in theory have a gate pipeline job that exercises those credentials somehow | 17:12 |
| fungi | though i believe the only job you have using it now is probably post-merge (github mirroring)? | 17:12 |
| fungi | in theory though, it will get exercised already immediately after it merges, so you can iterate with a new change if it doesn't work for some reason | 17:15 |
| eolivare | fungi, thanks, I see. It makes sense | 17:15 |
| eolivare | yes, that's what I was thinking. We are going to merge the patch and see if it works. | 17:16 |
| fungi | that's how we handled most of ours too, frankly | 17:16 |
| eolivare | worst case, it won't work, as it doesn't work now :) | 17:16 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!