| ianw | yeah i agree those three changes on the stable-3.5 branch don't look important | 00:00 |
|---|---|---|
| clarkb | ianw: at first glance the update to bootstrapping bridge looks sane. But I'll need to take a longer look tomorrow to make sure I understand it properly. | 00:02 |
| ianw | clarkb: no worries. i'm confident enough i think to rebase the bridge-abstraction bits ontop of it I think | 00:02 |
| ianw | i wasn't before, with probably good reason as you found the mistake in it :) | 00:03 |
| clarkb | it was exactly one year ago I sent the email to service-discuss about rsa keys and gerrit :) | 00:03 |
| clarkb | now I get to repsond to myself saying we fixed it | 00:03 |
| ianw | the wheels of big open source can turn slowly sometimes | 00:05 |
| JayF | (but grinds all) | 00:06 |
| clarkb | https://lists.opendev.org/pipermail/service-discuss/2022-October/000367.html | 00:09 |
| clarkb | and now dinner | 00:11 |
| opendevreview | Ian Wienand proposed opendev/system-config master: infra-prod-bootstrap-bridge: run directly on bridge https://review.opendev.org/c/opendev/system-config/+/861138 | 02:33 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Move clouds definitions into control-planes-clouds group https://review.opendev.org/c/opendev/system-config/+/861130 | 02:33 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Run jobs with a jammy bridge.openstack.org https://review.opendev.org/c/opendev/system-config/+/857799 | 02:33 |
| opendevreview | Ian Wienand proposed opendev/system-config master: testinfra: Update selenium calls https://review.opendev.org/c/opendev/system-config/+/858003 | 02:33 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Abstract name of bastion host for testing path https://review.opendev.org/c/opendev/system-config/+/858476 | 02:33 |
| opendevreview | Ian Wienand proposed opendev/system-config master: bootstrap-bridge: use abstracted hostname https://review.opendev.org/c/opendev/system-config/+/861031 | 02:33 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Convert production playbooks to bastion host group https://review.opendev.org/c/opendev/system-config/+/858486 | 02:33 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Run a base test against "old" bridge https://review.opendev.org/c/opendev/system-config/+/860802 | 02:33 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Switch testing bridge name to bridge01.opendev.org https://review.opendev.org/c/opendev/system-config/+/861112 | 02:33 |
| *** dhill is now known as Guest3008 | 03:06 | |
| *** bhagyashris is now known as bhagyashris|ruck | 05:02 | |
| opendevreview | Merged opendev/system-config master: Resync gerrit plugin versions to latest gerrit releases https://review.opendev.org/c/opendev/system-config/+/861270 | 05:11 |
| *** ysandeep|out is now known as ysandeep | 05:52 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: launch-node : make into a small package https://review.opendev.org/c/opendev/system-config/+/861284 | 06:10 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Switch testing bridge name to bridge01.opendev.org https://review.opendev.org/c/opendev/system-config/+/861112 | 06:22 |
| *** frenzy_friday is now known as frenzyfriday|rover | 06:29 | |
| *** arxcruz|rover is now known as arxcruz | 07:18 | |
| *** jpena|off is now known as jpena | 07:39 | |
| *** ysandeep is now known as ysandeep|lunch | 08:26 | |
| *** elodille1 is now known as elodilles | 08:44 | |
| *** ysandeep|lunch is now known as ysandeep | 09:08 | |
| *** lpiwowar8 is now known as lpiwowar | 09:27 | |
| *** rlandy|out is now known as rlandy | 10:30 | |
| *** ysandeep is now known as ysandeep|afk | 11:51 | |
| *** ysandeep|afk is now known as ysandeep | 12:22 | |
| *** pojadhav is now known as pojadhav|afk | 12:36 | |
| gnuoy | Can you request a flavor for the node that runs tests in gate or is it one size fits all? | 13:29 |
| Clark[m] | gnuoy: you can't request specific flavors no. Nodepool and zuul expose "labels" the vast majority of which are sized to have a fairly consistent expectation across the many cloud regions we source resources from. | 13:45 |
| Clark[m] | gnuoy: there are also some more specialty labels intended for use when necessary and knowing that they are provided by fewer clouds and thus may not have the same availability or long term reliability | 13:46 |
| Clark[m] | (there are some larger labels, nested virt labels, and arm64 labels) | 13:46 |
| gnuoy | Thanks Clark[m], are the labels documented somewhere? | 13:47 |
| Clark[m] | https://docs.opendev.org/opendev/infra-manual/latest/testing.html explains the default label sizes. | 13:48 |
| Clark[m] | https://zuul.opendev.org/t/openstack/labels lists the ones currently known to zuul | 13:48 |
| Clark[m] | And the nl0X files in https://opendev.org/openstack/project-config/src/branch/master/nodepool are the raw configs for them in different cloud regions | 13:49 |
| Clark[m] | gnuoy is there something specific you are trying to accomplish? | 13:49 |
| *** dasm|off is now known as dasm | 14:25 | |
| gnuoy | Clark[m], I'm trying out a new job and it seems to be randomly failing. I need to dig into it more but I was originally wondering if it was insufficient resources on the node it was running on. That is wild speculation and I have no evidence that that is the issue atm | 14:26 |
| *** dviroel is now known as dviroel|lunch | 14:57 | |
| fungi | gnuoy: the usual approach is to collect syslog so you can look for oom messages, possibly run something like systat or dstat in the job to collect real-time performance numbers, and so on | 15:07 |
| fungi | also look for filesystem full errors | 15:07 |
| *** ysandeep is now known as ysandeep|out | 15:08 | |
| gnuoy | Thanks fungi, I'm guessing its something like that or an issue pulling docker images. | 15:12 |
| gnuoy | On another topic does anyone know what mistake I've made here https://review.opendev.org/c/openstack/charm-zuul-jobs/+/861150 ? it doesn't seem to be landing. | 15:13 |
| clarkb | gnuoy: looks like you have only added check jobs not gate jobs. No gate jobs means no verified +2 which is required to merge. | 15:15 |
| clarkb | At the very least you should use the noop job to get zuul to report verified +2 back. But better yet is running the check job in gate | 15:15 |
| fungi | gnuoy: oh, if you run into errors pulling docker images from dockerhub, it could be that you've missed using our dockerhub caching proxy and are running afoul of their api quota limits | 15:15 |
| gnuoy | clarkb, ah I see, thank you | 15:17 |
| gnuoy | fungi, dockerhub caching proxy \o/ thats music to my ears! | 15:18 |
| gnuoy | fungi, do you have any details to hand ? If not I can go and dig around | 15:19 |
| *** marios is now known as marios|out | 15:25 | |
| fungi | gnuoy: include the use-docker-mirror role in a pre-run phase playbook like this: https://opendev.org/opendev/system-config/src/branch/master/playbooks/zuul/run-base-pre.yaml#L6 | 15:28 |
| gnuoy | That's fantastic. Thank you both for your help, | 15:28 |
| clarkb | fungi: https://review.opendev.org/c/opendev/system-config/+/861226 will add gitea-lb02 to our inventory. I've also got a zone change up to add it to dns and then a stack behind that that lowers TTLs and swaps out opendev.org and increases TTLs again to transition things once we're happy with the results of the deployment | 15:30 |
| fungi | gnuoy: it will set the appropriate fact for our proxy in that provider, for example: https://zuul.opendev.org/t/openstack/build/5b3caf8f6e9a4e91af120a963e50dfd7/console#1/0/26/bridge01.opendev.org | 15:30 |
| fungi | gnuoy: you can find the documentation for that role along with the other container-oriented utility roles in the zuul-jobs docs: https://zuul-ci.org/docs/zuul-jobs/container-roles.html | 15:34 |
| *** dviroel|lunch is now known as dviroel | 16:07 | |
| *** jpena is now known as jpena|off | 16:34 | |
| opendevreview | Merged opendev/system-config master: Add Jammy gitea-lb02 to our inventory https://review.opendev.org/c/opendev/system-config/+/861226 | 16:44 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Add an Ubuntu FIPS testing token https://review.opendev.org/c/openstack/project-config/+/861457 | 17:02 |
| clarkb | looks like it will be a bit before that new load balancer gets configured (the problem with updating the inventory file is many jobs trigger) | 17:12 |
| clarkb | I've been looking at this IBM IP spamming gerrit with its inability to negotiate ssh trying to figure out who to talk to about it. Unfortunately the IP doesn't seem to show up in the sshd log. I think it is failing well before it gets to that point | 17:40 |
| clarkb | (was hoping it would give me a username there) | 17:40 |
| clarkb | whois has it listed as IBM research in research triangle park. Anyone know anyone at IBM there? | 17:41 |
| clarkb | there is a separate seagate IP failing in a different way and is less persistent. I'm better both are for third party ci systems that have long been forgotten | 17:45 |
| fungi | the ibm system could be for powervm or system-z testing | 17:54 |
| fungi | the only person i knew working at that location switched over to working for oracle some years ago | 17:55 |
| clarkb | I'm beginning to realize that our best option may just be to block the IPs to quiet the logs | 18:03 |
| clarkb | its not like the ssh connections are succeeding anyway | 18:03 |
| fungi | wfm | 18:06 |
| clarkb | #status log Blocked two IPs using iptables that are having ssh connectivity issues to Gerrit in order quiet our logs | 18:16 |
| opendevstatus | clarkb: finished logging | 18:16 |
| clarkb | gitea-lb02 should start getting deployed in just under 15 minutes | 18:27 |
| clarkb | once its done I can override my local /etc/hosts for opendev.org and see how it does | 18:27 |
| clarkb | didn't we attempt to remove snapd from our images at one point? | 18:41 |
| clarkb | any idea if I'm misremembering or why we stopped removing it if that is the case? | 18:41 |
| fungi | clarkb: https://review.opendev.org/709293 Remove snap cleanup tasks | 18:46 |
| fungi | looks like that was specific to the install-kubectl role | 18:47 |
| clarkb | huh so we only did that for kubectl? We definitely don't remove it generally | 18:47 |
| clarkb | something to look into I guess | 18:47 |
| clarkb | also gitea-lb02 is up and running and seems to work with my overridden /etc/hosts | 18:47 |
| clarkb | fungi: https://review.opendev.org/c/opendev/zone-opendev.org/+/861227 is probably fine to land now. And maybe early next week once initial ptg stuff settles down we can look into swapping out and removing the old server | 18:48 |
| fungi | here's where we started removing it in base-server: https://review.openstack.org/605676 | 18:48 |
| clarkb | I'll leave my override in place just ot have something exercising it | 18:48 |
| clarkb | fungi: its no longer in that list and git log -p didn't show where we added it back due to the file path changing | 18:49 |
| clarkb | but that probably deserves more investigating | 18:49 |
| fungi | yeah, finding it | 18:49 |
| fungi | https://review.openstack.org/635593 removed it from the removals list | 18:51 |
| fungi | because we started using it to install kubectl on bridge.o.o | 18:51 |
| clarkb | oh! | 18:54 |
| clarkb | That helps a lot. I think the question then becomes do we still need to install kubectl? If yes can we install it from another location? if so do we want to remove snapd | 18:54 |
| fungi | would have been better if that change had only stopped removing it from bridge.o.o | 18:54 |
| fungi | https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/#install-using-native-package-management | 18:56 |
| fungi | that indicates the kubernetes community maintains debuntu packages | 18:56 |
| fungi | i checked and ubuntu itself still doesn't package kubectl | 18:57 |
| clarkb | ya a lot of things that have ended up in snaps don't get native packages | 18:57 |
| clarkb | but they tend to be more desktop/end user focused it seems like | 18:57 |
| fungi | which is odd because debian provides a package: https://packages.debian.org/search?keywords=kubernetes-client | 18:58 |
| fungi | i guess ubuntu actively decided not to import debian's package and instead push users to snaps | 18:58 |
| clarkb | ianw: ok I looked more closely at that change. Have a couple of thoughts (one is simple file matcher addition) the other is being cautious to not expose the private key via ansible output since we're using a different ansible runner in that playbook. I think we can double check the second thing via the test job? Or maybe add some no_logs or something | 21:06 |
| clarkb | ianw: but I think it will work | 21:06 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Fixup jinja-init image https://review.opendev.org/c/opendev/system-config/+/861473 | 21:22 |
| *** dviroel is now known as dviroel|out | 21:27 | |
| opendevreview | Clark Boylan proposed opendev/gerritbot master: Switch the docker image over to python 3.10 https://review.opendev.org/c/opendev/gerritbot/+/861474 | 21:27 |
| opendevreview | Clark Boylan proposed opendev/grafyaml master: Update Docker images to python 3.10 https://review.opendev.org/c/opendev/grafyaml/+/861475 | 21:32 |
| opendevreview | Clark Boylan proposed opendev/statusbot master: Update container images to python3.10 https://review.opendev.org/c/opendev/statusbot/+/861476 | 21:39 |
| fungi | i still need to find time to look at the bridge changes. ugh | 21:44 |
| clarkb | my plan re ^ is to push a change up removing our 3.8 image builds. But I'm working through updating everything using 3.8 today to 3.10 first and will have a depends on for each of those changes in the eventual removal change | 21:46 |
| clarkb | that will make space for python 3.11 which should release in about 10 days | 21:47 |
| fungi | yep | 21:48 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Drop python 3.8 base image builds https://review.opendev.org/c/opendev/system-config/+/861480 | 21:50 |
| clarkb | I'm realizing the real hurdle for 3.11 will be running things like unittests. I wonder of ubuntu will make a 3.11 package for jammy like they've done with newer pyhton on some of the other LTSs | 21:52 |
| clarkb | worst case we could use debian or something I guess | 21:53 |
| clarkb | (which to be clear isn't a bad situation either) | 21:53 |
| opendevreview | Clark Boylan proposed opendev/gerritbot master: Switch the docker image over to python 3.10 https://review.opendev.org/c/opendev/gerritbot/+/861474 | 22:32 |
| opendevreview | Clark Boylan proposed opendev/grafyaml master: Update Docker images to python 3.10 https://review.opendev.org/c/opendev/grafyaml/+/861475 | 22:34 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Move grafyaml check and gate jobs in repo https://review.opendev.org/c/openstack/project-config/+/861482 | 22:46 |
| opendevreview | Clark Boylan proposed opendev/grafyaml master: Update Docker images to python 3.10 https://review.opendev.org/c/opendev/grafyaml/+/861475 | 22:48 |
| opendevreview | Clark Boylan proposed opendev/grafyaml master: Run pep8 and unittest jobs out of in repo config https://review.opendev.org/c/opendev/grafyaml/+/861483 | 22:48 |
| opendevreview | Clark Boylan proposed opendev/grafyaml master: Update Docker images to python 3.10 https://review.opendev.org/c/opendev/grafyaml/+/861475 | 23:14 |
| opendevreview | Clark Boylan proposed opendev/grafyaml master: Make 'templating' optional https://review.opendev.org/c/opendev/grafyaml/+/861484 | 23:14 |
| clarkb | thats some tech debt. I think that broke in 2018 and voluptuous was pinned in response | 23:16 |
| clarkb | hrm that doesn't seem to fix it. I'll have to investigate closer I guess | 23:21 |
| ianw | clarkb: yeah, that was a huge prompt for the json-based graphs work i did | 23:23 |
| opendevreview | Ian Wienand proposed opendev/system-config master: infra-prod-bootstrap-bridge: run directly on bridge https://review.opendev.org/c/opendev/system-config/+/861138 | 23:35 |
| opendevreview | Clark Boylan proposed opendev/grafyaml master: Fix default handling with newer voluptious https://review.opendev.org/c/opendev/grafyaml/+/861484 | 23:36 |
| opendevreview | Clark Boylan proposed opendev/grafyaml master: Update Docker images to python 3.10 https://review.opendev.org/c/opendev/grafyaml/+/861475 | 23:36 |
| clarkb | ianw: I think ^ that will fix it and allow for python3.10, but zuul should tell us | 23:37 |
| ianw | ++ | 23:38 |
| opendevreview | Ian Wienand proposed opendev/system-config master: infra-prod-bootstrap-bridge: run directly on bridge https://review.opendev.org/c/opendev/system-config/+/861138 | 23:40 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Move clouds definitions into control-planes-clouds group https://review.opendev.org/c/opendev/system-config/+/861130 | 23:40 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Run jobs with a jammy bridge.openstack.org https://review.opendev.org/c/opendev/system-config/+/857799 | 23:40 |
| opendevreview | Ian Wienand proposed opendev/system-config master: testinfra: Update selenium calls https://review.opendev.org/c/opendev/system-config/+/858003 | 23:40 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Abstract name of bastion host for testing path https://review.opendev.org/c/opendev/system-config/+/858476 | 23:40 |
| opendevreview | Ian Wienand proposed opendev/system-config master: bootstrap-bridge: use abstracted hostname https://review.opendev.org/c/opendev/system-config/+/861031 | 23:40 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Convert production playbooks to bastion host group https://review.opendev.org/c/opendev/system-config/+/858486 | 23:40 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Run a base test against "old" bridge https://review.opendev.org/c/opendev/system-config/+/860802 | 23:40 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Switch testing bridge name to bridge01.opendev.org https://review.opendev.org/c/opendev/system-config/+/861112 | 23:40 |
| ianw | ^ needed a restack for the lb changes | 23:41 |
| *** dasm is now known as dasm|off | 23:58 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!