| openstackgerrit | Merged openstack/project-config master: Revert "Enable IAD opendev.org mirror" https://review.opendev.org/673710 | 00:03 |
|---|---|---|
| *** yamamoto has quit IRC | 00:04 | |
| *** sthussey has quit IRC | 00:06 | |
| *** aaronsheffield has quit IRC | 00:11 | |
| *** slaweq has joined #openstack-infra | 00:11 | |
| *** slaweq has quit IRC | 00:16 | |
| *** iurygregory has quit IRC | 00:16 | |
| openstackgerrit | Merged opendev/system-config master: Add tool to analyze check and gate success rates https://review.opendev.org/671777 | 00:28 |
| *** bobh has quit IRC | 00:44 | |
| *** diablo_rojo has joined #openstack-infra | 00:53 | |
| *** goldyfruit has joined #openstack-infra | 01:00 | |
| *** ricolin has joined #openstack-infra | 01:01 | |
| *** happyhemant has quit IRC | 01:18 | |
| *** SergeyLukjanov has quit IRC | 01:20 | |
| *** SergeyLukjanov has joined #openstack-infra | 01:22 | |
| *** ricolin has quit IRC | 01:31 | |
| *** jamesmcarthur has joined #openstack-infra | 01:35 | |
| *** yamamoto has joined #openstack-infra | 01:54 | |
| *** diablo_rojo has quit IRC | 01:59 | |
| *** yamamoto has quit IRC | 01:59 | |
| *** slaweq has joined #openstack-infra | 02:11 | |
| *** slaweq has quit IRC | 02:16 | |
| openstackgerrit | Merged zuul/nodepool master: Enable debug logs for openstack-functional tests https://review.opendev.org/672412 | 02:28 |
| openstackgerrit | Merged zuul/zuul master: Add change replacement field in doc for start-message https://review.opendev.org/665974 | 02:34 |
| *** ykarel has joined #openstack-infra | 02:37 | |
| *** jamesmcarthur has quit IRC | 02:38 | |
| openstackgerrit | Merged zuul/zuul master: Improve error reporting for zuul dequeue https://review.opendev.org/669813 | 02:43 |
| *** jamesmcarthur has joined #openstack-infra | 02:45 | |
| *** armax has quit IRC | 02:48 | |
| clarkb | ianw I've accepted the ssh host key for the ip address of the new fn mirror | 02:48 |
| clarkb | we use the ip addr and not fqdn with ansible | 02:49 |
| clarkb | so it should hopefully update on the next run now | 02:49 |
| *** tdasilva_ has quit IRC | 02:52 | |
| *** tdasilva_ has joined #openstack-infra | 02:52 | |
| fungi | though if you did ssh into it from there by fqdn, it would also accept the ip address | 02:53 |
| fungi | (whichever address it connected to if there was more than one in dns for that name) | 02:53 |
| clarkb | fungi: there were two entries in known hosts from the old mirror01 the hostname and the ip | 02:54 |
| clarkb | and ssh via ip was still failing after I accepted the key with the hostname | 02:54 |
| clarkb | replaced the one with the ip and now it seems to work | 02:54 |
| fungi | huh, weird | 02:56 |
| *** rcernin has quit IRC | 02:57 | |
| fungi | i mean, the multiple entries is not weird because ssh normally creates one for each address you connect to in addition to the fqdn you specify | 02:57 |
| fungi | though if the address it chooses is the aaaa and that's not the one specified as the ansible host in the inventory, then yeah you need to also accept that separately | 02:58 |
| fungi | which i suspect is what was going on there | 02:58 |
| clarkb | oh that could be | 03:00 |
| clarkb | (I did ipv4 as the ansible host beacuse ipv6 needs that openstacksdk bug to be properly working) | 03:01 |
| clarkb | *bug to be fixed | 03:01 |
| fungi | yep, and if you ssh by fqdn it likely prefers the v6 aaaa | 03:01 |
| clarkb | ansible is running there now | 03:01 |
| clarkb | the openafs dkms package is being installed currently | 03:01 |
| *** xinranwang has joined #openstack-infra | 03:02 | |
| fungi | i'll stay awake long enough to find out if we need to reboot this time too | 03:03 |
| fungi | ...and hopefully why | 03:05 |
| clarkb | Jul 31 03:00:52 mirror01 openafs-client-precheck[5713]: modprobe: FATAL: Module openafs not found in directory /lib/modules/4.15.0-55-generic | 03:08 |
| clarkb | that is why openafs-client fails according to journalctl -u openafs-client | 03:08 |
| clarkb | that is the currently running kernel | 03:08 |
| clarkb | and the dkms package build should've run | 03:09 |
| fungi | so maybe dkms isn't actually building against all installed kernels after all? | 03:10 |
| fungi | no, that's the latest installed kernel package version too | 03:12 |
| clarkb | /lib/modules/4.15.0-55-generic/updates/dkms/openafs.ko exists | 03:12 |
| fungi | /lib/modules/4.15.0-55-generic/updates/dkms/openafs.ko | 03:12 |
| clarkb | maybe it is looking for the module in the wrong spot | 03:12 |
| clarkb | and a reboot loads it fine so then the service works | 03:12 |
| fungi | yeah just checked myself | 03:12 |
| fungi | maybe it tried too soon? | 03:12 |
| *** rcernin has joined #openstack-infra | 03:13 | |
| clarkb | oh ya hrm | 03:13 |
| clarkb | the timestamp on the .ko file is 3:06 | 03:13 |
| clarkb | but timestamp from journalctl is 3:00:52 | 03:13 |
| fungi | and it tried at 03:00:52 | 03:13 |
| clarkb | package bug then? | 03:13 |
| fungi | what called modprobe? | 03:14 |
| clarkb | that is openafs-client service | 03:14 |
| fungi | okay, so tried starting the service before the module was built | 03:14 |
| clarkb | ya | 03:14 |
| fungi | during package installation | 03:14 |
| clarkb | yup | 03:15 |
| fungi | that does indeed sound like a packaging bug in that case | 03:15 |
| fungi | maybe one we've introduced | 03:15 |
| clarkb | should we see if a systemctl stop openafs-client && systemctl start openafs-client fixes it? | 03:15 |
| fungi | yeah | 03:15 |
| clarkb | I'll do that now | 03:15 |
| clarkb | that seems to have done it | 03:16 |
| clarkb | service is running and /afs/openstack.org is navigable | 03:16 |
| fungi | mystery solved (mostly) | 03:16 |
| clarkb | (and openafs is loaded according to lsmod) | 03:17 |
| fungi | i'll bow out in that case. have a good night! | 03:17 |
| clarkb | good night | 03:17 |
| clarkb | the next ansible pass should get apache running and sort out the le cert | 03:17 |
| clarkb | ianw: ^ I'm about to call it a day, maybe you can check in on it in a bit | 03:17 |
| ianw | yes sorry i got called for something and totally forgot to check back on that | 03:18 |
| *** jamesmcarthur has quit IRC | 03:22 | |
| *** jamesmcarthur has joined #openstack-infra | 03:23 | |
| ianw | i don't see how ansible would start the service before the prior step finished :/ | 03:25 |
| *** jamesmcarthur has quit IRC | 03:27 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:30 | |
| ianw | Jul 31 03:06:51 mirror01 ansible-systemd: Invoked with name=openafs-client state=started enabled=True daemon_reload=False daemon_reexec=False no_block=False force=None masked=None user=None scope=None | 03:31 |
| ianw | so ansible thought the service *was* started | 03:32 |
| *** bhavikdbavishi1 has joined #openstack-infra | 03:35 | |
| *** bhavikdbavishi has quit IRC | 03:36 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 03:36 | |
| *** psachin has joined #openstack-infra | 03:37 | |
| clarkb | ya it was started but it failed | 03:46 |
| clarkb | becauseit was started prior to the module being built (Ii think) | 03:46 |
| ianw | ok it's got a cert and looks up | 03:46 |
| ianw | clarkb: but that really shouldn't have happened. installing openafs-modules-dkms should have taken 6 minutes or so | 03:47 |
| ianw | Jul 31 03:00:41 mirror01 ansible-apt[5169]: Invoked with name=['openafs-modules-dkms'] state=latest package=['openafs-modules-dkms'] | 03:47 |
| clarkb | ya but if you loom at the journalctl -u openafs-client log output it started the service at 03:00:51 ish | 03:48 |
| clarkb | which is before dkms finished andbefore ansible tried to start it | 03:48 |
| ianw | hrm, in the apt history | 03:48 |
| clarkb | which is why we were thinking the package did it? | 03:48 |
| ianw | http://paste.openstack.org/show/755154/ | 03:49 |
| ianw | that seems to suggest openafs-client is pulled in by openafs-modules-dkms | 03:49 |
| *** jamesmcarthur has joined #openstack-infra | 03:53 | |
| ianw | Recommends: openafs-client (>= 1.8.3-1~bionic) | 03:56 |
| ianw | i wonder if this is some behaviour change where we didn't previously install recommends? | 03:56 |
| *** jamesmcarthur has quit IRC | 04:00 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: openafs: avoid pulling in client package before kernel modules https://review.opendev.org/673724 | 04:03 |
| ianw | clarkb / fungi: ^ i think that, see what ci says | 04:03 |
| *** gregoryo has joined #openstack-infra | 04:07 | |
| clarkb | oh our testnodes dont install recpmmends by default but the cloud images we get from $cloud likely do? | 04:09 |
| *** slaweq has joined #openstack-infra | 04:11 | |
| *** slaweq has quit IRC | 04:16 | |
| *** whoami-rajat has joined #openstack-infra | 04:19 | |
| *** apetrich has quit IRC | 04:20 | |
| *** udesale has joined #openstack-infra | 04:20 | |
| *** udesale has quit IRC | 04:21 | |
| *** udesale has joined #openstack-infra | 04:21 | |
| ianw | maybe ... it could be something we could disable in base to make platforms more homogeneous | 04:23 |
| ianw | https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/dpkg/pre-install.d/00-disable-apt-recommends | 04:27 |
| AJaeger | config-core, please add https://review.opendev.org/#/c/673563/ and https://review.opendev.org/672290 on your review queue | 04:29 |
| *** jamesmcarthur has joined #openstack-infra | 04:30 | |
| *** ykarel has quit IRC | 04:35 | |
| *** dchen has quit IRC | 04:37 | |
| *** jamesmcarthur has quit IRC | 04:38 | |
| *** dchen has joined #openstack-infra | 04:40 | |
| *** ykarel has joined #openstack-infra | 04:52 | |
| *** tdasilva_ has quit IRC | 04:54 | |
| *** tdasilva_ has joined #openstack-infra | 04:55 | |
| *** tkajinam has quit IRC | 05:04 | |
| *** tkajinam has joined #openstack-infra | 05:05 | |
| *** ykarel has quit IRC | 05:05 | |
| *** ramishra has joined #openstack-infra | 05:06 | |
| *** n-saito has quit IRC | 05:09 | |
| *** ykarel has joined #openstack-infra | 05:13 | |
| *** ykarel is now known as ykarel|away | 05:13 | |
| *** ykarel|away has quit IRC | 05:15 | |
| *** ykarel|away has joined #openstack-infra | 05:16 | |
| *** dchen has quit IRC | 05:25 | |
| *** dchen has joined #openstack-infra | 05:26 | |
| *** gyee has quit IRC | 05:26 | |
| *** ykarel|away has quit IRC | 05:27 | |
| *** n-saito has joined #openstack-infra | 05:28 | |
| *** jaosorior has quit IRC | 05:31 | |
| *** ramishra_ has joined #openstack-infra | 05:32 | |
| *** ramishra has quit IRC | 05:33 | |
| *** jamesmcarthur has joined #openstack-infra | 05:34 | |
| *** jamesmcarthur has quit IRC | 05:39 | |
| *** threestrands has joined #openstack-infra | 05:44 | |
| openstackgerrit | Merged openstack/project-config master: Rename "tripleo-ci-centos-7-scenario007-standalone" in Neutron https://review.opendev.org/672290 | 05:45 |
| *** raukadah is now known as chandankumar | 06:04 | |
| *** odicha has joined #openstack-infra | 06:07 | |
| *** dchen has quit IRC | 06:08 | |
| *** jamesmcarthur has joined #openstack-infra | 06:08 | |
| *** slaweq has joined #openstack-infra | 06:11 | |
| *** jamesmcarthur has quit IRC | 06:15 | |
| *** slaweq has quit IRC | 06:16 | |
| *** dayou_ has quit IRC | 06:20 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: base-server: disable install of suggests and recommends packages https://review.opendev.org/673739 | 06:21 |
| *** jaosorior has joined #openstack-infra | 06:23 | |
| *** dayou_ has joined #openstack-infra | 06:24 | |
| *** pgaxatte has joined #openstack-infra | 06:29 | |
| *** iurygregory has joined #openstack-infra | 06:32 | |
| *** dchen has joined #openstack-infra | 06:55 | |
| *** smrcascao9 has quit IRC | 06:55 | |
| *** jamesmcarthur has joined #openstack-infra | 06:57 | |
| *** slaweq has joined #openstack-infra | 06:59 | |
| *** Goneri has quit IRC | 07:01 | |
| *** rcernin has quit IRC | 07:02 | |
| *** jamesmcarthur has quit IRC | 07:03 | |
| *** xek has joined #openstack-infra | 07:09 | |
| *** ginopc has joined #openstack-infra | 07:10 | |
| *** tosky has joined #openstack-infra | 07:12 | |
| *** takamatsu has quit IRC | 07:17 | |
| *** ykarel|away has joined #openstack-infra | 07:21 | |
| *** dchen has joined #openstack-infra | 07:21 | |
| *** udesale has quit IRC | 07:24 | |
| *** udesale has joined #openstack-infra | 07:24 | |
| *** ykarel|away is now known as ykarel | 07:25 | |
| *** apetrich has joined #openstack-infra | 07:26 | |
| *** tesseract has joined #openstack-infra | 07:27 | |
| *** pcaruana has quit IRC | 07:35 | |
| *** ociuhandu has joined #openstack-infra | 07:35 | |
| *** happyhemant has joined #openstack-infra | 07:36 | |
| *** jpena|off is now known as jpena | 07:37 | |
| *** igordc has joined #openstack-infra | 07:37 | |
| *** aedc has joined #openstack-infra | 07:37 | |
| openstackgerrit | Slawek Kaplonski proposed openstack/project-config master: Add new job "tempest-ipv6-only" to neutron CI dashboard https://review.opendev.org/673746 | 07:37 |
| *** rpittau|afk is now known as rpittau | 07:37 | |
| *** igordc has quit IRC | 07:38 | |
| *** ociuhandu has quit IRC | 07:40 | |
| *** dchen has quit IRC | 07:42 | |
| *** pkopec has joined #openstack-infra | 07:43 | |
| *** ociuhandu has joined #openstack-infra | 07:44 | |
| *** ociuhandu has quit IRC | 07:44 | |
| *** dchen has joined #openstack-infra | 07:46 | |
| *** ramishra_ has quit IRC | 07:49 | |
| *** dchen has quit IRC | 07:50 | |
| *** dchen has joined #openstack-infra | 07:50 | |
| *** e0ne has joined #openstack-infra | 07:58 | |
| *** betherly has joined #openstack-infra | 08:01 | |
| *** dpawlik has joined #openstack-infra | 08:02 | |
| *** electrofelix has joined #openstack-infra | 08:03 | |
| *** jaosorior has quit IRC | 08:03 | |
| *** lpetrut has joined #openstack-infra | 08:04 | |
| *** e0ne has quit IRC | 08:07 | |
| *** Goneri has joined #openstack-infra | 08:08 | |
| *** lucasagomes has joined #openstack-infra | 08:10 | |
| *** pcaruana has joined #openstack-infra | 08:13 | |
| *** ociuhandu has joined #openstack-infra | 08:15 | |
| *** dchen has quit IRC | 08:16 | |
| *** dchen has joined #openstack-infra | 08:16 | |
| *** tkajinam has quit IRC | 08:19 | |
| *** dchen has quit IRC | 08:19 | |
| *** dchen has joined #openstack-infra | 08:20 | |
| *** dchen has quit IRC | 08:22 | |
| *** gregoryo has quit IRC | 08:24 | |
| openstackgerrit | Merged openstack/project-config master: Finish retiring tc-as-a-service https://review.opendev.org/673563 | 08:30 |
| openstackgerrit | Merged openstack/project-config master: Add new job "tempest-ipv6-only" to neutron CI dashboard https://review.opendev.org/673746 | 08:30 |
| *** takamatsu has joined #openstack-infra | 08:32 | |
| rpittau | good morning everybody! sorry to be bringer of bad news but we're experiencing degradation in the ironic CI in a couple of jobs since yesterday night and I'd like to exclude there was any change on infra side that might have influenced that | 08:36 |
| rpittau | I did a first search but couldn't find anything related, would be very much appreciated to have one or more set of eyes to help :) | 08:36 |
| *** derekh has joined #openstack-infra | 08:40 | |
| *** ricolin has joined #openstack-infra | 08:40 | |
| *** dikonoor has joined #openstack-infra | 08:43 | |
| *** ykarel is now known as ykarel|lunch | 08:46 | |
| *** takamatsu_ has joined #openstack-infra | 08:48 | |
| *** threestrands has quit IRC | 08:48 | |
| *** takamatsu has quit IRC | 08:49 | |
| *** pkopec has quit IRC | 08:55 | |
| openstackgerrit | Carlos Goncalves proposed openstack/project-config master: Allow registered users to vote for backport candidates https://review.opendev.org/673764 | 08:57 |
| *** pkopec has joined #openstack-infra | 08:57 | |
| *** n-saito has quit IRC | 08:58 | |
| *** panda has quit IRC | 08:58 | |
| *** panda has joined #openstack-infra | 08:59 | |
| *** jaosorior has joined #openstack-infra | 09:00 | |
| *** priteau has joined #openstack-infra | 09:05 | |
| *** e0ne has joined #openstack-infra | 09:06 | |
| *** ociuhandu has quit IRC | 09:12 | |
| *** ociuhandu has joined #openstack-infra | 09:12 | |
| *** ralonsoh has joined #openstack-infra | 09:13 | |
| *** rfolco|ruck has joined #openstack-infra | 09:17 | |
| *** kopecmartin|off is now known as kopecmartin | 09:23 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/project-config master: Linaro London: use new bigger flavour https://review.opendev.org/671445 | 09:23 |
| *** gfidente has joined #openstack-infra | 09:26 | |
| *** happy-ms-ge has joined #openstack-infra | 09:26 | |
| *** joeguo has quit IRC | 09:27 | |
| *** [yann] has joined #openstack-infra | 09:29 | |
| *** takamatsu_ has quit IRC | 09:30 | |
| *** dikonoor has quit IRC | 09:36 | |
| *** takamatsu has joined #openstack-infra | 09:37 | |
| *** [yann] has quit IRC | 09:37 | |
| *** happy-ms-ge has quit IRC | 09:39 | |
| *** sshnaidm|afk is now known as sshnaidm | 09:42 | |
| *** betherly has quit IRC | 09:43 | |
| *** jaosorior has quit IRC | 09:50 | |
| *** ramishra has joined #openstack-infra | 09:52 | |
| *** ykarel|lunch is now known as ykarel | 09:55 | |
| *** apetrich has quit IRC | 10:05 | |
| *** takamatsu has quit IRC | 10:08 | |
| *** dtantsur|afk is now known as dtantsur | 10:23 | |
| *** takamatsu has joined #openstack-infra | 10:24 | |
| *** lpetrut has quit IRC | 10:24 | |
| *** smrcascao has joined #openstack-infra | 10:26 | |
| AJaeger | rpittau: do those failures happen all in the same region? We're making a few changes but yours is the first report that I'm aware off | 10:30 |
| rpittau | AJaeger: good question, I didn't check actually | 10:31 |
| rpittau | AJaeger: just checked and I see 2 patches failing on different regions | 10:33 |
| AJaeger | rpittau: what exactly is failing? | 10:34 |
| *** jaosorior has joined #openstack-infra | 10:34 | |
| rpittau | AJaeger: here's an example http://logs.openstack.org/98/670698/5/check/ironic-standalone/d9d3d0e/job-output.txt.gz#_2019-07-31_03_56_36_492216 | 10:34 |
| AJaeger | rpittau: that looks more like a devstack or tempest issue - I suggest you ask on #openstack-qa | 10:35 |
| rpittau | AJaeger: thanks, I will! | 10:35 |
| *** smrcascao has quit IRC | 10:35 | |
| *** betherly has joined #openstack-infra | 10:44 | |
| *** pcaruana has quit IRC | 10:45 | |
| *** ginopc has quit IRC | 11:00 | |
| *** ricolin_ has joined #openstack-infra | 11:01 | |
| *** Lucas_Gray has joined #openstack-infra | 11:01 | |
| *** ricolin has quit IRC | 11:03 | |
| *** lucasagomes is now known as lucas-hungry | 11:06 | |
| *** smrcascao has joined #openstack-infra | 11:07 | |
| *** pgaxatte has quit IRC | 11:12 | |
| *** jtomasek has joined #openstack-infra | 11:18 | |
| *** jpena is now known as jpena|lunch | 11:31 | |
| donnyd | would it be possible for someone take a look at the nodepool logs for FN. The grafana dashboard shows the instances as deleting, but they have already been deleted. | 11:35 |
| donnyd | it would seem they come up and then are immediately deleted | 11:37 |
| *** udesale has quit IRC | 11:46 | |
| *** udesale has joined #openstack-infra | 11:47 | |
| *** jaosorior has quit IRC | 11:48 | |
| openstackgerrit | Pierre Riteau proposed opendev/irc-meetings master: Fix meeting ID of Public Cloud SIG https://review.opendev.org/673793 | 11:49 |
| *** ociuhandu has quit IRC | 11:49 | |
| *** ekultails has joined #openstack-infra | 12:00 | |
| *** rh-jelabarre has joined #openstack-infra | 12:01 | |
| *** takamatsu has quit IRC | 12:10 | |
| *** pcaruana has joined #openstack-infra | 12:10 | |
| *** pgaxatte has joined #openstack-infra | 12:11 | |
| *** rlandy has joined #openstack-infra | 12:11 | |
| *** xinranwang has quit IRC | 12:11 | |
| mordred | infra-root: I'm out for a bit this morning, will be back in a few hours | 12:14 |
| *** ociuhandu has joined #openstack-infra | 12:22 | |
| *** chandankumar has quit IRC | 12:27 | |
| *** chandankumar has joined #openstack-infra | 12:28 | |
| *** jtomasek has quit IRC | 12:29 | |
| *** lucas-hungry is now known as lucasagomes | 12:30 | |
| *** jpena|lunch is now known as jpena | 12:31 | |
| *** takamatsu has joined #openstack-infra | 12:31 | |
| *** aedc has quit IRC | 12:32 | |
| *** aedc has joined #openstack-infra | 12:35 | |
| Shrews | donnyd: only thing i see atm are quota failures for fortnebula. using 20/20 | 12:37 |
| *** priteau has quit IRC | 12:39 | |
| *** psachin has quit IRC | 12:41 | |
| *** jaosorior has joined #openstack-infra | 12:43 | |
| *** bobh has joined #openstack-infra | 12:43 | |
| *** apetrich has joined #openstack-infra | 12:43 | |
| *** apetrich has quit IRC | 12:44 | |
| *** apetrich has joined #openstack-infra | 12:45 | |
| *** jamesmcarthur has joined #openstack-infra | 12:46 | |
| *** bobh has quit IRC | 12:47 | |
| *** lpetrut has joined #openstack-infra | 12:50 | |
| *** ricolin_ is now known as ricolin | 12:50 | |
| donnyd | http://grafana.openstack.org/d/3Bwpi5SZk/nodepool-fortnebula?orgId=1&from=now-30m&to=now | 12:53 |
| Shrews | donnyd: don't know what the question is behind that link, but here is another error message: openstack.exceptions.ResourceNotFound: unable to find a floating ip pool | 12:57 |
| donnyd | Well that is super helpful. I am thinking it may be trying to add FIP's to ipv6 instances | 12:59 |
| Shrews | that one seems to have started almost 2 hours ago? | 12:59 |
| *** lpetrut has quit IRC | 12:59 | |
| donnyd | yea, that is when i turned it back online | 12:59 |
| *** lpetrut has joined #openstack-infra | 12:59 | |
| *** zxiiro has quit IRC | 13:16 | |
| *** gsantomaggio has quit IRC | 13:17 | |
| *** gsantomaggio has joined #openstack-infra | 13:17 | |
| *** zxiiro has joined #openstack-infra | 13:18 | |
| *** tesseract has quit IRC | 13:20 | |
| *** aaronsheffield has joined #openstack-infra | 13:21 | |
| *** tesseract has joined #openstack-infra | 13:24 | |
| *** lseki has joined #openstack-infra | 13:25 | |
| *** mriedem has joined #openstack-infra | 13:27 | |
| *** sthussey has joined #openstack-infra | 13:27 | |
| * mordred back | 13:31 | |
| *** Lucas_Gray has quit IRC | 13:33 | |
| donnyd | It looks like it may have been an rbac thing, but i still don't see any instances going into "In Use" | 13:34 |
| *** Lucas_Gray has joined #openstack-infra | 13:38 | |
| *** ociuhandu has quit IRC | 13:39 | |
| *** ociuhandu has joined #openstack-infra | 13:39 | |
| *** liuyulong has joined #openstack-infra | 13:46 | |
| *** ykarel is now known as ykarel|afk | 13:53 | |
| donnyd | I think this is it right here | 13:53 |
| donnyd | 2019-07-31 09:53:02.664 18060 INFO neutron.wsgi [req-11a9f9b2-b5af-4b44-a5ca-b5468e8fedb1 3c109a4413ca4b68b90560093ff2d79c e8fd161dc34c421a979a9e6421f823e9 - default default] 10.0.10.240 "GET /v2.0/floatingips.json HTTP/1.1" status: 200 len: 193 time: 0.0686009 | 13:53 |
| donnyd | 2019-07-31 09:53:03.083 18060 INFO neutron.api.v2.resource [req-6124f532-88ca-4824-9efd-e5cb83e06ea1 3c109a4413ca4b68b90560093ff2d79c e8fd161dc34c421a979a9e6421f823e9 - default default] create failed (client error): The server could not comply with the request since it is either malformed or otherwise incorrect. | 13:53 |
| donnyd | its trying to stick a floating ip address to a v6 network, and that doesn't work | 13:54 |
| *** eharney has joined #openstack-infra | 13:55 | |
| *** jamesmcarthur has quit IRC | 13:55 | |
| *** jcoufal has joined #openstack-infra | 13:57 | |
| *** odicha has quit IRC | 13:57 | |
| *** dkopper has joined #openstack-infra | 13:59 | |
| *** dkopper has left #openstack-infra | 14:00 | |
| *** Lucas_Gray has quit IRC | 14:04 | |
| *** dabukalam has joined #openstack-infra | 14:05 | |
| *** michael-beaver has joined #openstack-infra | 14:24 | |
| *** betherly has quit IRC | 14:24 | |
| fungi | mordred: Shrews: ^ likely openstacksdk guessing (incorrectly) that the instances need floating ips? is there a nodepool config option where we can override that? | 14:24 |
| Shrews | fungi: i was just wondering if we need a clouds.yaml config change, something like https://opendev.org/opendev/system-config/src/branch/master/playbooks/templates/clouds/nodepool_clouds.yaml.j2#L105 | 14:25 |
| Shrews | rereading https://docs.openstack.org/openstacksdk/latest/user/config/network-config.html | 14:25 |
| fungi | ahh, right, it would be clouds.yaml not nodepool itself | 14:26 |
| donnyd | Well it should be fixed now, but it would seem that there is something wrong | 14:26 |
| donnyd | If network is v6 = don't ask for FIP | 14:26 |
| clarkb | donnyd: that is the bug I filed with sdk yesterday. It needs to check subnet info | 14:27 |
| donnyd | So same bug fix should work for both then | 14:28 |
| *** takamatsu has quit IRC | 14:28 | |
| clarkb | possibly ya | 14:29 |
| clarkb | in the first to find the ipv4 network and in the second to find there is none | 14:29 |
| mordred | clarkb, corvus, fungi: https://review.opendev.org/673831 WIP Generate mapping list for github retirements | 14:30 |
| donnyd | still seems like instances are going directly into deleting | 14:30 |
| mordred | first step - I've got a generated list of things that will get retired - and some pseudo shell (mostly correct) in the commit message for actually doing the retirement - does that look good to y'all? | 14:31 |
| *** trident has quit IRC | 14:32 | |
| clarkb | we can also do a config like limestones | 14:32 |
| clarkb | mordred: ^ this stuff largely ends up being sdk behavior with auto ip | 14:32 |
| donnyd | the instances are reachable via their ipv6 addresses, its now setup the same way as before... | 14:32 |
| mordred | also - should I do these with one of our bot accounts? | 14:33 |
| mordred | clarkb: yah - I saw the scrollback - haven't dug in to the code yet | 14:33 |
| frickler | donnyd: this is the latest error http://paste.openstack.org/show/755176/ | 14:33 |
| frickler | I think we can fix that with adding networks like in the sample Shrews mentioned | 14:33 |
| donnyd | That is interesting because the way its setup now is the way it was setup last week... I don't think any of it changed... but I also don't know where to look to see that part. is it in nl02.openstack.org.yaml | 14:34 |
| clarkb | mordred: commit msg makes sense to me | 14:36 |
| *** trident has joined #openstack-infra | 14:36 | |
| clarkb | donnyd: it hasnt changed. The two files are n02.openstack.org | 14:36 |
| clarkb | er .yaml and the one shrews linked | 14:36 |
| clarkb | I think instead cloud state has changed enough to change sdk behavior | 14:36 |
| clarkb | or maybe sdk updated | 14:36 |
| donnyd | looking at my backup (the old controller is still up, just not in service) the only different thing I could find was the quota for FIPs in the old controller was set to zero | 14:38 |
| donnyd | so I just fixed that | 14:38 |
| *** dpawlik has quit IRC | 14:38 | |
| donnyd | and will wait to see what the sdk does now | 14:38 |
| fungi | oh, yep, that might have been mitigating it before | 14:38 |
| donnyd | hrm... doesn't seem to have worked this time | 14:40 |
| AJaeger | mordred: is ansible-role-cloud-launcher on your list? See https://review.opendev.org/#/c/662530/ - we missed that in renaming | 14:40 |
| openstackgerrit | Jens Harbott (frickler) proposed opendev/system-config master: Be explicit about fortnebula networks https://review.opendev.org/673835 | 14:40 |
| corvus | mordred: lgtm. don't forget to 'mkdir openstack-infra' before you start | 14:40 |
| frickler | I think this might help ^^ | 14:41 |
| * donnyd reads openstacksdk for nat_destination parameter | 14:44 | |
| *** trident has quit IRC | 14:47 | |
| openstackgerrit | Donny Davis proposed opendev/system-config master: Adding network information to FN to support v6 only https://review.opendev.org/673838 | 14:48 |
| mordred | corvus: I just pushed an update to https://review.opendev.org/673831 WIP Generate mapping list for github retirements | 14:49 |
| mordred | corvus: with the shell now in the script - and I went a slightly different direction with the mkdir ... | 14:49 |
| mordred | corvus: I ran it with the two lines commented out to make sure the results were as expected (also, it should be safe for someone else to do that to verify) | 14:50 |
| *** trident has joined #openstack-infra | 14:50 | |
| AJaeger | mordred: what do you want to do with ansible-role-cloud-launcher ? See my comment on your review as well | 14:51 |
| *** njohnston has joined #openstack-infra | 14:52 | |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: use opendev image building system for zuul-operator test https://review.opendev.org/673020 | 14:57 |
| mordred | AJaeger: it's not in the list - and I'm not currently really touching things in the x namespace. that said, once we move cloud-launcher I think it'll be easy to just manually retire it - if we think that's ok | 14:57 |
| AJaeger | mordred: we should just remember doing it ;) | 14:58 |
| mordred | AJaeger: ++ | 14:59 |
| *** trident has quit IRC | 15:00 | |
| *** armax has joined #openstack-infra | 15:02 | |
| *** trident has joined #openstack-infra | 15:03 | |
| *** ykarel|afk is now known as ykarel | 15:03 | |
| openstackgerrit | Natal Ngétal proposed openstack/diskimage-builder master: [Configuration] Switch to stestr. https://review.opendev.org/629414 | 15:06 |
| clarkb | frickler: I'm not sure https://review.opendev.org/#/c/673835/1/playbooks/templates/clouds/nodepool_clouds.yaml.j2 is what we want because we don't want any floating ips | 15:08 |
| *** cristoph has joined #openstack-infra | 15:08 | |
| clarkb | I'm not sure how to express that in clouds.yaml though | 15:08 |
| *** trident has quit IRC | 15:08 | |
| clarkb | mordred: can you check my comment on https://review.opendev.org/#/c/673835/1 re the fip stuff above? | 15:10 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Optionally allow zoned executors to process unzoned jobs https://review.opendev.org/673840 | 15:11 |
| openstackgerrit | Merged zuul/zuul-jobs master: Add mirror_info documentation https://review.opendev.org/669948 | 15:11 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Add linter rule disallowing use of var https://review.opendev.org/673841 | 15:13 |
| donnyd | clarkb: It should be more like this one | 15:13 |
| donnyd | https://review.opendev.org/#/c/673838/1/playbooks/templates/clouds/nodepool_clouds.yaml.j2 | 15:13 |
| clarkb | ah yup floating_ip_source: None is what I was looking for thanks | 15:14 |
| donnyd | unless the networks portion is specifying the tenant network | 15:14 |
| clarkb | frickler: ^ do you want to make that update? | 15:14 |
| donnyd | I just checked it in my cloud.yaml locally and it seems to function correctly | 15:14 |
| mordred | clarkb: responded | 15:15 |
| mordred | clarkb, corvus, AJaeger: we comfortable enough with the retirement script for me to let it fly? | 15:16 |
| *** trident has joined #openstack-infra | 15:16 | |
| clarkb | mordred: maybe do it against one repo first? | 15:17 |
| AJaeger | mordred: only glanced add it and it looked fine - no time for detail review right now | 15:17 |
| mordred | clarkb: kk. I'll uncomment the lines but not the exit and see how it goes | 15:18 |
| *** jamesmcarthur has joined #openstack-infra | 15:18 | |
| *** diablo_rojo has joined #openstack-infra | 15:19 | |
| mordred | clarkb: https://github.com/openstack-infra/bindep | 15:20 |
| *** gfidente has quit IRC | 15:22 | |
| clarkb | looks good to me | 15:23 |
| *** Goneri has quit IRC | 15:24 | |
| *** trident has quit IRC | 15:24 | |
| mordred | clarkb: cool. I'm going to let it fly then | 15:24 |
| *** trident has joined #openstack-infra | 15:27 | |
| frickler | clarkb: donnyd: so we might need to set auto-floating-ip=false instead? https://softwarefactory-project.io/docs/nodepool/configuration.html#attr-providers.[openstack].pools.auto-floating-ip | 15:28 |
| Shrews | clarkb: if we don't want *any* floating ips in FN, then we could use https://zuul-ci.org/docs/nodepool/configuration.html#attr-providers.[openstack].pools.auto-floating-ip | 15:28 |
| *** kjackal has joined #openstack-infra | 15:28 | |
| frickler | Shrews: ^5 ;) | 15:28 |
| * frickler needs to go get some food, bbl | 15:29 | |
| clarkb | frickler: Shrews how is that different than floating_ip_source: none which we use in limestone? | 15:29 |
| clarkb | but ya we dont want nodepool to request floating IPs there we want it to use public ipv6 andprivate ipv4 only | 15:29 |
| frickler | clarkb: it's one (or more) level higher I think | 15:30 |
| Shrews | clarkb: i *think* it's the same thing, just higher level | 15:30 |
| fungi | mordred: looks great, thanks! | 15:31 |
| fungi | i wonder if as an additional step we should just clear out the repository descriptions too | 15:32 |
| fungi | though that requires their api, not just git | 15:32 |
| mordred | fungi: yeah. I'll maybe do another pass with the api in a little bit | 15:32 |
| donnyd | http://logs.openstack.org/38/673838/1/check/tox-linters/83aadab/job-output.txt.gz#_2019-07-31_15_10_24_341177 | 15:33 |
| donnyd | Oh I c what I did wrong | 15:34 |
| *** lpetrut has quit IRC | 15:34 | |
| *** trident has quit IRC | 15:35 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Be explicit about fortnebula networks with nodepool https://review.opendev.org/673835 | 15:37 |
| openstackgerrit | David Shrewsbury proposed openstack/project-config master: Set auto-floating-ip to False for fortnebula https://review.opendev.org/673852 | 15:37 |
| clarkb | donnyd: fungi Shrews mordred ^ that is an update based on limestone's config | 15:37 |
| Shrews | lol | 15:37 |
| *** trident has joined #openstack-infra | 15:38 | |
| clarkb | oh I see one is a nodepool config and the other is clouds.yaml | 15:38 |
| openstackgerrit | Merged zuul/zuul master: js: correct action names https://review.opendev.org/673680 | 15:38 |
| clarkb | I think we should do clouds.yaml for consistency with limestone and so that if we manually boot test instances they work as expected too | 15:38 |
| donnyd | Yea I forgot the name parameter in mine | 15:38 |
| Shrews | clarkb: ok. i'll abandon mine | 15:38 |
| donnyd | I agree clarkb | 15:39 |
| donnyd | me too | 15:39 |
| clarkb | for auto add ip false I think that makes sense if the cloud has fips but we choose not to use them | 15:39 |
| donnyd | https://review.opendev.org/#/c/673835/2/playbooks/templates/clouds/nodepool_clouds.yaml.j2 | 15:39 |
| donnyd | this one lgtm | 15:39 |
| clarkb | but in the case of cloud can't do fips we should set that in clouds.yaml | 15:39 |
| mordred | why nat_destination: true? | 15:39 |
| clarkb | mordred: it is what limestone has | 15:39 |
| donnyd | FN can do FIPs, its just explicit for certain use cases | 15:39 |
| mordred | k. | 15:40 |
| clarkb | mordred: I think because if you were to fip explicitly then you want that? | 15:40 |
| mordred | ah - yeah- that makes sense | 15:40 |
| clarkb | donnyd: sorry I had missed you had pushed a change too. I was looking at fricklers because it went by first | 15:41 |
| donnyd | makes no difference | 15:42 |
| *** jamesmcarthur has quit IRC | 15:44 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Always record the log_url in sql reporter https://review.opendev.org/673858 | 15:46 |
| *** ramishra has quit IRC | 15:47 | |
| *** bhavikdbavishi has quit IRC | 15:48 | |
| donnyd | What i really don't understand is I pulled the config directly from the old control plane that worked fine, and it was still doing the fip thing | 15:48 |
| Shrews | clarkb: i think that may require a launcher restart once merged. nodepool doesn't monitor any config changes other than its own | 15:48 |
| clarkb | Shrews: good point | 15:49 |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: Add telnet to Docker Image https://review.opendev.org/672791 | 15:49 |
| *** e0ne has quit IRC | 15:49 | |
| *** eernst has joined #openstack-infra | 15:49 | |
| openstackgerrit | Merged zuul/zuul master: js: don't transform json logfiles https://review.opendev.org/673688 | 15:50 |
| *** e0ne has joined #openstack-infra | 15:50 | |
| *** gyee has joined #openstack-infra | 15:52 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Handle artifacts without metadata on build page https://review.opendev.org/673860 | 15:56 |
| *** efried is now known as efried_rollin | 15:57 | |
| *** icarusfactor has joined #openstack-infra | 15:58 | |
| *** lucasagomes has quit IRC | 15:58 | |
| *** ociuhandu has quit IRC | 15:59 | |
| *** factor has quit IRC | 15:59 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Handle artifacts without metadata on build page https://review.opendev.org/673860 | 16:00 |
| *** pgaxatte has quit IRC | 16:11 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Try out reporting the build page https://review.opendev.org/673863 | 16:12 |
| *** lpetrut has joined #openstack-infra | 16:14 | |
| *** lpetrut has quit IRC | 16:15 | |
| *** lpetrut has joined #openstack-infra | 16:15 | |
| *** gyee has quit IRC | 16:16 | |
| *** trident has quit IRC | 16:16 | |
| *** gyee has joined #openstack-infra | 16:17 | |
| openstackgerrit | Merged zuul/zuul master: Cleanup extra /'s in manifest render view https://review.opendev.org/673605 | 16:17 |
| mordred | infra-root: openstack-infra repos on github are all retired | 16:18 |
| corvus | mordred: zomg! | 16:18 |
| *** rpittau is now known as rpittau|afk | 16:18 | |
| * fungi celebrates | 16:18 | |
| *** dtantsur is now known as dtantsur|afk | 16:19 | |
| *** xek has quit IRC | 16:20 | |
| fungi | cncf staff are having a minor existential crisis as they're increasingly unable to track activity for projects listed in their "cloud-native landscape" because their software only knows how to check metrics from the github api and an ever growing number of projects are moving off github and not mirroring their source code to it either | 16:20 |
| fungi | can't say i'm terribly empathetic there | 16:21 |
| mordred | whoops | 16:21 |
| *** trident has joined #openstack-infra | 16:21 | |
| mordred | it's almost like git is a distributed version control system | 16:21 |
| corvus | fungi: what's the "D" in dvcs stand for again? | 16:22 |
| corvus | mordred: jinx | 16:22 |
| *** electrofelix has quit IRC | 16:22 | |
| *** kjackal has quit IRC | 16:22 | |
| fungi | indeed | 16:22 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Try out reporting the build page https://review.opendev.org/673863 | 16:24 |
| *** kjackal has joined #openstack-infra | 16:25 | |
| *** ociuhandu has joined #openstack-infra | 16:35 | |
| *** chandankumar is now known as raukadah | 16:36 | |
| *** igordc has joined #openstack-infra | 16:36 | |
| *** kjackal has quit IRC | 16:37 | |
| *** lpetrut has quit IRC | 16:37 | |
| *** jamesmcarthur has joined #openstack-infra | 16:38 | |
| *** eernst has quit IRC | 16:38 | |
| *** panda has quit IRC | 16:39 | |
| *** ykarel is now known as ykarel|away | 16:40 | |
| openstackgerrit | Merged zuul/zuul master: Attempt to report CORS and related errors https://review.opendev.org/673707 | 16:41 |
| *** panda has joined #openstack-infra | 16:41 | |
| *** diablo_rojo has quit IRC | 16:46 | |
| *** diablo_rojo has joined #openstack-infra | 16:46 | |
| openstackgerrit | Merged zuul/zuul-operator master: Verify Operator Pod Running https://review.opendev.org/670395 | 16:47 |
| *** takamatsu has joined #openstack-infra | 16:47 | |
| *** trident has quit IRC | 16:48 | |
| *** diablo_rojo has quit IRC | 16:48 | |
| *** lpetrut has joined #openstack-infra | 16:48 | |
| *** diablo_rojo has joined #openstack-infra | 16:49 | |
| *** jtomasek has joined #openstack-infra | 16:50 | |
| *** ricolin has quit IRC | 16:50 | |
| *** trident has joined #openstack-infra | 16:51 | |
| *** ociuhandu has quit IRC | 16:52 | |
| *** jpena is now known as jpena|off | 16:57 | |
| *** derekh has quit IRC | 16:57 | |
| *** takamatsu has quit IRC | 16:59 | |
| *** jamesmcarthur has quit IRC | 17:00 | |
| *** sshnaidm is now known as sshnaidm|afk | 17:00 | |
| *** lpetrut has quit IRC | 17:01 | |
| *** diablo_rojo_ has joined #openstack-infra | 17:01 | |
| AJaeger | mordred: want to update the title of https://github.com/openstack-infra, it reads "These are mirrors of projects hosted at https://git.openstack.org/cgit/openstack-infra/ related to the OpenStack community's project infrastructure." | 17:01 |
| openstackgerrit | Merged zuul/zuul-jobs master: Add auth config to kubelet user for buildset registries https://review.opendev.org/673351 | 17:01 |
| AJaeger | mordred: there's still https://github.com/openstack-infra/gitdm | 17:02 |
| mordred | AJaeger: interesting. and yes - I'll get the other thing | 17:02 |
| AJaeger | mordred: and https://github.com/openstack-infra/odsreg - go to the "last" entries in the list, some might not have been retired properly in the past... | 17:04 |
| *** iurygregory has quit IRC | 17:04 | |
| *** diablo_rojo has quit IRC | 17:04 | |
| *** ykarel|away has quit IRC | 17:06 | |
| *** icarusfactor has quit IRC | 17:10 | |
| *** diablo_rojo_ has quit IRC | 17:10 | |
| *** diablo_rojo has joined #openstack-infra | 17:11 | |
| *** factor has joined #openstack-infra | 17:11 | |
| mordred | AJaeger: maybe these are things we retired before the transition, so they weren't in the rename yamls | 17:14 |
| mordred | AJaeger: OH - I know what it is | 17:15 |
| openstackgerrit | Merged zuul/zuul master: Allow to select the merge method in Github https://review.opendev.org/617949 | 17:15 |
| mordred | AJaeger: these are all things that got renamed to x/ or osf/ or something similar | 17:15 |
| mordred | and I didn't touch those because where wasn't really a way in the script to know whether they were still active, if someone was still using them, etc | 17:16 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Try out reporting the build page https://review.opendev.org/673863 | 17:16 |
| *** e0ne has quit IRC | 17:18 | |
| fungi | mordred: however we do at least know the copies of them on gh are no longer being updated | 17:18 |
| openstackgerrit | Merged zuul/zuul master: Support squash merge in Github https://review.opendev.org/661096 | 17:20 |
| *** ociuhandu has joined #openstack-infra | 17:21 | |
| mordred | fungi, AJaeger: http://paste.openstack.org/show/755182/ | 17:24 |
| mordred | I think that's the new/missing list | 17:24 |
| mordred | if that looks reasonable, I'll re-run with that | 17:26 |
| *** ociuhandu has quit IRC | 17:26 | |
| fungi | heh, i was trying to load that in gertty and it went all crashytown on me... not a change url! | 17:26 |
| mordred | fungi: hahaha | 17:27 |
| *** udesale has quit IRC | 17:27 | |
| fungi | i wonder if the jjb maintainers have somewhere in gh they want that repo transferred | 17:28 |
| mordred | fungi: they seem to be using opendev | 17:28 |
| mordred | and I don't see any mention in their repo of mirroring to github anywhere | 17:28 |
| fungi | cool, probably fine then | 17:29 |
| mordred | (also, if they do, we can always transfer the repo and then the gerrit force-push will fix it all up :) ) | 17:29 |
| fungi | we can always transfer it for them later either way | 17:29 |
| mordred | yeah | 17:29 |
| fungi | yep, zactly | 17:29 |
| fungi | list lgtm | 17:29 |
| *** factor has quit IRC | 17:29 | |
| mordred | cool. I'm going to run these | 17:29 |
| AJaeger | thanks, mordred | 17:29 |
| mordred | fwiw - we don't have openstack/git-review -> opendev/git-review captured in our rename files | 17:30 |
| AJaeger | oh ;( Could you send an update, please? | 17:30 |
| fungi | i thought it was openstack-infra/git-review? | 17:31 |
| fungi | could that be the difference? | 17:31 |
| *** igordc has quit IRC | 17:32 | |
| *** cristoph has quit IRC | 17:35 | |
| clarkb | mordred: nice | 17:36 |
| clarkb | dinner is now in the crock pot /me pays attention to here again | 17:36 |
| openstackgerrit | Merged zuul/zuul master: Builds page - Fix bad labels display https://review.opendev.org/672973 | 17:40 |
| *** kjackal has joined #openstack-infra | 17:42 | |
| mordred | clarkb, fungi, AJaeger, corvus: how does https://github.com/openstack-infra/jenkins-job-builder look. I'm thinking the url in the description can go, can just set the description to RETIRED and the url field to the url | 17:44 |
| clarkb | mordred: ya the double up url looks funny | 17:45 |
| mordred | clarkb: better now? | 17:47 |
| clarkb | mordred: it is abbreviated, maybe put the url in the description instead (I don't think it was truncated there before | 17:48 |
| mordred | nod | 17:48 |
| openstackgerrit | Luigi Toscano proposed zuul/zuul-jobs master: fetch-subunit-output: collect additional subunit files https://review.opendev.org/673885 | 17:49 |
| openstackgerrit | Luigi Toscano proposed zuul/zuul-jobs master: fetch-subunit-output: collect additional subunit files https://review.opendev.org/673885 | 17:49 |
| mordred | clarkb: ok. yeah - that looks better - although I think I should make it MOVED: not RETIRED: | 17:50 |
| mordred | how does that look now? if that's good, I'm going to run that on the whole list | 17:51 |
| openstackgerrit | Merged opendev/system-config master: Be explicit about fortnebula networks with nodepool https://review.opendev.org/673835 | 17:51 |
| clarkb | ya that looks good | 17:51 |
| AJaeger | mordred: LGTM, thanks | 17:51 |
| mordred | cool. | 17:52 |
| donnyd | clarkb: please let me know when the launcher is restarted a nd I will turn the quota back up | 17:52 |
| clarkb | donnyd: will do | 17:53 |
| mordred | fwiw - doing this just with plain requests is very easy | 17:53 |
| *** sgw has quit IRC | 17:53 | |
| mordred | ok. they're all done except for https://github.com/openstack-infra?page=6 | 17:55 |
| AJaeger | what about https://github.com/openstack-infra/git-review ? | 17:56 |
| AJaeger | rest looks fine | 17:57 |
| *** ralonsoh has quit IRC | 17:57 | |
| openstackgerrit | Merged zuul/zuul master: Fix reference pipelines syntax coloration for Pagure driver https://review.opendev.org/672677 | 17:57 |
| *** tesseract has quit IRC | 18:05 | |
| *** goldyfruit has quit IRC | 18:11 | |
| mordred | AJaeger, clarkb: I believe they are all now fully archived. I included all the scripts and local working files I used. | 18:11 |
| mordred | mostly for documentation purposes in case someone else wants to do something similar - I did not spend time to make them good or flexible (as evidenced by the multiple copies) | 18:12 |
| AJaeger | mordred, thanks! | 18:13 |
| fungi | thanks mordred, looks great! | 18:13 |
| AJaeger | mordred: want to abandon your change now? | 18:13 |
| mnaser | does the opendev tenant have the ability to publish docs somewhere? | 18:13 |
| fungi | mnaser: we have a docs.opendev.org site intended for that purpose | 18:13 |
| mnaser | oh wonderful | 18:14 |
| fungi | mnaser: or you could choose to publish to read-the-docs or something | 18:14 |
| mordred | AJaeger: done | 18:15 |
| AJaeger | fungi: is docs.opendev.org up already? | 18:15 |
| corvus | AJaeger: yes any project can use it | 18:15 |
| fungi | right now only a couple of the opendev namespace repos are publishing there: https://docs.opendev.org/opendev/ | 18:15 |
| AJaeger | yeah, it's up - just has no index page... | 18:15 |
| fungi | i don't know if it's considered ready for prime time, nor which zuul tenants are supported | 18:16 |
| corvus | the jobs will work in any zuul tenant | 18:16 |
| fungi | awesome | 18:16 |
| corvus | they'll publish to the full opendev path (eg if nova used it, docs.opendev.org/openstack/nova) | 18:16 |
| corvus | (i'm not suggesting that openstack switch) | 18:16 |
| corvus | (just an example) | 18:16 |
| *** altlogbot_3 has quit IRC | 18:16 | |
| fungi | and yeah as far as i know we consider those the canonical locations for the opendev/base-jobs and opendev/infra-specs documentation builds | 18:17 |
| clarkb | donnyd: I have restarted the nodepool launcher on nl02 (confirmed the new clouds.yaml was written first too) | 18:17 |
| corvus | i have no idea what an index page should look like. we can totally add one if someone has an opinion? | 18:17 |
| fungi | not to be pedantic, but it already has an index page (courtesy of apache mod_autoindex) | 18:17 |
| corvus | fungi: hehe, you failed at not being pedantic :) | 18:18 |
| fungi | i find that perfectly serviceable, and it can even be themed if we so desire via header/footer/readme injection options in apache | 18:18 |
| corvus | but, yes, if someone has an idea for a *nice* index page we can do so. i'm ambivalent whether we should actually have directory listings | 18:18 |
| corvus | i sort of think that most projects would just deep-link to their docs | 18:19 |
| fungi | mod_autoindex is far more flexible than many folks realize | 18:19 |
| fungi | you can also do custom icons and stuff with it | 18:19 |
| *** altlogbot_3 has joined #openstack-infra | 18:20 | |
| clarkb | corvus: that and google et al do a really good job of deep linking when you search for stuff too | 18:20 |
| fungi | i used to have a wordpress-based site for a personal project and wrapped mod_autoindex in the wordpress site theming so it blended right in | 18:20 |
| clarkb | donnyd: looks like we have an in use node that seems happy | 18:21 |
| fungi | (actually called a cgi which fetched and cached the site template and generated the header/footer from that and rendered the readme via docutils) | 18:21 |
| zbr | clarkb: does https://review.opendev.org/#/c/631509/ still needs extra testing before wf? not that feel the need for it but I am doing some summer cleanup on open reviews. | 18:22 |
| openstackgerrit | Merged zuul/zuul master: Return dependency cycle failure to user https://review.opendev.org/672487 | 18:24 |
| clarkb | zbr: I don't see that it was ever added to review_dev.pp ? | 18:24 |
| zbr | ahh....! doing it now. | 18:25 |
| donnyd | clarkb: finally | 18:27 |
| donnyd | Replacing the control plane has been a good learning experience for me, and i truly appreciate everyone in infra's patience in getting it all back up and running | 18:28 |
| mordred | donnyd: it's been really interesting to get the insight into all of the stuff you've been doing | 18:30 |
| mordred | so thanks for being so engaged! | 18:30 |
| openstackgerrit | Sorin Sbarnea proposed opendev/system-config master: Recognize DISK_FULL failure messages (review_dev) https://review.opendev.org/673893 | 18:31 |
| clarkb | ++ I think we've learned a lot about openstack in the process too | 18:31 |
| donnyd | Well I am trying to keep this all as open book as possible. | 18:31 |
| openstackgerrit | Sorin Sbarnea proposed opendev/system-config master: Recognize DISK_FULL failure messages https://review.opendev.org/631509 | 18:31 |
| donnyd | Hopefully it will get archived and someone down the road can see what we did to get it all working | 18:31 |
| clarkb | zbr: once ^ merges you can leave a comment on review-dev.o.o that mimics a zuul abort, then we'll see it works thenwe approve the production update | 18:31 |
| zbr | ok. | 18:32 |
| zbr | btw, I seen that there are lots of differences between dev and non dev. | 18:32 |
| zbr | somehow they diverged | 18:32 |
| clarkb | I think because we haven't done a great job in applying to -dev first in the past | 18:33 |
| clarkb | donnyd: I now see ~17 instances building | 18:33 |
| zbr | another older review that makes sense to go in https://review.opendev.org/#/c/652708/2 | 18:34 |
| clarkb | and now many are in use and ready | 18:34 |
| donnyd | clarkb: I think the fix is in... Now I just hope all the work was worth it | 18:35 |
| mordred | clarkb, zbr: I'm hoping we can have these be more in-sync once we get these latest updates applied | 18:35 |
| donnyd | Also no error node launch attempts | 18:35 |
| *** njohnston has quit IRC | 18:35 | |
| clarkb | mordred: ya a lot of that config can likely go into the image | 18:36 |
| *** igordc has joined #openstack-infra | 18:36 | |
| *** e0ne has joined #openstack-infra | 18:37 | |
| mordred | fungi: the scripts I pushed up in should be a basis for someone doing something similar with the things in the openstack namespace that are no longer being replicated from opendev and have moved to x/ ... I didn't touch those in this round because I wasn't sure what people wanted to do - but it turns out it was pretty easy to deal with once I dug in | 18:38 |
| fungi | awesome, thanks | 18:40 |
| clarkb | donnyd: http://mirror.regionone.fortnebula.opendev.org/pypi/simple/ seems to timeout that is a proxy for pypi.org which appears to only have A records in dns | 18:42 |
| clarkb | I'm about to test if I can hit ipv4 addrs on the internet from that mirror | 18:42 |
| mnaser | hmm | 18:42 |
| fungi | mordred: what was the review number? | 18:42 |
| clarkb | oh wget shows it using an ipv6 addr | 18:42 |
| mnaser | whats the difference between tox-docs and opendev-tox-docs ? | 18:42 |
| clarkb | so maybe I just fail at dig | 18:42 |
| *** takamatsu has joined #openstack-infra | 18:43 | |
| clarkb | donnyd: ya traceroute6 shows nothing after 2001:470:e045:2::1 to pypi.org or google.com | 18:43 |
| openstackgerrit | Merged zuul/zuul master: Add Authorization Rules configuration https://review.opendev.org/639855 | 18:43 |
| donnyd | ok, lemme take a look see | 18:44 |
| clarkb | and I have no route to host ssh in via ipv6 | 18:44 |
| clarkb | (I remember testing that yseterday and it worked) | 18:44 |
| fungi | mnaser: comment here is elucidating: https://opendev.org/opendev/base-jobs/src/branch/master/zuul.d/jobs.yaml#L312-L313 | 18:45 |
| donnyd | ok try again | 18:46 |
| donnyd | im thinking maybe the bgp-agent was hung | 18:47 |
| donnyd | may have been mad about my earlier shenanigans | 18:47 |
| donnyd | looks like the link is working now, but I may need to keep an eye on it | 18:47 |
| mnaser | fungi: i guess that makes sense, so ill run opendev-tox-docs + tox-linters | 18:48 |
| clarkb | donnyd: works now | 18:48 |
| mnaser | seem annoyingly not clean, but eh /me shrugs | 18:48 |
| donnyd | 37 in use now, still no error launch attempts.. I think its safe to say your patch was the fix and it should be good again | 18:49 |
| clarkb | mnaser: fungi specifically I believe that opendev-tox-docs was the guinea pig for doing the gate build of docs artifacts then using promote to publish them | 18:50 |
| clarkb | I expect that tox-docs will get migrated over to that? but maybe not. I haven't thought about the backward compat ramifications of that | 18:50 |
| AJaeger | mnaser, as clarkb said: In OpenStack we build in check, build in gate, build again in post and publish | 18:50 |
| fungi | yes, there are several related docs jobs in that same jobs.yaml | 18:51 |
| clarkb | donnyd: yup looking good | 18:51 |
| AJaeger | and in opendev it's build in check, build in gate - and then instead of post building, we publish the result of gate. So, that needs a different post role | 18:51 |
| donnyd | and we have at least one successful build | 18:51 |
| donnyd | http://logs.openstack.org/44/672244/1/gate/openstack-tox-docs/ed8a8c1/job-output.txt | 18:51 |
| *** jtomasek has quit IRC | 18:51 | |
| mnaser | AJaeger: oh thats interesting, do i have the right combo here? https://review.opendev.org/#/c/673896/1/zuul.d/project.yaml | 18:51 |
| AJaeger | mnaser: no, you did not - the publish is for release pipeline, see https://opendev.org/opendev/base-jobs/src/branch/master/zuul.d/jobs.yaml#L333 | 18:53 |
| AJaeger | mnaser: left a comment | 18:53 |
| donnyd | clarkb: http://logs.openstack.org/94/94777977e940025f2a9f3bf0693c8a13f5ae7148/post/publish-openstack-releasenotes-python3/7301ab0/job-output.txt#_2019-07-31_18_46_11_820118 | 18:54 |
| donnyd | the mirror is still very slow | 18:55 |
| clarkb | donnyd: that is going through afs which may mean the cache is stale | 18:55 |
| AJaeger | mnaser: your zuul config looks fine now | 18:57 |
| clarkb | donnyd: I'm going to double check the mirrors cache size | 18:57 |
| donnyd | I know we want to keep all the mirrors the same for good reasons, that node could be massively increased... I don't have as many resources as the other providers, but if it was faster it could output the same in terms of jobs completeted | 18:58 |
| clarkb | AFS using 933247 of the cache's available 50000000 1K byte blocks. <- from fs getcacheparms. that looks good to me | 18:59 |
| donnyd | The good news is my control plane issues seemed to be solved, no rabbit timeouts or build failures (yet) | 18:59 |
| donnyd | ok cool, so we just need to wait for the cache to build up | 18:59 |
| clarkb | donnyd: it could also be something elsewith afs, I'm not entirely convinced throwing more resources at the problem will fix it just yet. Lets watch it and see if it gets better as caches warm up | 18:59 |
| donnyd | ok sounds like a good plan to me | 19:00 |
| donnyd | its only been running jobs for a few minutes.. maybe revisit tomorrow ? | 19:00 |
| clarkb | sure | 19:00 |
| donnyd | or instance i could give the mirror node 120G of memory | 19:01 |
| donnyd | for instance* | 19:01 |
| donnyd | just keep it in mind in case we want to check it out | 19:01 |
| donnyd | But I am a happy camper | 19:01 |
| *** e0ne has quit IRC | 19:02 | |
| clarkb | fetching http://mirror.regionone.fortnebula.opendev.org/ubuntu/lists/ubuntu_bionic_universe_amd64_Packages (which is 48MB large) on the mirror itself: first uncached request is done at ~1MBps. Second at 351MBps. | 19:04 |
| clarkb | (to give a sense for impact of the afs cache) | 19:05 |
| clarkb | if we continue to see slowness I think we'd have to look at networking given those numbers? | 19:05 |
| donnyd | :) yea that is a 351X increase in speed | 19:05 |
| fungi | clarkb: was that on a cache miss or hit? | 19:06 |
| clarkb | fungi: first request is a miss second is a hit | 19:06 |
| clarkb | (first request populates the cache) | 19:06 |
| fungi | oh, i misread | 19:06 |
| *** kjackal has quit IRC | 19:06 | |
| fungi | yeah | 19:06 |
| fungi | those numbers look like what i expect in that case | 19:06 |
| *** whoami-rajat has quit IRC | 19:08 | |
| clarkb | http://logs.openstack.org/48/673848/1/check/cross-nova-py36/b0e0e5b/job-output.txt is the only failure so far according to logstash post ipv6 fix | 19:08 |
| clarkb | and those look like valid unittest failures | 19:08 |
| donnyd | I was just looking at that :) yea, it would seem like everything is functioning properly and we are also back up to 60 with node issues on the control plane | 19:11 |
| *** e0ne has joined #openstack-infra | 19:12 | |
| openstackgerrit | Merged zuul/zuul master: Web: plug the authorization engine https://review.opendev.org/640884 | 19:12 |
| *** kjackal has joined #openstack-infra | 19:12 | |
| donnyd | this weekend I am going to work on getting a faster glance system, but it should be transparent to everything else | 19:14 |
| donnyd | And maybe a couple more hypervisors so I can get that # up around 100 | 19:16 |
| clarkb | exciting | 19:17 |
| donnyd | and somewhere in there get some logging aggregation so i can see what is busted on my end faster... last weekend I setup metrics so I can actually tell what is going on with loads | 19:18 |
| fungi | ceilometer/gnocchi or something else? | 19:18 |
| donnyd | Yes + zabbix | 19:22 |
| donnyd | Don't quite know how to operated gnocchi yet though | 19:22 |
| donnyd | Like I don't even know where to start with building my query | 19:23 |
| donnyd | But the api is up and has been collecting the things | 19:23 |
| fungi | exciting | 19:23 |
| mnaser | before i merge https://review.opendev.org/#/c/673896/ | 19:24 |
| mnaser | is there anything that needs to be setup for publishing to docs.opendev.org ? | 19:24 |
| donnyd | This is what I have for zabbix so far https://usercontent.irccloud-cdn.com/file/IzWFtdKx/zabbix%20dashboard | 19:24 |
| *** e0ne has quit IRC | 19:24 | |
| mnaser | i assume docs.opendev.org/vexxhost/ will probably automagically created? | 19:24 |
| *** pabelanger has joined #openstack-infra | 19:25 | |
| AJaeger | mnaser: go for it ;) | 19:25 |
| mnaser | ok im blaming AJaeger if it all comes burning down :) | 19:25 |
| AJaeger | mnaser: it should just work :) if not, you found a new bug :) | 19:25 |
| mnaser | cool cool | 19:25 |
| * AJaeger calls it a day soon, so what's the saying "after me..." ;) | 19:26 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Always record the log_url in sql reporter https://review.opendev.org/673858 | 19:28 |
| *** goldyfruit has joined #openstack-infra | 19:34 | |
| *** liuyulong has quit IRC | 19:35 | |
| corvus | clarkb, fungi, mordred: is there a LE changes for logs.o.o? | 19:35 |
| mnaser | oh cool, it worked.. https://docs.opendev.org/vexxhost/openstack-monitoring/latest/ | 19:35 |
| clarkb | corvus: I dont rhink so | 19:35 |
| corvus | mnaser: \o/ | 19:35 |
| AJaeger | mnaser: yeah! | 19:35 |
| mnaser | fyi something we lack is a "how to contribute to opendev-hosted projects" document | 19:36 |
| mnaser | i couldnt really find anything that fit | 19:36 |
| AJaeger | mnaser: then I can call it a day without regrets ;) | 19:36 |
| corvus | lemme see if i can get that going then, cause i think that's probably what we're going to want to do to resolve the zuul->logs https thing? | 19:36 |
| donnyd | mnaser: is this kinda like ops-tools? | 19:36 |
| mnaser | donnyd: a bit yeah, it's playbooks that do monitoring for openstack-ansible deployed clouds | 19:36 |
| mnaser | and we're moving all of our internal monitroing to be hosted in that repo so yeah | 19:36 |
| donnyd | I could surely use some monitoring, does it only work for OSA deployed clouds? | 19:36 |
| fungi | corvus: yep, i can help or review as soon as the storyboard meeting wraps up | 19:37 |
| openstackgerrit | Sorin Sbarnea proposed opendev/gerritbot master: Avoid silent execution failures https://review.opendev.org/618802 | 19:37 |
| openstackgerrit | Matt McEuen proposed openstack/project-config master: New project request: airship/porthole https://review.opendev.org/673898 | 19:37 |
| openstackgerrit | Luigi Toscano proposed zuul/zuul-jobs master: fetch-subunit-output: collect additional subunit files https://review.opendev.org/673885 | 19:38 |
| *** tdasilva_ has quit IRC | 19:39 | |
| *** tdasilva_ has joined #openstack-infra | 19:40 | |
| mnaser | donnyd: it makes certain assumption thats the variables used are for OSA | 19:40 |
| mnaser | i am sure you can fill those out and run the playbooks and itll be just fine | 19:40 |
| mnaser | such as group names / etc | 19:40 |
| *** kopecmartin is now known as kopecmartin|off | 19:41 | |
| donnyd | I will give it a spin mnaser :) | 19:42 |
| donnyd | thanks | 19:42 |
| fungi | donnyd: out of curiosity, what did you deploy with? i forgot, if you said | 19:43 |
| donnyd | overwhelming kinetic force.. | 19:43 |
| fungi | hah | 19:43 |
| fungi | deck-mounted cannon | 19:44 |
| donnyd | I have some playbooks I have been working on that deploy openstack in the simplest possible way from a sysadmin perspective | 19:44 |
| corvus | fungi, clarkb: so... logs.openstack.org is in the wrong domain to be LEncrypted | 19:44 |
| fungi | corvus: i thought the idea was to add a logs.opendev.org vhost parallel to it? | 19:44 |
| corvus | oh? i missed that | 19:44 |
| clarkb | fungi that wasmy suggestion yesterday | 19:45 |
| corvus | that seems like it maybe should work? | 19:45 |
| donnyd | I need to get my head wrapped around OSA in the near term | 19:45 |
| corvus | i'll give that a shot | 19:45 |
| clarkb | ya wemight have to set cors rules | 19:45 |
| clarkb | but it should work | 19:45 |
| openstackgerrit | Matt McEuen proposed openstack/project-config master: New project request: airship/kubernetes-entrypoint https://review.opendev.org/673900 | 19:45 |
| corvus | i assume it would be mostly a copy of the existing vhost which does have cors rules | 19:45 |
| mordred | copy/pasta the logs.openstack vhost and it should be fine | 19:45 |
| mordred | yeah | 19:45 |
| corvus | i'll give it a shot | 19:46 |
| donnyd | I'm also trying to work out what switches you can flip in Openstack configs that actually yield better performance for CI workloads than the defaults | 19:46 |
| mordred | should we have http://logs.openstack.org and http://logs.opendev.org throw redirects to https://logs.opendev.org? or is that too much? | 19:47 |
| *** takamatsu has quit IRC | 19:47 | |
| corvus | i don't think it'd hurt, but, you know, later, after we know it works :) | 19:47 |
| fungi | i would start simple. we can always add it | 19:47 |
| *** efried_rollin is now known as efried | 19:47 | |
| mordred | hah. THROW ALL THE SWITCHES | 19:47 |
| openstackgerrit | Merged zuul/zuul master: Handle artifacts without metadata on build page https://review.opendev.org/673860 | 19:48 |
| fungi | corvus: i want to say we did something similar recently to add am opendev.org vhost with le https to a server which already has a openstack.org vhost, but i'm struggling to remember which one it was | 19:48 |
| donnyd | I am thining OSA and tripleo have already worked out what works best for cloud at big scale... I am trying to work out what works for cloud at small scale | 19:48 |
| fungi | maybe i can find it in the dns zone | 19:48 |
| donnyd | like right between edge and public cloud | 19:48 |
| corvus | fungi: tarballs? | 19:48 |
| *** eharney has quit IRC | 19:48 | |
| corvus | is cname to files.openstack | 19:48 |
| mordred | clarkb: how did you generate the password for the github openstackadmin account? | 19:49 |
| fungi | corvus: tarballs, yes | 19:49 |
| fungi | just found it | 19:49 |
| donnyd | been watching the logs clarkb and i have only seen just that one failure, so time to move on (until it breaks again) | 19:49 |
| corvus | fungi: i'm um, having trouble typing on account of i can't stop laughing | 19:50 |
| fungi | yeah, sorry :/ | 19:50 |
| corvus | i can't wait for the next surprise game of "tarballs!" (see #openstack-meeting for the lols) | 19:52 |
| *** jcoufal has quit IRC | 19:52 | |
| fungi | sometimes lack of context can be a blast | 19:52 |
| mordred | clarkb: nevermind | 19:53 |
| *** jtomasek has joined #openstack-infra | 19:53 | |
| *** e0ne has joined #openstack-infra | 19:55 | |
| corvus | is the ansible inventory glob "static[0-9]*.open*.org" going to match static.openstack.org ? | 19:56 |
| *** tdasilva_ is now known as tdasilva | 19:56 | |
| corvus | no | 19:57 |
| mordred | infra-root: I have created the opendevadmin account on github | 19:59 |
| clarkb | mordred: ianw did it | 19:59 |
| mordred | clarkb: cool. well, I figured out something similar :) | 20:00 |
| openstackgerrit | James E. Blair proposed opendev/zone-opendev.org master: Add logs.opendev.org https://review.opendev.org/673902 | 20:00 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Add LE cert for logs.opendev.org to static https://review.opendev.org/673903 | 20:01 |
| fungi | corvus: correct, it will not because there is no digit in the name | 20:01 |
| weshay | is opendev/x accepting new projects at this time? just double checking as the doc says not quite yet | 20:01 |
| fungi | weshay: i think the hope is that folks adding new projects will make new namespaces, but i don't recall if we've come to any consensus on whether new projects can be added to the "x" prefix namespace | 20:03 |
| corvus | weshay: which doc? | 20:03 |
| weshay | ah k | 20:03 |
| weshay | https://opendev.org/ | 20:03 |
| weshay | Yes! However, as noted above it is still early days yet and the early experience might be a bit bumpy. Certain things may still say “OpenStack” on them as we figure out the transition. And while any moves should come with appropriate redirects, we may have some inadvertent misses. | 20:03 |
| weshay | you guys are well past the transition.. just wanted to check in before I told someone to go do a thing | 20:04 |
| fungi | we're past the cgit to gitea transition | 20:04 |
| fungi | but there's still a lot of general opendev resources which haven't been renamed (etherpad for example) | 20:04 |
| fungi | we've just added a tarballs.opendev.org and docs.opendev.org | 20:05 |
| fungi | and corvus is working on a logs.opendev.org vhost in 673902 and 673903 | 20:05 |
| mordred | clarkb: I have made opendevadmin the owner of opendevorg ... should we go ahead and remove openstackadmin? and is there anything else we wanted to make opendevadmin own? | 20:05 |
| clarkb | mordred: might be nice to have that account in openstack-infra/ in case we have to make changes there (migrating jjb out I dunno) and openstackadmin is more openstacky | 20:07 |
| mordred | clarkb: ++ | 20:08 |
| fungi | corvus: just to double-check since it's a different file and the commit message didn't mention it, but your fixes to the le doc were intended to be in that patch right? not an accidental squash from another change? | 20:10 |
| corvus | fungi: yes, sorry i forgot to mention in the commit msg | 20:10 |
| fungi | no worries, just double-checking it was intentional | 20:11 |
| fungi | both changes lgtm, though if something's missing i guess we'll find out | 20:11 |
| openstackgerrit | Merged opendev/zone-opendev.org master: Add logs.opendev.org https://review.opendev.org/673902 | 20:11 |
| clarkb | fungi: the new vhost is missing but that can happen in a followup | 20:12 |
| *** e0ne has quit IRC | 20:12 | |
| corvus | yeah i'm writing that now | 20:12 |
| *** jrist has quit IRC | 20:12 | |
| corvus | i figured it'd be good to get that merged and the cert on disk before the vhost is in place anyway | 20:12 |
| clarkb | ya the ordering you've got is fine | 20:12 |
| corvus | the vhost change is not small | 20:13 |
| clarkb | mriedem: do you know if http://logs.openstack.org/57/673557/2/check/grenade-py3/64f9d83/logs/grenade.sh.txt.gz#_2019-07-31_19_50_55_555 is a known issue? grenade failures testing cinder boot from volume | 20:15 |
| clarkb | looks like maybe cinder hasn't fully started again based on log timestamps of the cinder volume service? | 20:16 |
| clarkb | (I'm not sure if that is necessary for the volume to function) | 20:16 |
| donnyd | Is that infra related? | 20:17 |
| mriedem | clarkb: i'd chalk that up to ssh failures in the guest yeah? like we always see in tempest runs | 20:17 |
| fungi | clarkb: the lack of vhost was mentioned in the change | 20:17 |
| mriedem | like maybe related to slaweq's patch to use memcache for the nova-meta api | 20:17 |
| clarkb | donnyd: ssh failures to guests are an unfortunately common problem in these tests wanting to rule out fortnebula but I'm fairly certain fn isn't the cause | 20:17 |
| clarkb | mriedem: oh interesting | 20:17 |
| donnyd | curious if the v6 stuff has anything to do with it | 20:18 |
| mriedem | iow, it's not a cinder thing, it's just a boot from volume scenario where cinder is involved, | 20:18 |
| mriedem | but so is nova and neutron | 20:18 |
| clarkb | donnyd: probably not, this test runs on a single instance and everything is self contained | 20:18 |
| clarkb | donnyd: if it was a multinode job then it could be but being single node none of that network traffic leaves the VM fn provides | 20:19 |
| mordred | infra-root: ok - opendevadmin account in github created, 2fa enabled, info in passwords file - it also now is an owner of opendevorg and openstack-infra on github | 20:19 |
| zbr | corvus: if we can merge https://review.opendev.org/#/c/652708/ it would be great because I want to propose another simplification which needs that patch first. | 20:19 |
| mriedem | having said that, it's mostly hitting on fortnebula nodes | 20:19 |
| mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22%2B%20%2Fopt%2Fstack%2Fnew%2Fgrenade%2Fprojects%2F70_cinder%2Fresources.sh%3Acreate%5C%22%20AND%20message%3A%5C%22SSH%20to%20the%20client%20did%20not%20work%2C%20something%20very%20wrong'%5C%22%20tags%3A%5C%22console%5C%22&from=7d | 20:19 |
| donnyd | ok makes sense clarkb | 20:19 |
| clarkb | mriedem: could be a timing thing maybe? | 20:19 |
| mriedem | that's my guess, | 20:19 |
| mriedem | nodes are slower or faster than others, something like that | 20:19 |
| clarkb | ya | 20:20 |
| fungi | possibly faster, yes | 20:20 |
| fungi | any time we add another provider we tend to shake out new timing and race issues in software | 20:20 |
| fungi | changes in cpu vs disk performance or processor count changing parallelization or... | 20:21 |
| donnyd | mriedem: almost 50% of the time | 20:22 |
| corvus | zbr: i'm really tired of dealing with linters (especially ansible-lint); i'm +0 on that. i would be +2 on removal. | 20:22 |
| donnyd | but anything prior to today is likely from storage contention | 20:22 |
| zbr | i guess most of frustration comes from using them wrongly (and thus causing issues). | 20:23 |
| donnyd | That was why everything was refactored to hopefully eliminate it as an issue | 20:23 |
| zbr | by wrongly i mean in a way that is hard to run, maintain, slow or unstable. i was planning to make a POC that gets rid of most of this trouble (and removes a big number of lines of boilerplate code) | 20:26 |
| fungi | zbr: i'm unsure what the right way is to use a style checker if you mostly only care that things remain readable and parse correctly and aren't particularly driven by a need for complete consistency of style | 20:26 |
| *** jrist has joined #openstack-infra | 20:27 | |
| fungi | i'm reminded by guido's comment about "consistency" in pep 8 | 20:28 |
| *** jrist has quit IRC | 20:28 | |
| *** eernst_ has joined #openstack-infra | 20:28 | |
| fungi | or more particularly "needless consistency" | 20:28 |
| donnyd | node_provider:"fortnebula-regionone" AND filename:job-output.txt AND message:"Upload logs to log server" AND build_name:"grenade-py3" shows that for the most part the build are successful | 20:29 |
| zbr | glad you opened the subject: time to mention black | 20:29 |
| fungi | sorry, "foolish consistency" which i guess was actually a ralph waldo emerson quote he included for balance | 20:29 |
| *** jrist has joined #openstack-infra | 20:30 | |
| *** jrist has quit IRC | 20:30 | |
| zbr | i seen it adopted in few project and I was very glad: no more wasted time with discussions when it makes decisions for you, so you can focus on the meat | 20:30 |
| mordred | well - I can say I like the idea of putting the exclusions into a config file instead of just inline in tox - it seems like a no-op in terms of our jobs and makes things a little simpler, so without getting in to my feelings on ansible-lint's choices, I think this patch improves the state of our current decision to run the tool | 20:31 |
| corvus | zbr: look, i appreciate that you're trying to help, but this is not the right approach. black is a distraction here. it's a python linter, not ansible. so let's please not talk about it in this context. | 20:31 |
| zbr | in fact it may be possible to put the config inside setup.cfg, but not tox.ini - because tox.ini is tox specific and is not officially recognized. (in case the issue was about "adding new file in root") | 20:32 |
| *** jrist has joined #openstack-infra | 20:32 | |
| corvus | zbr: i personally think that ansible-lint is a waste of time because it catches very few actual errors, and mostly reports things that either don't matter to us in the context we use it, or are just plain wrong. | 20:32 |
| mordred | that said - I think ansible-lint has made some questionable choices recently and has been more behaivng like pylint than like pyflakes | 20:32 |
| fungi | also, software imposing opinions on coding style of any sort is mostly useful in communities which are likely to have endless discussions about irrelevant style choices and "foolish consistency" | 20:32 |
| mordred | yeah. things like "you should use the git modules instead of shell: git" | 20:32 |
| corvus | zbr: so i don't think it's time well spent for me to review patches which move things around with ansible lint. i don't think it's time well spent for you to write them either. because at any point, it might go a way entirely. | 20:33 |
| fungi | communities which aren't so concerned about style are likely to spend more time fighting style checkers than they benefit from the opinions those checkers impose | 20:33 |
| corvus | zbr: so i'm going going to +2 that patch, but if you find other folks who are happy to +2 it, great. just please be aware of that. | 20:33 |
| zbr | mordred: true ansible-lint made quite a few mistakes in the past, but pinning it should keep us safe, and letting other deal with bleading edge issues. | 20:33 |
| corvus | zbr: and please be aware of the context in which you are asking these things. i'm trying to be as up front as possible here and save everyone from wasting time | 20:33 |
| zbr | thanks! | 20:34 |
| zbr | this should be enough to allow me to make the POC | 20:34 |
| corvus | unfortunately, that has backfired, as i have now spent far more time talking about that than i should have. for what it's worth, that has only reinforced my idea that ansible-lint is not worth the time (as even attempting to ignore it is now wasting time) | 20:35 |
| *** eharney has joined #openstack-infra | 20:35 | |
| clarkb | zbr: note pbr's setup.cfg is not compatible with setuptools. (this is unfortunate but not something we have any control over given pbr's was there first and based on the upstream pep) | 20:35 |
| fungi | though i and others have made some attempts to get options which make sense for both setuptools and pbr to be consistently-named | 20:36 |
| zbr | i guess pbr is going to give use quite a good number of headaches related to pep 517 and the project.toml, right? | 20:36 |
| mordred | hopefully not any time in the near future, no | 20:37 |
| fungi | zbr: i don't think so, unless you try to use both | 20:37 |
| fungi | and even then it'll probably be fine as long as you keep setup.cfg around for pbr options | 20:37 |
| fungi | zbr: also i thought the file name was pyproject.toml, did it change? | 20:38 |
| mordred | yeah. I have tried to think about pbr + pep517 a few times, and I've mostly given up | 20:38 |
| zbr | i guess at some point it will happen, but hopefully after py27 last ceremony ;) | 20:39 |
| mordred | it should be called pyproject.misogynist.vcfunded.cantbebotheredtoreadexistingspecs.narcicism | 20:39 |
| mordred | but, you know, who cares | 20:39 |
| corvus | my values are incompatible with toml | 20:40 |
| zbr | i am glad that 517 happens, but not necessarily how it happens, pbr used to much simpler (toml being only some part of the problem) | 20:40 |
| mordred | maybe systemd will migrate to it and I can get double-duty on my latent industry annoyances | 20:41 |
| fungi | latent? ;) | 20:41 |
| mordred | fungi: sssh | 20:41 |
| *** jtomasek has quit IRC | 20:41 | |
| corvus | i have no doubt we're only seeing the *latent* annoyances | 20:42 |
| corvus | the active ones would clear the room | 20:42 |
| mordred | corvus: I haven't cleared the room yet? | 20:42 |
| * mordred is really losing his touch | 20:42 | |
| *** joeguo has joined #openstack-infra | 20:42 | |
| mordred | zbr: I think the concepts behind 517 are quite nice actually | 20:43 |
| clarkb | just wait for someone to tell mordred that rust uses toml for its packaging too | 20:43 |
| mordred | clarkb: literally the only reason I havent' starter rewriting everything we do in rust is that fact | 20:43 |
| corvus | it's one of the biggest barriers to adoption of rust for me | 20:43 |
| mordred | yu[ | 20:43 |
| mordred | yup | 20:43 |
| mordred | same here | 20:43 |
| openstackgerrit | Merged opendev/base-jobs master: Use standard ansible-lint config file https://review.opendev.org/652708 | 20:44 |
| *** trident has quit IRC | 20:44 | |
| *** mriedem has quit IRC | 20:44 | |
| *** mriedem has joined #openstack-infra | 20:45 | |
| zbr | mordred: yep, good idea. still reality is a bit different. for example in molecule we adopted 517, and in order to keep compatibility with some older setuptools, here is how we have in setup.py: https://github.com/ansible/molecule/blob/master/setup.py | 20:45 |
| zbr | now lets remember how the pbr version was looking, all this code is backward-compatibility-boilerplate. | 20:46 |
| mordred | zbr: that's revolting | 20:46 |
| *** trident has joined #openstack-infra | 20:46 | |
| zbr | it is. i did not write that. | 20:47 |
| zbr | i only hope that at some moment that file will vanish. | 20:47 |
| mordred | zbr: well, I mean - I've written terrible code before - the need for that to exist is revolting | 20:47 |
| clarkb | it is unfortunate that the commit that replaced pbr doesn't say why | 20:48 |
| mordred | which is the whole problem with setup.py existing in the first place - which is why we wrote pbr in the first place ... which has been working for us for what, 8 years? oh well | 20:48 |
| zbr | i am so happy that the perl code i wrote ~15 years ago was not open source | 20:48 |
| mordred | zbr: +1000 | 20:48 |
| corvus | clarkb: this commit? https://github.com/ansible/molecule/commit/5314ebd1ae6e5413137eef8d3f7be505e0c847f1#diff-2eeaed663bd0d25b7e608891384b7298 | 20:49 |
| corvus | clarkb: it clearly explains it: "package emoji" | 20:49 |
| clarkb | corvus: ya that one | 20:50 |
| mordred | zbr: I once wrote a perl script that used typeglobs to generate perl that interacted with a database. I'm not sure I'd be employable if anyone could see that :) | 20:50 |
| clarkb | mordred: ya pbr seems to have done extremely well. And setuptools finally supporting setup.cfg seems like others agree (just that we of course did it wrongly) | 20:50 |
| zbr | that was the moment when I desired to write new feature in the linter for forbidding emoji in commit messages. | 20:50 |
| *** goldyfruit has quit IRC | 20:50 | |
| *** goldyfruit has joined #openstack-infra | 20:51 | |
| corvus | zbr: now that's an argument to redeem linters if i've ever heard one :) | 20:51 |
| mordred | clarkb: yah. it's almost like we knew something about packaging things and build systems. too bad we made the mistake of not being oen of the cool kids | 20:51 |
| mordred | corvus: ++ | 20:51 |
| mordred | corvus: you know - we could write a gerrit plugin to reject commits that had emjoi in the commit message ... | 20:51 |
| clarkb | freenode just let me know they are changing the +z channel mode | 20:52 |
| mordred | just with the message "nope" | 20:52 |
| clarkb | apparently clients have to support it, but we aren't +z so should be fine | 20:52 |
| clarkb | (clients that ops use) | 20:52 |
| mordred | yeah | 20:53 |
| openstackgerrit | Merged opendev/system-config master: Add LE cert for logs.opendev.org to static https://review.opendev.org/673903 | 20:54 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Add logs.opendev.org vhost https://review.opendev.org/673912 | 20:54 |
| *** mriedem has quit IRC | 20:54 | |
| corvus | fungi, clarkb, mordred: ^ okay i think that's the vhost. ugh. | 20:55 |
| corvus | i will be very happy when we switch to swift and all that goes away | 20:55 |
| *** mriedem has joined #openstack-infra | 20:55 | |
| corvus | oops i see probs | 20:56 |
| clarkb | mordred: reading pep517 the api they describe seems pretty compatible with pbr. I mean we'll likely have to hook things in but for example build_sdist is a thing pbr knows how to do | 20:56 |
| clarkb | I guess the struggle is if setuptools itself goes away | 20:56 |
| clarkb | (which pbr uses to do that work) | 20:56 |
| mordred | well - I think that would actually make things easier | 20:56 |
| mordred | because setuptools is a bitch to work with | 20:56 |
| clarkb | ya but potentially means more effort to get working and simple hook shimming won't be enough | 20:57 |
| mordred | clarkb: I agree - I think it's actually going to be largely compatible | 20:57 |
| clarkb | but I don't know what the long term plans are for setuptools | 20:57 |
| clarkb | (like will it be a supported build backend?) | 20:57 |
| clarkb | ianw: when your day starts I was hoping to talk about the devstack, osc, sdk, api performance stuff. We can move over to -qa too if you want | 20:58 |
| fungi | clarkb: i wonder if +z would be useful to ops in #openstack-unregistered | 20:58 |
| *** bobh has joined #openstack-infra | 21:00 | |
| clarkb | fungi: in conjunction with +m ? | 21:00 |
| fungi | yeah | 21:00 |
| fungi | since we silence everyone who joins that channel (or at least i think we do) | 21:00 |
| clarkb | I don't think we do | 21:02 |
| fungi | ahh, maybe we didn't because we didn't know of a way for those folks to get help when forwarded into it | 21:04 |
| fungi | i guess if it gets hit hard by spam then we can look into that option | 21:04 |
| clarkb | fungi: corvus my review on the vhost change hopefully captures what we want to do to address fungi's concern? let me know if that makes sense | 21:05 |
| *** bobh has quit IRC | 21:05 | |
| *** kjackal has quit IRC | 21:05 | |
| *** fdegir has quit IRC | 21:07 | |
| fungi | it would be close, if we decide what we want is to redirect http to https, but the redirect rules you're seeing are for redirecting non-canonical hostnames | 21:08 |
| fungi | (serveraliases) | 21:08 |
| clarkb | we would drop the rewritecond | 21:09 |
| clarkb | (thank you for pointing that out) | 21:09 |
| fungi | honestly i would just copy/paste all the osla stuff from the https vhost into the http vhost, or maybe we decide that osla isn't necessary on logs.opendev.org? | 21:16 |
| fungi | in which case drop it from both | 21:17 |
| clarkb | fungi: because you want http to work too? | 21:17 |
| corvus | oh i figured we could have http -> https? | 21:17 |
| clarkb | corvus: ya that is what my comment describes | 21:17 |
| corvus | for the new vhost at least | 21:17 |
| clarkb | I missed the bit about dropping the rewrite condition | 21:17 |
| fungi | i don't necessarily want http to work, but when mordred brought it up before it sounded like there was concern for it being more complexity | 21:17 |
| clarkb | but otherwise that was my intent | 21:17 |
| corvus | i think for the new vhost, this is going to be simpler | 21:18 |
| corvus | changing the current one is complex | 21:18 |
| fungi | wfm, then what clarkb suggests makes more sense | 21:18 |
| corvus | i'm also going through and hardcoding the stuff that currently comes from openstackci::logserver | 21:18 |
| fungi | though we could also consider just not using osla in the new vhosts if zuul's logs browser will provide roughly equivalent functionality | 21:18 |
| corvus | i think we need to swith to this before we can switch zuul | 21:19 |
| fungi | ahh, okay | 21:19 |
| corvus | so we need to have zuul returning logs.opendev.org as log_url before we can have zuul use the new url in the report | 21:19 |
| fungi | makes sense, and that's global(ish) yeah | 21:19 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Add logs.opendev.org vhost https://review.opendev.org/673912 | 21:21 |
| corvus | okay let's see if that's better | 21:21 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Always record the log_url in sql reporter https://review.opendev.org/673858 | 21:24 |
| clarkb | I bet tempest-dsvm doesn't really match any of our modern jobs either (just noticing that in your update) | 21:24 |
| clarkb | maybe the legacy jobs | 21:24 |
| clarkb | the change is approved | 21:25 |
| fungi | yeah, there are likely still legacy job names matching that pattern | 21:25 |
| corvus | i think i need one more change there | 21:26 |
| corvus | i copied the certs path from the tarballs.opendev.org vhost, but i don't see the logs certs at the same path | 21:27 |
| corvus | ie, logs.opendev.org is in /etc/letsencrypt-certs but not in /etc/ssl/certs | 21:27 |
| corvus | somehow, on files.o.o the certs for docs.opendev.org do end up in /etc/ssl/certs | 21:27 |
| *** kjackal has joined #openstack-infra | 21:27 | |
| fungi | huh, symlink? looking | 21:27 |
| clarkb | I'll remove my approval | 21:28 |
| corvus | oh | 21:28 |
| fungi | i see them in /etc/letsencrypt-certs/tarballs.opendev.org/ | 21:29 |
| corvus | did we put that in hiera? | 21:29 |
| fungi | oh | 21:29 |
| fungi | docs.opendev.org though | 21:29 |
| corvus | oh yeah, which should i be looking at, docs or tarballs? | 21:29 |
| corvus | tarballs! | 21:29 |
| corvus | yeah, docs is in hiera, so forget about it. tarballs is LE | 21:30 |
| clarkb | ah | 21:30 |
| corvus | and they are only in the LE path | 21:31 |
| corvus | so i copypastad the wrong path there | 21:31 |
| corvus | at least that makes sense :) | 21:31 |
| clarkb | that makes sense re not making extra copies of that data | 21:31 |
| fungi | yep, in hiera | 21:31 |
| fungi | for docs.opendev | 21:31 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Add logs.opendev.org vhost https://review.opendev.org/673912 | 21:32 |
| clarkb | corvus: is it .cer or .cert ? | 21:32 |
| corvus | clarkb, fungi: ^ i verified those 3 files exist on the host now | 21:32 |
| corvus | it's .cer | 21:32 |
| clarkb | +2 | 21:32 |
| *** tdasilva has quit IRC | 21:33 | |
| fungi | i also confirmed the files exist on the server already | 21:34 |
| fungi | and approved | 21:34 |
| fungi | and yeah, .cer instead of .crt through me for a moment | 21:34 |
| fungi | s/through/threw/ | 21:34 |
| fungi | s/.*/tarballs!/ | 21:35 |
| zbr | in case someone got bored: https://github.com/yaml/pyyaml/issues/325 | 21:35 |
| *** e0ne has joined #openstack-infra | 21:37 | |
| *** kjackal has quit IRC | 21:38 | |
| *** goldyfruit has quit IRC | 21:40 | |
| *** takamatsu has joined #openstack-infra | 21:40 | |
| clarkb | mriedem: I've been sifting through those neutron grenade logs and understand the test a bit better. Grenade is creating a server and a volume then attaching the volume to the server. Once the volume shows in use it is doing a ping check and when that succeeds it is doing an ssh check | 21:49 |
| clarkb | mriedem: I think it reasonably likely that the 30 second timeout isn't long enough for ssh, however I'm not sure how quick cirros is these days | 21:49 |
| clarkb | connection refused tells me that there isn't something listening on port 22 yet | 21:50 |
| fungi | or something caused the sshd to die | 21:53 |
| clarkb | oh also it is boot from volume | 21:53 |
| clarkb | as far as I can tell the network bits are working | 21:54 |
| clarkb | (due to the ping and neutron logs) | 21:54 |
| *** kjackal has joined #openstack-infra | 21:54 | |
| fungi | could it be delay generating sshd's host key(s)? | 21:54 |
| clarkb | that could be, though cirros uses dropbear and not sure how it handles host keys | 21:55 |
| clarkb | I think what we want to do is have grenade dump the console log for that server when it times out | 21:55 |
| fungi | yeah, no clue either | 21:55 |
| clarkb | I'll write that change now | 21:55 |
| *** pkopec has quit IRC | 21:59 | |
| clarkb | fungi: mriedem donnyd remote: https://review.opendev.org/673923 Capture console logs from failed server create | 22:00 |
| donnyd | I will give you my +1 clarkb | 22:01 |
| donnyd | probably won't get you anywhere, but I am happy to do it | 22:01 |
| clarkb | donnyd: if we are luck the change will have failed tests premerge and we can just look at those logs. I'll probably recheck a few times in hopes of catching one | 22:03 |
| clarkb | oh wait I think I put it in the wrong spot | 22:04 |
| mriedem | yup | 22:04 |
| mriedem | i just crushed your gd soul | 22:04 |
| *** slaweq has quit IRC | 22:04 | |
| clarkb | mriedem: ya it was wrong spot | 22:05 |
| *** rcernin has joined #openstack-infra | 22:05 | |
| mriedem | +7 | 22:05 |
| cmurphy | does logstash index logs for unit test jobs? i'm trying to build a query for it but looking at the main dashboard i don't see any non-tempest jobs from the last few hours | 22:05 |
| mriedem | cmurphy: yes, they'd be in the console logs | 22:06 |
| mriedem | hard to be useful though since they are not multi-line | 22:06 |
| mriedem | so you get MismatchError on one line and some relevant part of the trace on another and you can't group those in a query | 22:06 |
| mriedem | cmurphy: link me to the line you're trying to query | 22:07 |
| cmurphy | mriedem: the search i want is simpler than that, i think jobs like http://logs.openstack.org/76/673476/1/gate/openstack-tox-lower-constraints/d31bc96/ are failing often just because we're generating too much log output | 22:08 |
| cmurphy | i just want to see how many failed in the last few minutes/hours/days | 22:08 |
| fungi | we also have graphite data for that | 22:09 |
| clarkb | cmurphy: my desktop stopped responding after opening that logfile | 22:09 |
| cmurphy | clarkb: yeah | 22:09 |
| clarkb | how big is it overall? it may not be indexed becauseit is crashing the processors if big enough to oom them | 22:09 |
| clarkb | theseessages come to you via my phone becausecomputer is quite sad | 22:10 |
| cmurphy | http://logs.openstack.org/76/673476/1/gate/openstack-tox-lower-constraints/d31bc96/ says 12M | 22:10 |
| mriedem | cmurphy: yeah nova/cinder have had that issue, usually shows up as a subunit parser failure | 22:11 |
| mriedem | lots of warnings and tracebacks in test runs for negative tests and stuff | 22:11 |
| mriedem | http://status.openstack.org/elastic-recheck/#1813147 | 22:11 |
| clarkb | compressed so maybe 120mb uncompressed? | 22:11 |
| smcginnis | cmurphy: stestr has a latent bug somewhere deep down (or one if its libraries) that can't handle unicode streams when the tests end up outputting a largish amount of text. | 22:12 |
| smcginnis | The only way I've seen to get around it is to cut back on what gets emitted during a test run. | 22:12 |
| cmurphy | if you look at the testr_results it's all green | 22:12 |
| clarkb | smcginnis: I think it goes back all the way to subunit v1 | 22:12 |
| mriedem | holy god those policy warnings | 22:12 |
| mriedem | you need to filter warnings in the test runs | 22:13 |
| cmurphy | yeah | 22:13 |
| clarkb | there was a max packet size and the way we attach logging to the packets runs up against that | 22:13 |
| smcginnis | clarkb: That could be. Matt and I spent quite awhile trying to trace things to the source, but neither of us made it very far. | 22:13 |
| mriedem | https://github.com/openstack/nova/blob/master/nova/tests/fixtures.py#L812 | 22:13 |
| *** tosky has quit IRC | 22:13 | |
| smcginnis | Only happens with py3 for some reason. Py2 is just peachy. | 22:13 |
| mriedem | nova has add warnings filters as we hit them | 22:13 |
| clarkb | smcginnis: well python3 strings are 4x the size | 22:14 |
| mriedem | including duh duh duh "message="Policy .* failed scope check"," | 22:14 |
| clarkb | now do I alt sysrq b to get back? | 22:14 |
| mriedem | cmurphy: also if it helps and keystone doesn't do it yet, we have a logging fixture that ignores debug level logs in test runs https://github.com/openstack/nova/blob/master/nova/tests/fixtures.py#L121 | 22:15 |
| mriedem | but still validate for formatting errors in debug messages https://github.com/openstack/nova/blob/master/nova/tests/fixtures.py#L104 | 22:16 |
| cmurphy | mriedem: in this case these are actual warnings not debug logs, we need to fix it to be less noisy in general because it's hurting operators too | 22:17 |
| cmurphy | but i wanted to get some data to prove it was hurting our dev velocity too | 22:17 |
| mriedem | ok but my point is in a unit test run, you don't need to log these oslo policy deprecation warnings more than once per run | 22:17 |
| mriedem | b/c that's destroying the console output that gets pushed through logstash | 22:17 |
| cmurphy | yes agreed | 22:18 |
| fungi | clarkb: i usually alt+sysrq+s first followed by alt+sysrq+u and only *then* alt+sysrq+b (with a few seconds delay between each) | 22:18 |
| *** rlandy is now known as rlandy|bbl | 22:19 | |
| openstackgerrit | Merged opendev/system-config master: Add logs.opendev.org vhost https://review.opendev.org/673912 | 22:22 |
| clarkb | fungi: I wasn't too worried about losing data as I didn't have anything writing at the time | 22:26 |
| clarkb | but did end up needing to force reboot via sysrq command | 22:27 |
| clarkb | apparently f will call oomkiller | 22:27 |
| clarkb | which might've saved me all the trouble if it killed firefox | 22:27 |
| clarkb | cmurphy: but ya when logs get that large we have trouble idnexing them | 22:27 |
| fungi | i usually just do s,u,b out of reflex, and to hopefully save myself fsck and/or fs corruption | 22:28 |
| clarkb | fungi: Oh I'm running btrfs | 22:28 |
| clarkb | so there is no hope either way :P | 22:29 |
| fungi | hah | 22:29 |
| cmurphy | clarkb: in general though, i'm not seeing anything like 'openstack-tox-py37' in http://logstash.openstack.org/#/dashboard/file/logstash.json so i'm not sure how to experiment with querying | 22:29 |
| cmurphy | looking in the build_name field | 22:29 |
| fungi | build_name:openstack-tox-py37 AND build_result:FAILED AND ... | 22:29 |
| cmurphy | okay that shows some things | 22:30 |
| clarkb | my wget of that file just corssed 200MB | 22:30 |
| mriedem | AND project:openstack/keystone | 22:30 |
| clarkb | and ended at 220MB | 22:30 |
| *** kjackal has quit IRC | 22:31 | |
| clarkb | that is a really good compression ratio | 22:31 |
| fungi | er, build_status not build_result | 22:31 |
| *** goldyfruit has joined #openstack-infra | 22:31 | |
| cmurphy | clarkb: when it's all the same log message repeated... | 22:31 |
| donnyd | uggg... node_provider:"fortnebula-regionone" AND filename:job-output.txt AND message:"RUN END RESULT_TIMED_OUT" | 22:34 |
| donnyd | I may be oversubscribing too much, but this is pretty much what was happening before | 22:34 |
| fungi | ahh and it's FAILURE not FAILED too | 22:35 |
| donnyd | :) | 22:35 |
| donnyd | LOL fungi | 22:35 |
| *** goldyfruit has quit IRC | 22:35 | |
| clarkb | cmurphy: there is a way to make warnings emit only once iirc | 22:36 |
| donnyd | I will scale it back down to 40 and see if that fixes the issue, but i will have to wait for nodepool to slow down later tonight | 22:36 |
| clarkb | since the tests run largely in the same processes that may work to quiet things down | 22:36 |
| clarkb | warnings.simplefilter("once") ? | 22:37 |
| mriedem | that's what we have in the nova warnings filter, just copy that code | 22:40 |
| mriedem | https://github.com/openstack/nova/blob/master/nova/tests/fixtures.py#L812 | 22:40 |
| donnyd | foods.... be back | 22:41 |
| *** mriedem has quit IRC | 22:43 | |
| ianw | donnyd / clarkb: going back a bit, chatting with others over the kafs stuff with our working set we may not see much difference over just using "regular" dcache anyway, the disk cache might be overhead | 22:47 |
| clarkb | ianw: with openafs you mean? | 22:47 |
| clarkb | (there was a clear difference when I made consecutive requests fwiw) | 22:47 |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Switch to logs.opendev.org https://review.opendev.org/673925 | 22:49 |
| *** gagehugo has quit IRC | 22:49 | |
| openstackgerrit | Merged zuul/zuul-operator master: use opendev image building system for zuul-operator test https://review.opendev.org/673020 | 22:50 |
| *** gagehugo has joined #openstack-infra | 22:50 | |
| *** dchen has joined #openstack-infra | 22:51 | |
| ianw | clarkb: probably with kafs and openafs; both should keep pages just in dcache anyway as i understand it. so there's overhead getting them to disk and off disk, which if they don't get used enough to amortize before they change again, can be counterproductive | 22:51 |
| clarkb | ianw: I think the problem with openafs is taht it has the fixed window size for its udp packets | 22:51 |
| clarkb | ianw: and so throughput falls out significantly as rtt increases | 22:51 |
| ianw | yeah, i think we'd need much more detailed monitoring to understand exactly what fiddling parameters do, and as you say very heterogeneous mirror nodes, between possibly connected with fibre to tin-cans across the atlantic | 22:53 |
| *** aedc has quit IRC | 22:54 | |
| *** aedc has joined #openstack-infra | 22:56 | |
| *** tkajinam has joined #openstack-infra | 22:57 | |
| corvus | https://logs.opendev.org/ exists | 23:00 |
| corvus | https://review.opendev.org/673925 should be gtg whenever we're ready | 23:00 |
| *** gagehugo has quit IRC | 23:01 | |
| fungi | indeed, and the http->https redirect seems fine | 23:02 |
| donnyd | clarkb: I was talking about the timeouts | 23:02 |
| *** panda has quit IRC | 23:03 | |
| fungi | testing a docs draft render and a job log htmlification and an ara report, all seem to work fine on the new vhost | 23:04 |
| *** panda has joined #openstack-infra | 23:04 | |
| donnyd | https://usercontent.irccloud-cdn.com/file/DBo9iloX/Screenshot%20from%202019-07-31%2019-04-15.png | 23:04 |
| *** gagehugo has joined #openstack-infra | 23:04 | |
| donnyd | There are a few spikes, but for the most part cpu utilization is what i would expect | 23:04 |
| donnyd | 50-70% | 23:05 |
| *** notmyname has quit IRC | 23:05 | |
| corvus | fungi: dinner tonight is at an okonomiyaki popup at a brewery down the street | 23:08 |
| fungi | corvus: wow! | 23:08 |
| fungi | enjoy, sounds amazing | 23:08 |
| fungi | my home okonomiyaki is improving, but still has a long way to go | 23:08 |
| corvus | they apparently make hiroshima- and osaka- styles | 23:09 |
| corvus | which is a distinction that is new to me. we should order one of each :) | 23:09 |
| corvus | (layered vs mixed, apparently) | 23:10 |
| *** slaweq has joined #openstack-infra | 23:11 | |
| *** slaweq has quit IRC | 23:15 | |
| clarkb | logs change is approved | 23:15 |
| clarkb | I've got carnitas in the slow cooker | 23:17 |
| *** jamesmcarthur has joined #openstack-infra | 23:22 | |
| *** dchen has quit IRC | 23:22 | |
| openstackgerrit | Merged opendev/system-config master: Correct emergency file reference in launch script https://review.opendev.org/672996 | 23:22 |
| clarkb | ianw: for the osc/devstack stuff you mentioned you were trying to compare runtimes? | 23:28 |
| clarkb | ianw: what I've run into trying to apply what I did to create_keystone_accounts to the other create_*_accounts functions is that they rely a lot on environment state of the running stack.sh. Rather than encode all of that directly in python I was thinking that having stack.sh write out a large "do these things to keystone" manifests file that osc/sdk/something could read back in might be a good | 23:29 |
| clarkb | appraoch | 23:29 |
| clarkb | but curious if you had better ideas after looking at it | 23:29 |
| ianw | clarkb; yeah did you see the stats from https://ethercalc.openstack.org/rzuhevxz7793 ? | 23:31 |
| ianw | as a baseline | 23:31 |
| openstackgerrit | Merged opendev/base-jobs master: Switch to logs.opendev.org https://review.opendev.org/673925 | 23:31 |
| clarkb | I did not (was that over in -qa I'm apparently not in that channel currently) | 23:33 |
| *** e0ne has quit IRC | 23:33 | |
| ianw | clarkb: check out my comment in https://review.opendev.org/#/c/673018/ | 23:33 |
| clarkb | ianw: as frickler points out on https://review.opendev.org/#/c/673018/ that isn't much quicker, but that led me to https://review.opendev.org/#/c/673108/6 which is signficantly faster | 23:36 |
| clarkb | I think a good deal of the overhead is tied up in ~3 things. startup overhead, requesting new tokens every time osc is invoked, converting names to ids every time osc is invoked | 23:36 |
| clarkb | which is why replacing it all with a single script (one startup cost, one token, no name -> id conversions because all ids are remembered) is quick | 23:37 |
| clarkb | one approach we might take is to try and have osc use a preallocated token (we can do that iirc) and use ids for everything | 23:37 |
| clarkb | but I think I've mostly convinced myself if we want to go down this path batching up many operations as much as possible is the ideal | 23:38 |
| *** dchen has joined #openstack-infra | 23:38 | |
| ianw | yeah, i'd agree. i'd say there's no particular reason it was done like that, and it makes sense to pipeline it | 23:39 |
| openstackgerrit | Merged zuul/zuul master: Always record the log_url in sql reporter https://review.opendev.org/673858 | 23:40 |
| ianw | my only real concern was that people might have considered it as testing various things the way it was, but it sounds like not | 23:41 |
| ianw | (various things being osc interfaces, osc->services interfaces, etc) | 23:41 |
| clarkb | ya | 23:42 |
| *** betherly has joined #openstack-infra | 23:42 | |
| clarkb | mostly I see this as a very promising option for reducing devstack runtime and so wanted to see what other devstack cores had to think about it. Particularly since while the poc is straightforward making the transition properly at a wider scale becomes tricky | 23:43 |
| clarkb | (it really is awkward to ahve all that state in env vars then try and push that down into python script(s)) | 23:43 |
| *** eharney has quit IRC | 23:45 | |
| *** betherly has quit IRC | 23:47 | |
| ianw | so you're saying take like get_or_create_service() and turn it from an instant action to something that writes out a manifest, then process that later? | 23:48 |
| clarkb | ya | 23:49 |
| fungi | i feel like "can you set up devstack" is a mediocre, opaque and terribly complex "test" so whatever we can do to just make that efficient instead and then explicitly test the things we thought that might be exercising is probably better for everyone involved | 23:49 |
| clarkb | ianw: my message to the mailing list had an initial idea of turning this block https://opendev.org/openstack/devstack/src/branch/master/stack.sh#L1146-L1161 and turning that into one manifest invocation | 23:50 |
| *** sthussey has quit IRC | 23:50 | |
| clarkb | (and that block does things like call get_or_create_service) | 23:50 |
| clarkb | that way the logic of what needs to be created can stay in bash | 23:51 |
| clarkb | but the work of creating the resources happens ine one "long lived" python process using one token | 23:51 |
| *** notmyname has joined #openstack-infra | 23:52 | |
| *** notmyname has quit IRC | 23:53 | |
| clarkb | one big drawback to that approach is it makes it hard for other systems to reconsume it | 23:53 |
| *** notmyname has joined #openstack-infra | 23:53 | |
| ianw | hrm, you could intercept "get_or_create_<service|endpoint>" but i wonder if any plugins are written to assume that the moment that function is called, it's active | 23:57 |
| ianw | most seem to be written with that happening in post-config; then their services are started in "extra" | 23:58 |
| ianw | random example https://opendev.org/x/mogan/src/branch/master/devstack/plugin.sh#L27 | 23:58 |
| clarkb | and we also have a bunch of osc to do things like create all the neutron networks and routers and subnets | 23:59 |
| clarkb | but those are probably fine to be self contained manifests | 23:59 |
| *** takamatsu has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
