| *** d34dh0r53 has quit IRC | 00:16 | |
| *** cloudnull has quit IRC | 00:16 | |
| *** cloudnull has joined #openstack-infra | 00:18 | |
| *** cloudnull has quit IRC | 00:18 | |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Alphabetize ssldomains file https://review.opendev.org/701481 | 00:19 |
|---|---|---|
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Add insecure-ci-registry to ssldomains https://review.opendev.org/701482 | 00:19 |
| fungi | clarkb: i just got notified by namecheap about another cert which is expiring soon ^ | 00:20 |
| clarkb | thansk | 00:21 |
| *** cloudnull has joined #openstack-infra | 00:24 | |
| *** d34dh0r53 has joined #openstack-infra | 00:25 | |
| *** ahosam has quit IRC | 00:25 | |
| *** d34dh0r53 has quit IRC | 00:25 | |
| fungi | we likely overlooked it in previous audits due to not listening on 443/tcp | 00:26 |
| clarkb | could be | 00:27 |
| clarkb | down to 8k replication tasks now | 00:27 |
| *** cloudnull has quit IRC | 00:27 | |
| *** d34dh0r53 has joined #openstack-infra | 00:28 | |
| *** d34dh0r53 has quit IRC | 00:30 | |
| *** d34dh0r53 has joined #openstack-infra | 00:35 | |
| *** d34dh0r53 has quit IRC | 00:36 | |
| *** d34dh0r53 has joined #openstack-infra | 00:39 | |
| *** d34dh0r53 has quit IRC | 00:39 | |
| *** aedc has joined #openstack-infra | 00:40 | |
| *** d34dh0r53 has joined #openstack-infra | 00:43 | |
| *** d34dh0r53 has quit IRC | 00:44 | |
| *** aedc_ has joined #openstack-infra | 00:48 | |
| *** aedc has quit IRC | 00:48 | |
| mordred | clarkb, fungi: the amount of time a full replication takes makes me reluctant to want to do one necessarily on a gitea restart - but maybe we just have to - we don't do them that frequently anyway | 00:51 |
| fungi | or is there a way to "pause" gerrit replication and resume it after the gitea restarts conclude? i'm not aware of one, but it's possible i've overlooked it | 00:52 |
| *** d34dh0r53 has joined #openstack-infra | 00:52 | |
| *** d34dh0r53 has quit IRC | 00:53 | |
| fungi | or maybe if whatever the behavior is that causes gitea to silently discard pushes when that lock is present could be fixed to properly refuse/error | 00:53 |
| fungi | or maybe we can temoprarily remove the firewall rule which allows ssh access? | 00:53 |
| fungi | this probably becomes much simpler when multiple gitea servers are able to correctly share a common file backend | 00:54 |
| mordred | yah | 00:54 |
| mordred | at least - I hope it does | 00:54 |
| fungi | since we could probably use a load balancer to direct replication to whichever gitea server has the "write token" at any point in time, and then move that around during restarts | 00:55 |
| *** ricolin_ has joined #openstack-infra | 00:57 | |
| *** ricolin_ has quit IRC | 00:58 | |
| *** d34dh0r53 has joined #openstack-infra | 00:59 | |
| *** d34dh0r53 has quit IRC | 00:59 | |
| *** d34dh0r53 has joined #openstack-infra | 01:01 | |
| *** cloudnull has joined #openstack-infra | 01:02 | |
| *** d34dh0r53 has quit IRC | 01:04 | |
| *** cloudnull has quit IRC | 01:04 | |
| *** laurenceee has quit IRC | 01:08 | |
| mordred | fungi, clarkb: the changes to compression landed today right? | 01:12 |
| fungi | yes | 01:12 |
| mordred | I think there is an impact from that | 01:12 |
| mordred | https://zuul.opendev.org/t/openstack/build/b92ccc05d3b84cb69a6eb2483305b262/logs | 01:12 |
| mordred | click the raw link from testr_results.html.gz | 01:12 |
| fungi | except for the default change we've held back for the announcement window | 01:12 |
| fungi | woo! | 01:13 |
| fungi | yeah, that's not what people probably want to see when they click there | 01:13 |
| mordred | yeah | 01:13 |
| mordred | I think this might be one fo those complicated things | 01:13 |
| fungi | so we're serving compressed data to browsers now, at least for that file | 01:14 |
| fungi | and without a content encoding header to tell them to decompress it? | 01:14 |
| *** d34dh0r53 has joined #openstack-infra | 01:14 | |
| fungi | i guess the reason is that the job is pre-compressing that file | 01:16 |
| fungi | https://zuul.opendev.org/t/openstack/build/b92ccc05d3b84cb69a6eb2483305b262/log/testrepository.subunit.gz shows a similar issue | 01:16 |
| clarkb | mordred: fungi the change we are holding iff on until next week fixes that | 01:17 |
| fungi | d'oh! | 01:17 |
| mordred | gotcha | 01:17 |
| clarkb | but it changes behavuor in zuul-jobs | 01:17 |
| fungi | so i guess we live with it until the 14th? | 01:18 |
| *** liuyulong has quit IRC | 01:23 | |
| mordred | it's a good question - cause for now all of the testr results are pretty much busted | 01:26 |
| *** d34dh0r53 has joined #openstack-infra | 01:28 | |
| *** rfolcOUT has quit IRC | 01:28 | |
| *** d34dh0r53 has quit IRC | 01:28 | |
| clarkb | we could fork the role | 01:30 |
| mordred | clarkb: oh - if you're still around, I fixed your comments on https://review.opendev.org/#/c/691800/ | 01:31 |
| clarkb | or is there a var to not compress? if so set that | 01:31 |
| *** d34dh0r53 has joined #openstack-infra | 01:33 | |
| *** d34dh0r53 has quit IRC | 01:34 | |
| *** d34dh0r53 has joined #openstack-infra | 01:37 | |
| *** d34dh0r53 has quit IRC | 01:38 | |
| *** d34dh0r53 has joined #openstack-infra | 01:41 | |
| *** d34dh0r53 has quit IRC | 01:42 | |
| *** cloudnull has joined #openstack-infra | 01:47 | |
| *** cloudnull has quit IRC | 01:48 | |
| *** cloudnull has joined #openstack-infra | 01:50 | |
| *** cloudnull has quit IRC | 01:51 | |
| *** d34dh0r53 has joined #openstack-infra | 01:52 | |
| *** d34dh0r53 has quit IRC | 01:52 | |
| *** cloudnull has joined #openstack-infra | 01:54 | |
| *** cloudnull has quit IRC | 01:54 | |
| *** cloudnull has joined #openstack-infra | 01:56 | |
| *** d34dh0r53 has joined #openstack-infra | 01:56 | |
| *** cloudnull has quit IRC | 01:56 | |
| *** d34dh0r53 has quit IRC | 01:57 | |
| *** cloudnull has joined #openstack-infra | 01:57 | |
| *** cloudnull has quit IRC | 01:57 | |
| *** d34dh0r53 has joined #openstack-infra | 02:02 | |
| *** d34dh0r53 has quit IRC | 02:03 | |
| *** cloudnull has joined #openstack-infra | 02:04 | |
| *** cloudnull has quit IRC | 02:04 | |
| *** cloudnull has joined #openstack-infra | 02:04 | |
| *** cloudnull has quit IRC | 02:04 | |
| *** d34dh0r53 has joined #openstack-infra | 02:05 | |
| *** d34dh0r53 has quit IRC | 02:05 | |
| *** d34dh0r53 has joined #openstack-infra | 02:09 | |
| *** d34dh0r53 has quit IRC | 02:10 | |
| *** d34dh0r53 has joined #openstack-infra | 02:19 | |
| *** d34dh0r53 has quit IRC | 02:19 | |
| *** Lucas_Gray has quit IRC | 02:21 | |
| *** d34dh0r53 has joined #openstack-infra | 02:21 | |
| *** d34dh0r53 has quit IRC | 02:22 | |
| *** hamzy has joined #openstack-infra | 02:22 | |
| *** d34dh0r53 has joined #openstack-infra | 02:38 | |
| *** cloudnull has joined #openstack-infra | 02:38 | |
| *** d34dh0r53 has quit IRC | 02:39 | |
| *** cloudnull has quit IRC | 02:39 | |
| *** dSrinivas has joined #openstack-infra | 02:39 | |
| *** cloudnull has joined #openstack-infra | 02:39 | |
| *** cloudnull has quit IRC | 02:39 | |
| *** ykarel has joined #openstack-infra | 02:44 | |
| *** cloudnull has joined #openstack-infra | 02:44 | |
| *** d34dh0r53 has joined #openstack-infra | 02:45 | |
| *** d34dh0r53 has quit IRC | 02:45 | |
| *** cloudnull has quit IRC | 02:45 | |
| *** cloudnull has joined #openstack-infra | 02:45 | |
| *** cloudnull has quit IRC | 02:46 | |
| *** rfolcOUT has joined #openstack-infra | 02:48 | |
| *** cloudnull has joined #openstack-infra | 02:49 | |
| *** cloudnull has quit IRC | 02:49 | |
| *** ykarel has quit IRC | 02:50 | |
| *** cloudnull has joined #openstack-infra | 02:56 | |
| *** cloudnull has quit IRC | 02:56 | |
| *** cloudnull has joined #openstack-infra | 02:58 | |
| *** cloudnull has quit IRC | 02:58 | |
| *** cloudnull has joined #openstack-infra | 03:01 | |
| *** d34dh0r53 has joined #openstack-infra | 03:01 | |
| *** cloudnull has quit IRC | 03:01 | |
| *** d34dh0r53 has quit IRC | 03:01 | |
| *** d34dh0r53 has joined #openstack-infra | 03:09 | |
| *** d34dh0r53 has quit IRC | 03:09 | |
| *** rlandy|bbl is now known as rlandy | 03:10 | |
| *** cloudnull has joined #openstack-infra | 03:11 | |
| *** cloudnull has quit IRC | 03:11 | |
| *** rlandy has quit IRC | 03:16 | |
| *** cloudkiller has joined #openstack-infra | 03:18 | |
| *** d34dh0r53 has joined #openstack-infra | 03:19 | |
| *** d34dh0r53 has quit IRC | 03:19 | |
| *** cloudkiller has quit IRC | 03:19 | |
| *** cloudkiller has joined #openstack-infra | 03:21 | |
| *** cloudkiller has quit IRC | 03:21 | |
| *** d34dh0r53 has joined #openstack-infra | 03:28 | |
| *** d34dh0r53 has quit IRC | 03:28 | |
| *** ykarel has joined #openstack-infra | 03:30 | |
| *** psachin has joined #openstack-infra | 03:33 | |
| *** ykarel has quit IRC | 03:34 | |
| *** d34dh0r53 has joined #openstack-infra | 03:36 | |
| *** cloudkiller has joined #openstack-infra | 03:40 | |
| *** d34dh0r53 has joined #openstack-infra | 03:41 | |
| *** d34dh0r53 has quit IRC | 03:41 | |
| *** cloudkiller has quit IRC | 03:41 | |
| *** cloudkiller has joined #openstack-infra | 03:42 | |
| *** cloudkiller has quit IRC | 03:42 | |
| *** cloudkiller has joined #openstack-infra | 03:46 | |
| *** cloudkiller has quit IRC | 03:46 | |
| *** cloudkiller has joined #openstack-infra | 03:48 | |
| *** d34dh0r53 has joined #openstack-infra | 03:48 | |
| *** ykarel has joined #openstack-infra | 04:02 | |
| *** ykarel has quit IRC | 04:14 | |
| *** cloudkiller is now known as cloudnull | 04:39 | |
| *** d34dh0r53 has quit IRC | 04:40 | |
| *** cloudnull has quit IRC | 04:40 | |
| *** hwoarang has quit IRC | 04:43 | |
| *** d34dh0r53 has joined #openstack-infra | 04:43 | |
| *** d34dh0r53 has quit IRC | 04:44 | |
| *** cloudnull has joined #openstack-infra | 04:45 | |
| *** d34dh0r53 has joined #openstack-infra | 04:45 | |
| *** d34dh0r53 has quit IRC | 04:45 | |
| *** cloudnull has quit IRC | 04:45 | |
| *** d34dh0r53 has joined #openstack-infra | 04:47 | |
| *** d34dh0r53 has quit IRC | 04:48 | |
| *** cloudnull has joined #openstack-infra | 04:51 | |
| *** hwoarang has joined #openstack-infra | 04:51 | |
| *** ykarel has joined #openstack-infra | 04:51 | |
| *** cloudnull has joined #openstack-infra | 04:51 | |
| *** cloudnull has quit IRC | 04:51 | |
| *** d34dh0r53 has joined #openstack-infra | 04:53 | |
| *** d34dh0r53 has quit IRC | 04:53 | |
| *** cloudnull has joined #openstack-infra | 04:54 | |
| *** d34dh0r53 has joined #openstack-infra | 04:54 | |
| *** d34dh0r53 has quit IRC | 04:56 | |
| *** cloudnull has quit IRC | 04:56 | |
| *** gyee has quit IRC | 04:57 | |
| *** d34dh0r53 has joined #openstack-infra | 04:58 | |
| *** d34dh0r53 has joined #openstack-infra | 05:01 | |
| *** cloudnull has joined #openstack-infra | 05:02 | |
| *** udesale has joined #openstack-infra | 05:03 | |
| *** spsurya has joined #openstack-infra | 05:05 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-infra | 05:33 | |
| *** surpatil has joined #openstack-infra | 05:34 | |
| *** dave-mccowan has quit IRC | 05:36 | |
| *** lmiccini has joined #openstack-infra | 05:43 | |
| *** zxiiro has quit IRC | 06:27 | |
| *** SurajPatil has joined #openstack-infra | 06:28 | |
| *** surpatil has quit IRC | 06:31 | |
| *** dpawlik has joined #openstack-infra | 06:52 | |
| *** ykarel is now known as ykarel|lunch | 07:27 | |
| *** pgaxatte has joined #openstack-infra | 07:28 | |
| *** surpatil has joined #openstack-infra | 07:34 | |
| *** SurajPatil has quit IRC | 07:36 | |
| *** pcaruana has joined #openstack-infra | 07:38 | |
| *** ociuhandu has joined #openstack-infra | 07:46 | |
| *** jtomasek has joined #openstack-infra | 07:56 | |
| *** ociuhandu has quit IRC | 07:58 | |
| *** tosky has joined #openstack-infra | 08:05 | |
| *** tesseract has joined #openstack-infra | 08:06 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Adds support to control log fetching with variables https://review.opendev.org/701381 | 08:18 |
| *** dchen has quit IRC | 08:18 | |
| *** gfidente|afk is now known as gfidente | 08:23 | |
| *** ykarel|lunch is now known as ykarel | 08:25 | |
| *** SurajPatil has joined #openstack-infra | 08:27 | |
| *** lbragstad has quit IRC | 08:28 | |
| *** tkajinam has quit IRC | 08:29 | |
| *** surpatil has quit IRC | 08:30 | |
| *** apetrich has joined #openstack-infra | 08:31 | |
| *** tosky_ has joined #openstack-infra | 08:33 | |
| *** tosky has quit IRC | 08:34 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Control log archive and user preservation with vars https://review.opendev.org/701381 | 08:35 |
| *** aedc_ has quit IRC | 08:39 | |
| *** tosky_ is now known as tosky | 08:39 | |
| *** lbragstad has joined #openstack-infra | 08:40 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Retire js-openstack-lib (step 2) https://review.opendev.org/701509 | 08:42 |
| *** iurygregory has quit IRC | 08:42 | |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Retire js-openstack-lib: Remove jobs and templates https://review.opendev.org/701510 | 08:43 |
| *** jpena|off has joined #openstack-infra | 08:44 | |
| *** jpena|off is now known as jpena | 08:44 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Finish retiring js-openstack-lib https://review.opendev.org/701513 | 08:48 |
| *** surpatil has joined #openstack-infra | 08:50 | |
| *** ralonsoh has joined #openstack-infra | 08:52 | |
| *** SurajPatil has quit IRC | 08:52 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Finish retiring js-openstack-lib https://review.opendev.org/701513 | 08:56 |
| *** SurajPatil has joined #openstack-infra | 08:58 | |
| *** surpatil has quit IRC | 09:01 | |
| *** xek has joined #openstack-infra | 09:05 | |
| *** tetsuro has joined #openstack-infra | 09:06 | |
| *** diablo_rojo has joined #openstack-infra | 09:07 | |
| *** iurygregory has joined #openstack-infra | 09:08 | |
| *** lucasagomes has joined #openstack-infra | 09:17 | |
| *** rpittau|afk is now known as rpittau | 09:25 | |
| *** surpatil has joined #openstack-infra | 09:32 | |
| yoctozepto | morning | 09:33 |
| *** SurajPatil has quit IRC | 09:34 | |
| yoctozepto | clarkb: AJaeger sent me to you regarding keeping js-openstack-lib | 09:35 |
| yoctozepto | I have yet to check details (incl. governance) on it | 09:35 |
| yoctozepto | I wrote on the ML already but: I would be glad to pick this up because I planned some integrations like this on my own | 09:36 |
| yoctozepto | I thought it might be also wise to preserve this since there are posts now and then that horizon is reaching its limit and a JS lib might be beneficial for any possible replacement (as it can run from the browser) | 09:37 |
| AJaeger | yoctozepto: there're two things IMHO: Keep it alive - and keep it as part of infra. I don't see it as infra deliverable. I suggest you check whether it's an option to keep it alive and then governance is the second part. | 09:39 |
| AJaeger | yoctozepto: note that clarkb is US east coast based, so will be around in 6 to 7 hours only. | 09:40 |
| yoctozepto | AJaeger: yeah, hence I left a whole monologue :-) | 09:40 |
| AJaeger | :) | 09:40 |
| yoctozepto | AJaeger: I think you are right this is not infra thingy | 09:40 |
| yoctozepto | I added todo note to test this if this is usable | 09:41 |
| *** asettle has joined #openstack-infra | 09:42 | |
| *** derekh has joined #openstack-infra | 09:42 | |
| asettle | o/ | 09:42 |
| AJaeger | morning asettle ! Happy new year! | 09:44 |
| asettle | Thank you AJaeger :D | 09:44 |
| asettle | How was your break? | 09:44 |
| AJaeger | excellent! | 09:45 |
| AJaeger | and yours? | 09:45 |
| *** ociuhandu has joined #openstack-infra | 10:02 | |
| *** dpawlik has quit IRC | 10:07 | |
| *** dSrinivas has quit IRC | 10:07 | |
| *** dpawlik has joined #openstack-infra | 10:14 | |
| *** pkopec has joined #openstack-infra | 10:14 | |
| *** ociuhandu has quit IRC | 10:19 | |
| sean-k-mooney | AJaeger: thanks for reviewing the dpdk testing patches. is there a similar template to build but not publish docs to the build-release-notes-jobs-python3 template | 10:47 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/project-config master: New project "Ideas" https://review.opendev.org/701530 | 10:47 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/project-config master: New project "Ideas" https://review.opendev.org/701530 | 10:48 |
| *** dtantsur|afk is now known as dtantsur | 10:50 | |
| *** sshnaidm|afk is now known as sshnaidm | 10:57 | |
| AJaeger | sean-k-mooney: https://docs.openstack.org/infra/openstack-zuul-jobs/project-templates.html#project_template-build-openstack-docs-pti | 10:58 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/project-config master: New project "Ideas" https://review.opendev.org/701530 | 10:58 |
| evrardjp | morning AJaeger :) | 10:59 |
| evrardjp | and others | 10:59 |
| *** ociuhandu has joined #openstack-infra | 10:59 | |
| sean-k-mooney | AJaeger: great thanks ill add that when i remove the noop job | 11:00 |
| evrardjp | Frohes Neujahr | 11:00 |
| AJaeger | Frohes Neues, evrardjp ! | 11:00 |
| evrardjp | though I guess we are not the first January, so I am not sure Neujahr applies ? German 101 here | 11:01 |
| AJaeger | evrardjp: it still applies... | 11:02 |
| evrardjp | woot! :) | 11:02 |
| *** portdirect has quit IRC | 11:06 | |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Ensure correct cleanup on repo update and reset https://review.opendev.org/701531 | 11:09 |
| *** ociuhandu has quit IRC | 11:18 | |
| *** ramishra has quit IRC | 11:24 | |
| *** ramishra has joined #openstack-infra | 11:24 | |
| *** ociuhandu has joined #openstack-infra | 11:29 | |
| sean-k-mooney | AJaeger: possibly dumb question but where does zuul clone the requireed projects? is it in the zuul users home directory or to /opt/stack? | 11:29 |
| sean-k-mooney | oh is it in /opt/git https://zuul-ci.org/docs/zuul-jobs/general-roles.html#role-prepare-workspace-git | 11:30 |
| sean-k-mooney | basically im trying to figure out how to set the devstack_plugins line to use the local copy | 11:32 |
| sean-k-mooney | networking-ovs-dpdk: https://opendev.org/x/networking-ovs-dpdk | 11:33 |
| sean-k-mooney | instead of having it point to the upstream git repo | 11:33 |
| sean-k-mooney | does it matter i guess it would have cloning disabled and devstack woudl just use the copy in /opt/stack/networking-ovs-dpdk ? | 11:34 |
| *** ociuhandu has quit IRC | 11:34 | |
| *** diablo_rojo has quit IRC | 11:38 | |
| *** sshnaidm is now known as sshnaidm|afk | 12:00 | |
| *** rfolcOUT is now known as rfolco | 12:07 | |
| AJaeger | sorry, don't know enough about devstack - but you need to use the local copy since that one has your changes | 12:08 |
| AJaeger | sean-k-mooney: I advise you check what other repos do - or ask on #openstack-qa | 12:09 |
| *** surpatil has quit IRC | 12:09 | |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Ensure correct cleanup on repo update and reset https://review.opendev.org/701531 | 12:24 |
| *** rosmaita has joined #openstack-infra | 12:34 | |
| *** ykarel is now known as ykarel|afk | 12:35 | |
| *** jpena is now known as jpena|lunch | 12:36 | |
| *** ykarel|afk has quit IRC | 12:47 | |
| *** udesale has quit IRC | 12:52 | |
| openstackgerrit | Liron Kuchlani proposed openstack/os-testr master: subunit2html dealing with incorrect data format https://review.opendev.org/700778 | 12:53 |
| openstackgerrit | Liron Kuchlani proposed openstack/os-testr master: subunit2html dealing with incorrect data format https://review.opendev.org/700778 | 12:55 |
| *** rlandy has joined #openstack-infra | 12:56 | |
| *** liuyulong has joined #openstack-infra | 12:58 | |
| *** liuyulong has quit IRC | 12:59 | |
| sean-k-mooney | AJaeger: looking at all the uses via codesearch they all use the url to the git repo. | 13:01 |
| sean-k-mooney | so i think its fine ill need to double check the devstack logs but for now im going to assume its correct | 13:02 |
| AJaeger | sean-k-mooney: Then devstack might just use the path for it, is their documentation for it? | 13:03 |
| sean-k-mooney | for the plugin interface? yes but devstack run in the age in a mode where it will not clone so its likely just using the plugin name then looking for it in /opt/stack and assuming it is already present | 13:05 |
| sean-k-mooney | ill check the devstack code later to confirm | 13:05 |
| sean-k-mooney | /age/gate/ | 13:07 |
| *** liuyulong has joined #openstack-infra | 13:08 | |
| *** liuyulong_ has joined #openstack-infra | 13:08 | |
| *** goldyfruit_ has quit IRC | 13:08 | |
| *** ykarel has joined #openstack-infra | 13:08 | |
| *** ociuhandu has joined #openstack-infra | 13:09 | |
| openstackgerrit | sean mooney proposed openstack/project-config master: Add ovs and dpdk source to available repos https://review.opendev.org/701539 | 13:13 |
| openstackgerrit | sean mooney proposed openstack/project-config master: Add ovs and dpdk source to available repos https://review.opendev.org/701539 | 13:17 |
| *** ociuhandu has quit IRC | 13:17 | |
| slaweq | hi infra-root | 13:26 |
| slaweq | do You know why e.g. on https://55e5ed78ab8a48d2f6f2-536b57e3f93d8003b521e21618376b89.ssl.cf2.rackcdn.com/671293/30/gate/openstack-tox-py36/a8ca0cd/testr_results.html.gz I see strange chars only, instead of proper html page? | 13:26 |
| slaweq | I have it like that for every patch I checked today | 13:27 |
| slaweq | and every job which I checked | 13:27 |
| mordred | yoctozepto: if you're interested in keeping it alive, I'd be happy to adopt it in sdk from a governance perspective, if you think that would be helpful | 13:27 |
| sean-k-mooney | slaweq: im pretty sure its not serving the page with the correct content type | 13:28 |
| sean-k-mooney | slaweq: so you are seeign the raw gz bytes | 13:28 |
| mordred | slaweq: yes, sean-k-mooney is right | 13:29 |
| sean-k-mooney | it does have content-encoding gzime and content type text/html | 13:29 |
| mordred | this is an issue from a change yesterday to fix a different issue | 13:29 |
| slaweq | mordred: sean-k-mooney: ok, thx | 13:29 |
| sean-k-mooney | /gzime/gzip | 13:30 |
| slaweq | but my question was more: are You aware of it already and is someone working on fix? :) | 13:30 |
| sean-k-mooney | im not sure why that is not being decoded | 13:30 |
| slaweq | TBH I really like those page with results summary and I miss it a lot now :) | 13:31 |
| mordred | I don't know that we've reached a conclusion as to the best way to fix it for this week... last night we were discussing a couple of options | 13:31 |
| slaweq | mordred: ok, it's good to hear that You are already aware of it | 13:31 |
| slaweq | thx a lot | 13:31 |
| mordred | sean-k-mooney: we had an issue where things were being double encoded / double decoded because of an "interesting" behavior from some swifts. so we fixed that - which broke this | 13:32 |
| mordred | slaweq: I'm with you - I was trying to see why a test job failed last night and it was ... not fun | 13:32 |
| sean-k-mooney | mordred: ah ok | 13:32 |
| sean-k-mooney | i guess you can still donwload it and decompress it locally | 13:33 |
| sean-k-mooney | strange ok so looking at workign links it looks like the content type and encoding are the same | 13:34 |
| sean-k-mooney | so on the broken one its seam like its double compressed | 13:35 |
| sean-k-mooney | just comparing ot https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_0c7/700894/2/check/tempest-integrated-compute/0c73f5a/testr_results.html.gz | 13:35 |
| *** dpawlik has quit IRC | 13:37 | |
| *** dpawlik has joined #openstack-infra | 13:37 | |
| slaweq | sean-k-mooney: yes, decompressing it locally works, but that isn't fun ;) | 13:37 |
| *** ociuhandu has joined #openstack-infra | 13:38 | |
| *** jpena|lunch is now known as jpena | 13:38 | |
| AJaeger | mordred: could you review the stack at https://review.opendev.org/#/c/699683 again, please? | 13:40 |
| *** kjackal has joined #openstack-infra | 13:41 | |
| mordred | AJaeger, smcginnis: we should add some more people to that core team who work on openstack and have interest ... | 13:42 |
| *** ociuhandu has quit IRC | 13:42 | |
| openstackgerrit | Monty Taylor proposed openstack/reviewstats master: Clean up requirements https://review.opendev.org/699717 | 13:43 |
| *** Goneri has joined #openstack-infra | 13:44 | |
| mordred | AJaeger: stack is +2 - finding a second +2 might be hard - maybe given the content we shoudl consider your +1 a +2 and just +A it? | 13:46 |
| *** rh-jelabarre has joined #openstack-infra | 13:50 | |
| *** zxiiro has joined #openstack-infra | 13:55 | |
| AJaeger | mordred: smcginnis is core as well, so take that into account and just +A... | 13:56 |
| mordred | AJaeger: ++ | 13:58 |
| smcginnis | That core list could probably be cleaned up a bit. | 13:59 |
| AJaeger | Let's ask clarkb what he wants to do with reviewstats ^ | 14:01 |
| *** hrw has joined #openstack-infra | 14:03 | |
| hrw | morning | 14:03 |
| evrardjp | ttx: are you usual tc stats generated with reviewstats? | 14:04 |
| hrw | how often ubuntu-cloud-archive mirror is synced? patch adding arm64 got merged yesterday so I though that ~24h will be enough ;D | 14:04 |
| openstackgerrit | Merged openstack/reviewstats master: Switch to py37 jobs https://review.opendev.org/699683 | 14:10 |
| openstackgerrit | Merged openstack/reviewstats master: Drop py2 testing and metadata https://review.opendev.org/699680 | 14:10 |
| openstackgerrit | Merged openstack/reviewstats master: Clean up requirements https://review.opendev.org/699717 | 14:10 |
| fungi | hrw: seems that adding a new architecture has caused it to start logging messages like this: | 14:33 |
| fungi | New architecture 'arm64' in 'bionic-updates/train'. Perhaps you want to call `reprepro flood 'bionic-updates/train' 'arm64'` to populate it with architecture 'all' packages from other architectures. | 14:33 |
| fungi | i'll give that a try | 14:33 |
| hrw | fungi: thank you | 14:33 |
| *** ociuhandu has joined #openstack-infra | 14:38 | |
| fungi | okay, i've run that command for each configured suite all the way back to trusty-updates/liberty | 14:39 |
| fungi | now to see if it will update | 14:39 |
| hrw | thanks | 14:40 |
| *** eharney has quit IRC | 14:42 | |
| *** ociuhandu has quit IRC | 14:43 | |
| *** ociuhandu has joined #openstack-infra | 14:44 | |
| fungi | hrw: http://files.openstack.org/mirror/ubuntu-cloud-archive/dists/bionic-updates/train/main/binary-arm64/ | 14:44 |
| hrw | nice | 14:45 |
| hrw | hm. ussuri is not mirrored? | 14:46 |
| hrw | or in progress? | 14:46 |
| ttx | evrardjp: not really | 14:47 |
| *** ociuhandu has quit IRC | 14:47 | |
| fungi | hrw: i don't know. is there already a ussuri uca? | 14:54 |
| fungi | if so, it's probably just a matter of extending that same file | 14:54 |
| *** sshnaidm|afk is now known as sshnaidm | 14:54 | |
| hrw | fungi: http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/bionic-updates/ussuri/ | 14:56 |
| hrw | will look in system-config and create patch if not | 14:56 |
| openstackgerrit | Marcin Juszkiewicz proposed opendev/system-config master: UCA: mirror Ussuri packages https://review.opendev.org/701552 | 14:58 |
| *** rcernin has quit IRC | 15:01 | |
| openstackgerrit | Merged openstack/reviewstats master: Remove fuel project https://review.opendev.org/699981 | 15:02 |
| *** priteau has joined #openstack-infra | 15:06 | |
| *** dave-mccowan has joined #openstack-infra | 15:06 | |
| AJaeger | fungi, could you review this infra-manual change, please? https://review.opendev.org/#/c/700636/ | 15:08 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: spec: add a zuul-runner cli https://review.opendev.org/681277 | 15:12 |
| *** liuyulong has quit IRC | 15:18 | |
| *** liuyulong_ has quit IRC | 15:18 | |
| *** jkt has joined #openstack-infra | 15:18 | |
| *** ociuhandu has joined #openstack-infra | 15:18 | |
| jkt | hi there, I'm wondering about opendev and hosting random open source projects -- is that something that's possible? | 15:18 |
| jkt | it's about a corporate-backed, but FLOSS project, https://github.com/Telecominfraproject/oopt-gnpy , we currently do not have a budget for sysadmining | 15:19 |
| fungi | jkt: it's more than possible. folks have been hosting random open source projects in opendev for a while | 15:19 |
| jkt | so far we've been on github + travis, and we more or less got fed up with that crazy workflow of force pushes | 15:19 |
| * fungi refrains from making a joke about the randomness of openstack | 15:19 | |
| jkt | :) | 15:19 |
| fungi | https://opendev.org/explore/organizations | 15:20 |
| fungi | that's basically a quick listing of the git repository namespaces in opendev now | 15:20 |
| fungi | (it already spans more than a page) | 15:21 |
| jkt | fungi: what should I do to get this project involved? | 15:21 |
| jkt | also, can you integrate with GerritHub? Gerrit versions, polygerrit UI improvements, etc | 15:21 |
| corvus | re gerrit versions -- we're working on an upgrade | 15:23 |
| fungi | not sure what integration gerrithub offers. opendev is a shared gerrit, currently on an older version (2.13.12) but mordred is nearing completion on a revamp of our configuration management for it which should allow us to rapidly fast-forward to the latest release | 15:23 |
| *** ociuhandu has quit IRC | 15:23 | |
| mordred | yeah - hopefully we'll be on pollygerrit RSN | 15:24 |
| jkt | nice -- I got used to polygerrit at $work and I really like it | 15:24 |
| mordred | yeah - it's super nice | 15:24 |
| jkt | btw, it's super-nice to see that work on checks plugin | 15:24 |
| mordred | wait til you see the zuul integration with latest gerrit corvus has been working on | 15:24 |
| mordred | ah - youve already seen it :) | 15:24 |
| fungi | we're very excited by the gerrit community's interest in using zuul too | 15:25 |
| jkt | before I decided to ask here, I wanted to see if it's possible to get TravisCI tied with Gerrit without running extra VMs | 15:25 |
| jkt | so I played with IBM's free tier for their functions, the workflow was with a webhooks plugin at gerrithub which triggers a "serverless function" in IBM's freebie offering which in turn massges Travis to build against Gerrit's ref | 15:26 |
| mordred | jeez | 15:27 |
| jkt | it kinda-works, except that GerritHub is replicated, which means that every build gets triggered twice, and I have not sorted out propagation in the other way yet :) | 15:27 |
| jkt | yeah, so I thought that it's probably better to ask someone for hosting :) | 15:27 |
| *** eharney has joined #openstack-infra | 15:28 | |
| mordred | \o/ | 15:29 |
| fungi | just to set expectations, the ci system integrated in opendev is zuul, you basically can't merge anything without at least getting it added to zuul's tenant configuration | 15:29 |
| fungi | and triggering travis jobs and having it report in opendev's gerrit isn't something which seems like a good use of time | 15:30 |
| fungi | but zuul ought to be able to run anything travis can | 15:31 |
| jkt | oh yes, I actually much prefer Zuul | 15:31 |
| openstackgerrit | Merged openstack/infra-manual master: translation: drop babel extractor definitions https://review.opendev.org/700636 | 15:32 |
| jkt | I wasn't sure if opendev is open to external organizations before, that's why I tried to investigate that abomination | 15:32 |
| fungi | cool, we've had folks on another project ask about temporary travis integration for migration purposes, but it looked like a lot of work for not much gain | 15:32 |
| *** xek has quit IRC | 15:32 | |
| openstackgerrit | Paul Albertella proposed zuul/zuul master: Fix Gerrit config issue in tutorial demo https://review.opendev.org/701562 | 15:33 |
| *** xek has joined #openstack-infra | 15:33 | |
| fungi | and yeah, i think there's an faq entry on the front page of https://opendev.org/ identical to your question | 15:33 |
| fungi | in case you need to point anyone to that | 15:33 |
| corvus | DNS:www.zuul-ci.org, DNS:www.zuulci.org, DNS:zuul-ci.org, DNS:zuulci.org | 15:35 |
| corvus | \o/ | 15:35 |
| corvus | fungi, mordred: i think https://review.opendev.org/701472 is ready | 15:35 |
| mordred | corvus: woot | 15:35 |
| jkt | travis also appears to have undocumented API rate limit of 50 "requests" per hour (and it seems that actually mean "build" with "request", and that their idea of an hour is different from mine) | 15:36 |
| mordred | haha | 15:36 |
| AJaeger | config-core, could please review https://review.opendev.org/700913, https://review.opendev.org/701244 , and https://review.opendev.org/700909 ? | 15:37 |
| *** udesale has joined #openstack-infra | 15:39 | |
| *** ociuhandu has joined #openstack-infra | 15:42 | |
| openstackgerrit | Paul Albertella proposed zuul/zuul-jobs master: Add Bazel build and install roles https://review.opendev.org/693513 | 15:43 |
| openstackgerrit | Merged openstack/project-config master: Add ozj to docs site https://review.opendev.org/701244 | 15:48 |
| AJaeger | thanks, mordred | 15:49 |
| *** ociuhandu has quit IRC | 15:50 | |
| *** ociuhandu has joined #openstack-infra | 15:51 | |
| openstackgerrit | Merged openstack/project-config master: Add `sushy-cli` project https://review.opendev.org/700909 | 15:52 |
| *** ykarel is now known as ykarel|afk | 15:52 | |
| fungi | #status log trove instance subunit2sql-MySQL was down briefly between 05:00 and 05:25 utc due to an offline migration for impending host issues | 15:55 |
| openstackstatus | fungi: finished logging | 15:55 |
| *** dtantsur is now known as dtantsur|bbl | 15:55 | |
| *** portdirect has joined #openstack-infra | 15:55 | |
| *** ociuhandu has quit IRC | 15:58 | |
| *** eernst has joined #openstack-infra | 15:59 | |
| yoctozepto | mordred: re: sdk and js-openstack-lib - thanks for proposing that, I thought it fits there best indeed :-) | 16:02 |
| *** ykarel|afk is now known as ykarel | 16:03 | |
| openstackgerrit | Merged opendev/system-config master: Use LE certs for zuul-ci.org https://review.opendev.org/701472 | 16:07 |
| clarkb | AJaeger: yoctozepto I ront think the je lib shpuld be an infra deliverable. We (likely) wont use it and are very unlikely to care for it | 16:09 |
| efried | o/ infra | 16:10 |
| efried | I've noticed yesterday and today that when I punch through to a *.gz from job results, it shows up gobbledygook. Presumably not unzipped, though I can't tell that from the browser -- but when I punch the corresponding `raw` link the .gz file downloads (and is a real .gz file). | 16:10 |
| clarkb | bah "dont think the js lib" | 16:10 |
| efried | e.g. https://zuul.opendev.org/t/openstack/build/35e0907add8046369796b05834965174/log/logs/etc/nova/nova.conf.txt.gz | 16:10 |
| efried | doesn't seem to be a browser thing, cause same result in ff/chrome | 16:10 |
| corvus | efried: we did make a change regarding that; must have missed something | 16:10 |
| clarkb | corvus: the fix is the change you asked to wait on merging | 16:11 |
| corvus | clarkb: er, i thought we set a site var? | 16:11 |
| efried | phew, glad it wasn't another situation where it's just me :P | 16:11 |
| clarkb | the sitr var handled everything but subunut and test results files I think | 16:12 |
| corvus | clarkb: ^nova.conf.gz | 16:12 |
| clarkb | oh hrm is that a legacy job? | 16:12 |
| fungi | anything which has been precompressed isn't being written to swift with a content encoding | 16:12 |
| clarkb | devstack-gate may still compress internally | 16:12 |
| fungi | is that the gist? | 16:12 |
| clarkb | fungi: yes | 16:12 |
| fungi | (well, anything which has a .gz extension when we collect it anyway) | 16:13 |
| corvus | clarkb: yes it's legacy | 16:14 |
| corvus | efried: ^ | 16:15 |
| efried | "It used to work"? | 16:15 |
| fungi | for some definition of work | 16:15 |
| efried | used to show up as text in my browser. | 16:16 |
| efried | rather than having to download and unzip manually | 16:16 |
| fungi | this is fallout from trying to prevent rackspace's swift/cdn from helpfully decompressing .tar.gz files when fetched with wget/curl, which was breaking other things | 16:16 |
| AJaeger | clarkb: I agree - and mordred proposed to move js-openstack-lib into SDK | 16:17 |
| efried | I don't think nova-next is a legacy job, and this is similar: https://21ca3af7674a9a12fa4d-6282d3869df7fb619a0818038dd2e45a.ssl.cf5.rackcdn.com/693460/9/check/nova-next/2e09d2a/testr_results.html.gz | 16:17 |
| AJaeger | (if yoctozepto wants to help) | 16:17 |
| *** mattw4 has joined #openstack-infra | 16:17 | |
| clarkb | efried: subunit and test result files are a subset of the problem with a fix up | 16:18 |
| corvus | yeah, we should be able to merge that next week | 16:18 |
| efried | okay, if there's a fix in the pipe, cool, I'll work around for now. | 16:18 |
| clarkb | (basically weneed to not precompress things in jobs unless their intended state is compressed otherwise we break things that are intended to be compressed) | 16:18 |
| clarkb | webreak things like release artifacts | 16:19 |
| corvus | but the nova.conf one will either need a fix to devstack-gate or the job needs to be upgraded to zuulv3 | 16:19 |
| yoctozepto | clarkb, AJaeger: agreed, never suggested keeping it as infra deliverable, just not dropping completely | 16:19 |
| efried | oh | 16:19 |
| fungi | there are likely other things d-g is precompressing too | 16:19 |
| yoctozepto | I have yet to sit down to play with it | 16:19 |
| fungi | i need to disappear to grab lunch and run some errands, but should be back in an hour or so | 16:19 |
| yoctozepto | to see if it has any real value | 16:19 |
| efried | so this is, like, a roundabout way of encouraging folks to upgrade legacy jobs :P | 16:19 |
| AJaeger | config-core, please review https://review.opendev.org/701446 and https://review.opendev.org/700913 | 16:20 |
| fungi | efried: take note there's also an upcoming openstack release goal for that too | 16:20 |
| melwitt | hi, I was wondering does anyone know why these conf files don't view as text in the browser whereas they do for other jobs? https://zuul.opendev.org/t/openstack/build/35e0907add8046369796b05834965174/log/logs/etc/nova/nova.conf.txt.gz | 16:20 |
| efried | melwitt: rewind about 10 minutes :) | 16:21 |
| melwitt | lol, oops. will read | 16:21 |
| efried | melwitt: but yeah, I came here after looking at the same one | 16:21 |
| efried | (Though I pretended it worked for me. But I had to download and manually unzip.) | 16:21 |
| melwitt | cool. I had it on my todo list to ask first thing today bc I was running into it last night when no one was around to ask | 16:22 |
| efried | clarkb, corvus, fungi: Not sure if this was clear/known, but this definitely didn't start happening earlier than when I buggered off in mid-December. So like, it's not related to the swift cutover or anything. | 16:23 |
| *** Goneri has quit IRC | 16:24 | |
| *** Goneri has joined #openstack-infra | 16:24 | |
| melwitt | I miss having the build name in the url in the past. trying to figure out which build this was when handed a direct link to a file from the build | 16:24 |
| efried | oh, my bad | 16:25 |
| clarkb | efried: it is new and known. It happened because we just realized that sometimes our release artifacts are broken | 16:25 |
| clarkb | efried: on tracking that back it was a result of wgetting foo.tar.gz from swift and getting back an uncompressed file | 16:25 |
| clarkb | efried: things that then expect gzipped file break | 16:25 |
| clarkb | efried: the change we made was to stop setting the encoding type on gzipped files when uploading them to swift so that whenever you download them you get backed compressed data. This fixes the release artifacts but breaks things like your config file | 16:26 |
| openstackgerrit | Merged opendev/system-config master: UCA: mirror Ussuri packages https://review.opendev.org/701552 | 16:26 |
| clarkb | efried: the fix for things that don't actually need to be compressed is to stop compressing them upfront. swift uploading will still compress them on upload but we can logically work with them as if they were uncompressed | 16:26 |
| smcginnis | melwitt: Not as convenient, but if you drop the "logs/..." end of the URL, it does get you to the summary report. | 16:26 |
| efried | okay, so fix $overthere related to releases had ripple effects $overhere in day to day job archaeology, got it. Thanks for the explanation. | 16:27 |
| clarkb | efried: we set a global var in our zuul install to stop compressing things upfront in the zuulv3 role that does that | 16:27 |
| clarkb | efried: things that don't use the zuulv3 role to do that (subunit, testresults, devstack-gate) need to updated separately | 16:27 |
| melwitt | smcginnis: that's helpful, I was missing where it shows the name altogether. thanks | 16:27 |
| *** rishabhhpe has joined #openstack-infra | 16:27 | |
| hrw | ok. uca/ussuri merged. tomorrow will do recheck for kolla uca/ussuri patch then | 16:28 |
| efried | clarkb: so tldr I should continue to report if I see this, because it probably indicates a corner we haven't swept yet? | 16:28 |
| rishabhhpe | Hi All , Can anyone help me out for setting up third party CI setup for zuul v3 along with latest version of devstack ? | 16:28 |
| smcginnis | melwitt: Better than nothing at least. ;) | 16:28 |
| melwitt | yeah, wfm | 16:28 |
| clarkb | efried: I guess? I don't expect the infra team will be fixing the legacy devstack-gate stuff which is likely to be a large chunk of it | 16:29 |
| clarkb | (we've been trying to assert that is unmaintained tooling for a long time now and really only modify it to keep grenade working :/) | 16:30 |
| melwitt | oh, does anyone here know anything about grenade? based on some observation, it looks like grenade jobs always have to pull devstack from the master branch and are incapable of using the contents of a proposed patch? can anyone confirm that that's true? I looked through the code and wasn't 100% sure what I was seeing. based on this it looked like it confirmed my suspicion that devstack can only be used from master and not a proposed | 16:30 |
| melwitt | patch https://github.com/openstack/grenade/blob/master/inc/bootstrap#L263-L268 | 16:30 |
| *** openstackgerrit has quit IRC | 16:30 | |
| clarkb | melwitt: the repo should already be there checked out to the correct ref | 16:31 |
| clarkb | melwitt: that is what devstack-gate does | 16:32 |
| melwitt | clarkb: ok, hm. in this patch we were trying to get some additional debug info by adding some settings to nova.conf in a DNM patch https://review.opendev.org/692402 and it is *not* reflecting the values into grenade jobs like grenade-py3 where we need it | 16:33 |
| melwitt | and I can't figure out what's going wrong | 16:33 |
| efried | melwitt: ah - and I picked nova-live-migration at random, where it does show up | 16:33 |
| * melwitt nods | 16:34 | |
| clarkb | melwitt: is it because the greande job does train -> master and the train side sets the config? | 16:34 |
| efried | okay, afaict we (nova) have two legacy jobs in check/gate (nova-live-migration, nova-grenade-multinode) and a bunch in experimental. Sounds like we need to prioritize porting those jobs to zv3. This is where I wish we could send up the mriedem signal. | 16:34 |
| clarkb | melwitt: I think you may need that change to be on stable/train devstack | 16:34 |
| efried | Or the sean-k-mooney signal? | 16:34 |
| melwitt | clarkb: that was my theory so I tried a nova patch that Depends-On that one https://review.opendev.org/701478 and that didn't work. I could try a stable/train change but would that get us the settings on the ussuri side too? | 16:35 |
| efried | ("...prioritize porting those jobs to zv3..." meaning the two check/gate jobs. I'm happy to leave the experimental ones for now.) | 16:35 |
| melwitt | I mean, it might be good enough anyway because I think things have been failing on the old side anyway | 16:36 |
| melwitt | *always failing | 16:36 |
| clarkb | melwitt: I think grenade doesn't update settings in the upgrade unless there are specific actions to do so in the grenade upgrade scripts | 16:37 |
| clarkb | melwitt: this ensures that when you upgrade a cloud you don't break because N+1 stopped accepting a config option | 16:37 |
| melwitt | ah I see | 16:37 |
| clarkb | that means the config for the master test is written by train devstack currently | 16:37 |
| corvus | efried: i can't volunteer to do it, but i'd be happy to help with any questions. a year or 2 ago when we were really focused on this, it was supposed to be pretty easy to switch them over -- like, almost just a mechanical exercise in translating env vars to yaml. | 16:38 |
| melwitt | clarkb: ohhh ok. thank you, that's helpful | 16:38 |
| clarkb | then you update N+1 devstack to stop writing deprecated configs and upgrade to N+2 can drop support for that config setting | 16:38 |
| clarkb | AJaeger: for the fuel stuff I think this may be a good reason for me to set up gertty again | 16:39 |
| corvus | (the real unfortunate thing here is that *new* jobs were created based on legacy jobs) | 16:39 |
| clarkb | unless corvus or fungi want to escalate privs and use their existing gertty to mass merge things ? | 16:39 |
| efried | corvus: Thanks. These are particularly complicated jobs, which is why I suspect they haven't been tackled yet. I'm still pretty green to this stuff, so I'm going to do my best to delegate the switch, but there's a nonzero probability I (or whoever) will take you up on that. | 16:39 |
| clarkb | AJaeger: let me start on that | 16:39 |
| AJaeger | clarkb: in a call for the next 60 minutes | 16:40 |
| clarkb | AJaeger: k | 16:40 |
| *** lmiccini has quit IRC | 16:40 | |
| clarkb | good time for me to get that set up then | 16:40 |
| AJaeger | ;) | 16:40 |
| sean-k-mooney | efried: sorry was on a call you rang :) | 16:43 |
| cloudnull | hey all, anyone around able to take a look at https://review.opendev.org/#/c/700452 | 16:45 |
| *** gyee has joined #openstack-infra | 16:45 | |
| corvus | cloudnull: let me ask about that in #zuul | 16:48 |
| sean-k-mooney | efried: melwitt there is a patch to make a zuulv3 native grenatde job | 16:49 |
| sean-k-mooney | it seamded to work but it also was missing some functionality | 16:49 |
| sean-k-mooney | https://review.opendev.org/#/c/548936/ | 16:50 |
| *** rishabhhpe has quit IRC | 16:54 | |
| * cloudnull goes to join #zuul | 16:54 | |
| efried | sean-k-mooney: Can I ask you to take the lead on getting nova-live-migration and the grenade jobs ported over? It sounds like we're going to be running into compressed CI artifacts in those, which will be a real PITA very quickly, so it has become a higher priority. And you're the most experty zuul person I know of... | 16:54 |
| cloudnull | thanks corvus | 16:54 |
| *** rishabhhpe has joined #openstack-infra | 16:55 | |
| sean-k-mooney | efried: then we are all doomed.... but ok i can try that said i need to check if mriedem/stephenfin have start on those | 16:56 |
| stephenfin | sean-k-mooney: mriedem had, yeah | 16:56 |
| sean-k-mooney | efried: i am planing to spend then next few weeks working on ci anyway so ill bring this up on the list | 16:56 |
| *** openstackgerrit has joined #openstack-infra | 16:57 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/project-config master: New repo: "ideas" https://review.opendev.org/701530 | 16:57 |
| sean-k-mooney | i just got a networking-ovs-dpdk job working so next on the list is move the os-vif legacy job to zuulv3 which ill do tomorow then i can start on the nova ones | 16:57 |
| *** lucasagomes has quit IRC | 17:00 | |
| clarkb | I'm going to find breakfast while gertty syncs things for me | 17:01 |
| clarkb | efried: sean-k-mooney stephenfin re grenade tosky has been working on getting a zuulv3 version of grenade going | 17:02 |
| clarkb | I think it is very close (I reviewed the stack pre holidays) and assistance getting that rolled out would be much appreciated I expect | 17:02 |
| *** ociuhandu has joined #openstack-infra | 17:03 | |
| tosky | yep, sorry; thanks to the general help here I was able to refresh the patches, but something is broken when using plugins (see all the patches with the same topic) | 17:03 |
| tosky | I will be able to recheck them next week | 17:03 |
| sean-k-mooney | clarkb: yes i was hoping we could move our grenade jobs to the version when it was complete | 17:04 |
| *** pgaxatte has quit IRC | 17:05 | |
| *** udesale has quit IRC | 17:05 | |
| *** ociuhandu has quit IRC | 17:08 | |
| *** psachin has quit IRC | 17:12 | |
| *** iurygregory has quit IRC | 17:13 | |
| johnsom | Hi all, I see in the scroll back there is some discussion of the broken gz links in zuul. Is there a guide of what I need to change for our jobs or is that change being reverted? | 17:15 |
| *** ykarel is now known as ykarel|away | 17:15 | |
| johnsom | Example: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d5e/701485/1/check/openstack-tox-py37/d5ed048/testr_results.html.gz | 17:15 |
| johnsom | I did't see any e-mail on this, but that doesn't mean I didn't miss it while out for vacation | 17:16 |
| johnsom | These are template jobs, so we will need to update this in a bunch of repositories I suspect. | 17:17 |
| donnyd | FYI Fortnebula just had a power outage - looks like one of my UPS batteries are in need to replacing | 17:17 |
| *** rishabhhpe has quit IRC | 17:18 | |
| clarkb | johnsom: the testr and subunit stuff will be handled for you | 17:18 |
| efried | johnsom: you're magic #3. I think a ML post would probably be a good thing, esp because the answer isn't a simple one. | 17:18 |
| johnsom | Yeah, I saw it yesterday, but thought I would give it a day. grin | 17:18 |
| corvus | clarkb: what about splitting the subunit change from the stage-output change | 17:18 |
| *** rishabhhpe has joined #openstack-infra | 17:19 | |
| clarkb | corvus: I can do that | 17:19 |
| efried | Sounds to me like there are at least three categories: | 17:19 |
| efried | - testr/subunit on v3, which should be fine now | 17:19 |
| efried | - legacy jobs: you have been warned, go convert them | 17:19 |
| efried | - something something patch proposed but not yet merged | 17:19 |
| clarkb | efried: 1) is not fine yet, but will be soon | 17:19 |
| clarkb | 3) is related to 1^ | 17:19 |
| *** kjackal has quit IRC | 17:19 | |
| * efried clearly is not qualified to author the suggested ML post | 17:20 | |
| clarkb | https://review.opendev.org/#/c/701282/ I'll split that change into two so we can merge the subunit fix early | 17:20 |
| corvus | if we split clarkb's patch in 2 and merge half of it now, we can reduce that list to "legacy jobs" | 17:20 |
| johnsom | Yeah, it's not just the tox template jobs. Our octavia logs from zuulv3 jobs are also broken | 17:20 |
| clarkb | johnsom: right the subunit thing is global | 17:21 |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: Stop compressing files during intermediate steps part 2 https://review.opendev.org/701282 | 17:24 |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: top compressing files during intermediate steps part 1 https://review.opendev.org/701578 | 17:24 |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: Stop compressing files during intermediate steps part 1 https://review.opendev.org/701578 | 17:24 |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: Stop compressing files during intermediate steps part 2 https://review.opendev.org/701282 | 17:24 |
| clarkb | corvus: ^ split now | 17:24 |
| sean-k-mooney | clarkb: so is stage_compress_logs=True why the result html is double compressed | 17:25 |
| clarkb | sean-k-mooney: no. That is a swift implementation thing | 17:26 |
| sean-k-mooney | oh ok | 17:26 |
| mordred | clarkb: nice side-effect of this is that the links to testr_results.html are going to be more pleasant | 17:26 |
| clarkb | sean-k-mooney: assuming its the deflate + gzip double compress | 17:26 |
| clarkb | that should be fixed once the changes above merge too | 17:26 |
| *** ociuhandu has joined #openstack-infra | 17:26 | |
| *** ociuhandu has quit IRC | 17:27 | |
| sean-k-mooney | ya well i guess we will see shortly | 17:27 |
| *** ociuhandu has joined #openstack-infra | 17:27 | |
| *** kjackal has joined #openstack-infra | 17:29 | |
| *** tbarron_ is now known as tbarron | 17:30 | |
| *** ociuhandu has quit IRC | 17:31 | |
| *** evrardjp has quit IRC | 17:33 | |
| *** evrardjp has joined #openstack-infra | 17:34 | |
| *** ociuhandu has joined #openstack-infra | 17:34 | |
| clarkb | efried: email sent | 17:35 |
| gmann | need one more d-g core to approve this. this is needed to un-block Tempest, Patrole, stable/queens|pike|ocata gate - https://review.opendev.org/#/c/701404/1 | 17:37 |
| openstackgerrit | Merged zuul/zuul-jobs master: Stop compressing files during intermediate steps part 1 https://review.opendev.org/701578 | 17:40 |
| *** ociuhandu has quit IRC | 17:41 | |
| corvus | apache failure on files02, investigating | 17:42 |
| corvus | AH00016: Configuration Failed | 17:43 |
| corvus | that's an extremely lame error message | 17:43 |
| sean-k-mooney | that seamed to work https://4a5f7324f50c525e768c-c393df920c3ba3d74686e2bffd0842f4.ssl.cf2.rackcdn.com/701578/2/gate/tox-py37/ab32af6/testr_results.html | 17:44 |
| *** ociuhandu has joined #openstack-infra | 17:44 | |
| sean-k-mooney | although im not sure if the result issue only effeced legacy jobs? | 17:44 |
| mordred | corvus: that message tells you SO MUCH | 17:45 |
| clarkb | sean-k-mooney: no testr_results and subunit_log.txt.gz or whatever that files is called was universal | 17:45 |
| clarkb | sean-k-mooney: it should now be fixed wherever fetch-subunit-results role is used | 17:45 |
| mordred | \o/ | 17:45 |
| sean-k-mooney | well that was from the change that just merges so yes it appears to be | 17:46 |
| mordred | that should take care of a large % of the issues I think | 17:46 |
| mordred | except for the ones it doesn't | 17:46 |
| cloudnull | is docs.openstack.org down for anyone else ? I'm getting connection refused (both ipv4/6) ? | 17:47 |
| clarkb | cloudnull: corvus just mentioned an apache configuration error and is investigating | 17:47 |
| johnsom | Same here | 17:47 |
| cloudnull | ah. ok. thanks clarkb corvus | 17:47 |
| *** ociuhandu has quit IRC | 17:49 | |
| corvus | clarkb, mordred, fungi: i have moved the zuul conf files out of the way and restarted apache | 17:50 |
| corvus | i do not know what changed | 17:50 |
| corvus | nor do i understand the error | 17:50 |
| corvus | additional help would be appreciated | 17:50 |
| *** rpittau is now known as rpittau|afk | 17:51 | |
| clarkb | corvus: that implies the error is in the zuul config? where can I see those on files02? | 17:51 |
| corvus | clarkb: ~root/back2 | 17:51 |
| *** factor has joined #openstack-infra | 17:53 | |
| corvus | note that it does not appear that the files were updated with the new cert paths | 17:53 |
| *** tosky has quit IRC | 17:54 | |
| corvus | [Wed Jan 08 17:49:30.331912 2020] [ssl:emerg] [pid 18629:tid 140370605315968] AH02561: Failed to configure certificate zuul-ci.org:443:0, check /etc/ssl/certs/zuul-ci.org.pem | 17:54 |
| corvus | [Wed Jan 08 17:49:30.332007 2020] [ssl:emerg] [pid 18629:tid 140370605315968] SSL Library Error: error:0906D06C:PEM routines:PEM_read_bio:no start line (Expecting: CERTIFICATE) -- Bad file contents or format - or even just a forgotten SSLCertificateKeyFile? | 17:54 |
| mordred | corvus: the cert contents look off | 17:54 |
| corvus | mordred: yep, sure does | 17:54 |
| mordred | I thnk that wants to be a link, yeah? | 17:54 |
| fungi | okay, back. now to find out what excitement i've missed | 17:55 |
| corvus | no, the config file was supposed to be updated with that path | 17:55 |
| corvus | okay, i know what to do to fix things manually, i'll do that now | 17:55 |
| mordred | oh - the config file is supposed to have /etc/letsencrypt-certs/zuul-ci.org/zuul-ci.org.cer instead? | 17:55 |
| mordred | cool | 17:55 |
| corvus | yep | 17:55 |
| mordred | nod | 17:55 |
| mordred | well - at least it explains why apache had a sad | 17:55 |
| clarkb | did we set the contents var to == the file path var? | 17:56 |
| clarkb | that is what it looks like | 17:56 |
| clarkb | which would explain why the path didn't update but the contents did | 17:56 |
| * clarkb looks | 17:56 | |
| *** derekh has quit IRC | 17:57 | |
| clarkb | yup | 17:58 |
| clarkb | I'll have a fix shortly | 17:58 |
| *** dpawlik has quit IRC | 17:58 | |
| corvus | i've restarted apache with a working zuul-ci.org config | 17:58 |
| corvus | looks like we did not get a git.zuul-ci.org cert | 17:58 |
| corvus | clarkb: so we should revert that part of the change too | 17:58 |
| clarkb | corvus: ok | 18:00 |
| corvus | fatal: [adns1.opendev.org]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'ansible.vars.hostvars.HostVarsVars object' has no attribut | 18:00 |
| corvus | e 'acme_txt_required'\n\nThe error appears to be in '/opt/system-config/playbooks/roles/letsencrypt-install-txt-record/tasks/main.yaml': line 5, column 3, but may\nbe elsewhere i | 18:00 |
| corvus | n the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Build key list\n ^ here\n"} | 18:00 |
| corvus | if we're wondering why that didn't happen | 18:00 |
| *** priteau has quit IRC | 18:01 | |
| *** gfidente is now known as gfidente|afk | 18:03 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Fix zuul-ci.org vhost cert paths https://review.opendev.org/701582 | 18:04 |
| clarkb | corvus: ^ I think that handles the path issue and reverts the git.zuul-ci.org bits temporarily | 18:05 |
| corvus | clarkb, fungi, mordred: ^ approved and enqueued into gate | 18:06 |
| fungi | okay, all caught up now. seems i did miss some excitement | 18:06 |
| corvus | the fortnebula outage is why we didn't get the cert | 18:07 |
| clarkb | corvus: because we couldn't get into mirror? | 18:07 |
| corvus | clarkb: presumably; it processes the fortnebula mirror before files02 | 18:08 |
| fungi | ahh, i also missed that the fn mirror is offline. when did that start? | 18:08 |
| *** jpena is now known as jpena|off | 18:08 | |
| corvus | so i'm assuming an earlier step failed for that host, which then caused the value of the required_txt var to be missing | 18:08 |
| donnyd | That is a likely scenario - the generator kicked on and what is up now is being run by it - but it would appear one of the UPS has went offline and is not responding - I won't be able to physically look at it for a while | 18:08 |
| corvus | unfortunately this means we have another place where our control plane is tied to our clouds | 18:09 |
| donnyd | I was made aware at 1217 | 18:09 |
| donnyd | and I let infra know in the same minute fungi | 18:09 |
| corvus | so we either need to make the letsencrypt role more robust, or run a separate playbook for the clouds | 18:09 |
| donnyd | Hopefully my bad timed outage doesn't cause too much heart ache | 18:10 |
| clarkb | corvus: can we set failed_when: false type deal to have it not stop in ansible for that bit of the role? | 18:11 |
| clarkb | then have a verification step on the other side I guess? | 18:11 |
| corvus | clarkb: i think we may just need a soft dict lookup; this is a task that runs only on adns1; ansible has already marked the fortebula host as failed and stopped running tasks on it. | 18:12 |
| *** ociuhandu has joined #openstack-infra | 18:13 | |
| clarkb | ah | 18:13 |
| *** dtantsur|bbl is now known as dtantsur | 18:13 | |
| efried | thanks clarkb (email) | 18:13 |
| fungi | donnyd: ahh, thanks! i see it in scrollback now (at 17:17 utc), i missed it when skimming all the lunchtime backlog | 18:15 |
| *** tesseract has quit IRC | 18:15 | |
| corvus | i'll work on an improvement to the role | 18:16 |
| openstackgerrit | Zane Bitter proposed openstack/project-config master: gerritbot: notify in #heat for stable branches https://review.opendev.org/701584 | 18:17 |
| *** ociuhandu has quit IRC | 18:17 | |
| *** michael-beaver has joined #openstack-infra | 18:18 | |
| *** ykarel|away has quit IRC | 18:23 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Handle offline hosts in LE role https://review.opendev.org/701586 | 18:23 |
| *** ykarel|away has joined #openstack-infra | 18:23 | |
| corvus | clarkb, fungi, mordred: ^ should be more robust; i tested that locally with a mock inventory | 18:24 |
| openstackgerrit | Merged opendev/system-config master: Fix zuul-ci.org vhost cert paths https://review.opendev.org/701582 | 18:25 |
| *** pcaruana has quit IRC | 18:27 | |
| donnyd | Well both ups are finally back up and it looks like I lost both batteries this time around - uggg.... | 18:28 |
| *** ykarel|away has quit IRC | 18:29 | |
| clarkb | poor batteries | 18:31 |
| fungi | ouch | 18:33 |
| * fungi has too many upses at home, constantly replacing batteries | 18:33 | |
| *** dtantsur is now known as dtantsur|afk | 18:34 | |
| *** kjackal has quit IRC | 18:37 | |
| *** rishabhhpe has quit IRC | 18:44 | |
| rm_work | bnemec: OK so it looks like it IS possible to send argv through to the wsgi app (eg --pyargv="--config-file my.conf"), it just relies on the app being set up to accept it properly (some are not) | 18:50 |
| * mordred is back online and enjoying the lovely metal tube ... looks like the patches from the earlier thing are +A already | 19:02 | |
| donnyd | They are 5Kw ups - so its like 32 batteries this time around - /me shops for two new ups | 19:09 |
| * donnyd goes shopping for new ups | 19:10 | |
| clarkb | infra-root I'm going to put myself in project bootstrappers so that I can approve ajaeger's fuel project retirements | 19:11 |
| clarkb | now to see how to get gertty to update its vote options | 19:13 |
| bnemec | rm_work: Good to know, thanks. | 19:13 |
| clarkb | corvus: ^ any hints on how to get gertty to update the review fields available to me after updating my group membership? | 19:15 |
| openstackgerrit | Brian Haley proposed openstack/project-config master: Add periodic panel to Octavia dashboard https://review.opendev.org/701596 | 19:20 |
| corvus | clarkb: reload change | 19:23 |
| corvus | clarkb: ctrl-r by default | 19:24 |
| *** aedc has joined #openstack-infra | 19:27 | |
| *** rlandy is now known as rlandy|mtg | 19:29 | |
| *** pkopec has quit IRC | 19:29 | |
| donnyd | fungi: what are you using for UPS? I have HP ups atm - but they are getting pretty old and I am thinking of doing something different | 19:29 |
| *** jtomasek has quit IRC | 19:32 | |
| AJaeger | config-core, please review https://review.opendev.org/701446 (retire dox) and https://review.opendev.org/700913 (remove duplicated ACLs) | 19:36 |
| clarkb | I'm out of the bootstrapper group. Now going to abandon the existing fuel changes | 19:36 |
| fungi | donnyd: oh, far worse probably. some old non-rackmountable apc back-ups ns 1250 units (i used to lay them sideways on rack shelves because i could squeeze them into a 3u space and monitor over a usb data connection), and also some tripp-lite internet900u compact upses because i can wall-mount them behind things like televisions | 19:38 |
| corvus | donnyd, fungi: the folks at eaton (formerly mge) do a lot of work on open source drivers/monitoring software. i'm unfamiliar with current models though, as my 15yo mge is still happy. | 19:38 |
| fungi | yeah, eaton powerware used to be a customer of mine back when i was working for an isp/hosting company | 19:38 |
| fungi | i didn't know they made consumer-grade gear | 19:39 |
| corvus | fungi: i think they "started" that when they bought mge | 19:39 |
| fungi | aha, that would 'splain it, that was about the time i lost track of them | 19:39 |
| fungi | i mostly knew them for large room-sized equipment | 19:40 |
| clarkb | AJaeger: I think the retire chagne merges are all done as are the other change abandons | 19:40 |
| mwhahaha | any issues with the fortnebula mirror? seeing pip timesouts | 19:40 |
| clarkb | AJaeger: can you check and confirm | 19:40 |
| corvus | i have a 1500va 1u unit from mge; only company doing that capacity in that ff. but i think eaton discontinued that one. | 19:40 |
| mwhahaha | context https://review.opendev.org/#/c/701479/ | 19:40 |
| clarkb | AJaeger: if so I think you can unwip https://review.opendev.org/#/c/699849/ ? | 19:41 |
| corvus | mwhahaha: yeah, fn was completely offline -- maybe now it's partially offline and we need to disable it? | 19:41 |
| AJaeger | clarkb: done | 19:42 |
| mwhahaha | probably | 19:42 |
| mwhahaha | i rechecked again | 19:43 |
| AJaeger | config-core, want to finalize retiring of Fuel repos? I need another +2A on https://review.opendev.org/#/c/699849/ , please | 19:43 |
| AJaeger | clarkb: yes, all looks great. Thanks a lot! | 19:43 |
| openstackgerrit | James E. Blair proposed openstack/project-config master: Temporarily disable fortnebula https://review.opendev.org/701602 | 19:43 |
| mnaser | AJaeger: that's a satisfying -2780 | 19:43 |
| mnaser | thank you for doing a lot of that | 19:43 |
| corvus | clarkb, fungi, donnyd, mwhahaha: https://review.opendev.org/701602 | 19:44 |
| corvus | maybe we want to force-merge that | 19:44 |
| AJaeger | mnaser: thanks for the review | 19:44 |
| AJaeger | mnaser, clarkb, want to review the dox retirement as well, please? https://review.opendev.org/701446 | 19:44 |
| clarkb | corvus: change lgtm, happy for it to be force merged too | 19:44 |
| corvus | clarkb: i'll merge it then | 19:45 |
| mnaser | AJaeger: got it | 19:45 |
| mwhahaha | gracias | 19:45 |
| openstackgerrit | Merged openstack/project-config master: Temporarily disable fortnebula https://review.opendev.org/701602 | 19:45 |
| AJaeger | mnaser: great | 19:45 |
| *** bnemec has quit IRC | 19:47 | |
| *** ramakris has joined #openstack-infra | 19:49 | |
| *** kjackal has joined #openstack-infra | 19:50 | |
| *** bnemec has joined #openstack-infra | 19:52 | |
| openstackgerrit | Merged openstack/project-config master: Retire x/fuel repos (step 4) https://review.opendev.org/699849 | 19:56 |
| AJaeger | \o/ | 19:57 |
| sean-k-mooney | ERROR: No matching distribution found for oslo.service===1.41.1 (from -c /home/zuul/src/opendev.org/openstack/requirements/upper-constraints.txt (line 47)) | 20:00 |
| sean-k-mooney | so is this a one off or is this hitting other jobs? | 20:00 |
| clarkb | sean-k-mooney: can you link to it? | 20:00 |
| clarkb | that can be caused by a number of issues (including potentially the fn mirror outage) | 20:01 |
| sean-k-mooney | ah its on FN | 20:01 |
| sean-k-mooney | Looking in indexes: http://mirror.regionone.fortnebula.opendev.org/pypi/simple, http://mirror.regionone.fortnebula.opendev.org/wheel/ubuntu-18.04-x86_64 | 20:01 |
| clarkb | sean-k-mooney: we've disabled that cloud so mirror related problems there should go away (since jobs will stop running there) | 20:01 |
| *** eharney has quit IRC | 20:01 | |
| sean-k-mooney | ok cool ill ignore it then | 20:02 |
| sean-k-mooney | just said i would check as this is normally either an infra issue like mirrors or we have just done a release | 20:02 |
| openstackgerrit | Merged openstack/project-config master: Retire x/dox (step 2) https://review.opendev.org/701446 | 20:03 |
| openstackgerrit | Merged opendev/system-config master: Handle offline hosts in LE role https://review.opendev.org/701586 | 20:07 |
| AJaeger | config-core, dox is now retired as well - please review https://review.opendev.org/701447 to finalize | 20:07 |
| mordred | AJaeger: \o/ | 20:08 |
| *** rlandy|mtg is now known as rlandy | 20:09 | |
| AJaeger | oh, we need https://review.opendev.org/#/c/701448/ to merge first - config-core, please review | 20:12 |
| AJaeger | thanks, clarkb and mordred ! | 20:14 |
| clarkb | I'm going to pop out for a bike ride and lunch now | 20:14 |
| mordred | AJaeger: thanks for driving that! | 20:14 |
| *** eharney has joined #openstack-infra | 20:15 | |
| AJaeger | you're welcome | 20:16 |
| *** fdegir has quit IRC | 20:22 | |
| *** fdegir has joined #openstack-infra | 20:23 | |
| AJaeger | infra-root, what's up with arm64? I see changes for 6+ hours in the check-arm64 queue waiting for a node - is all good? | 20:24 |
| mnaser | AJaeger: i assume that there's either not enough capacity or the cloud is having issues | 20:24 |
| fungi | there is a very limited pool there, something like 10 nodes | 20:25 |
| * mordred is landing ... or rather, in a metal tube that is landing - likely AFK for the rest of the day, will talk to folks tomorrow! | 20:25 | |
| fungi | i'll take a look and see how many are in use | 20:25 |
| AJaeger | thanks, fungi | 20:25 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-helm master: Add Zuul charts https://review.opendev.org/700460 | 20:25 |
| *** ramakris has quit IRC | 20:25 | |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: WIP: documentation reorg https://review.opendev.org/701608 | 20:29 |
| fungi | looks like we have 2x ubuntu-bionic-arm64 in-use and 3x deleting | 20:29 |
| fungi | for a total of 5 nodes | 20:29 |
| fungi | a couple of the deleting nodes have been in that state for ~8 minutes, so seems it may be slow to process those | 20:30 |
| fungi | the in-use nodes have only been in their state less than a minute | 20:31 |
| *** openstackgerrit has quit IRC | 20:31 | |
| AJaeger | that looks fine - ok, so they might be used not only in the check-arm64 queue | 20:32 |
| fungi | looks like linaro-london (the only provider with those nodes) has max-servers: 8 currently | 20:32 |
| fungi | possible we have some leakage there, i can look at the launcher logs shortly | 20:33 |
| *** openstackgerrit has joined #openstack-infra | 20:37 | |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Remove legacy-dox-buildimages job https://review.opendev.org/701448 | 20:37 |
| *** eharney has quit IRC | 20:38 | |
| openstackgerrit | Merged openstack/project-config master: Finish retiring x/dox (step 4) https://review.opendev.org/701447 | 20:47 |
| *** aedc_ has joined #openstack-infra | 20:48 | |
| *** aedc has quit IRC | 20:51 | |
| fungi | nodepool.driver.NodeRequestHandler[nl03-16591-PoolWorker.linaro-london-main]: Predicted remaining provider quota: {'compute': {'ram': 58272, 'cores': -16, 'instances': 91}} | 20:57 |
| fungi | i guess we're short on available cores there | 20:57 |
| *** Goneri has quit IRC | 20:59 | |
| fungi | openstack server list for that project only shows five instances | 21:01 |
| fungi | all are os.extra.large.infra flavor, a mix of debian-buster-arm64 and ubuntu-xenial-arm64 images | 21:02 |
| fungi | so far it seems like the only nodes i've seen in use are debian-buster-arm64 and i only ever see ubuntu-xenial-arm64 in a deleting state | 21:05 |
| *** ociuhandu has joined #openstack-infra | 21:10 | |
| *** michael-beaver has quit IRC | 21:18 | |
| hrw | there is work in progress to add more arm64 nodes | 21:24 |
| hrw | but can not give ETA | 21:24 |
| hrw | in kolla(-ansible) we only use debian-buster-arm64 and have one job in review which will use ubuntu-bionic-arm64 (but this one waits for uca/ussuri/arm64 to be available and more nodes in infra too) | 21:26 |
| *** rfolco has quit IRC | 21:29 | |
| openstackgerrit | Brian Haley proposed openstack/project-config master: Add periodic panel to Octavia dashboard https://review.opendev.org/701596 | 21:30 |
| fungi | hrw: i'm guessing repeated attempts to run that job is what is eating so many of the nodes, since 60% of the quota is perpetually consumed by ubuntu-xenial-arm64 nodes in a "deleting" state | 21:30 |
| *** ralonsoh has quit IRC | 21:30 | |
| fungi | (doesn't help that it seems to take upwards of 10 minutes for that cloud to delete an instance_ | 21:31 |
| hrw | kevinz: ^^ can you take a look why it takes 10 minutes to delete an instance? | 21:31 |
| hrw | fungi: kevinz is linaro-london admin | 21:31 |
| hrw | lives in China | 21:32 |
| *** spsurya has quit IRC | 21:32 | |
| fungi | the cpu quota also seems to be at odds with the max-servers. we only have enough allowed cores to boot 5 nodes there (according to the errors nodepool is logging) even though max-servers is set to 8 | 21:32 |
| *** armax has joined #openstack-infra | 21:32 | |
| fungi | so probably either the core limit or the flavor needs adjusting | 21:32 |
| hrw | fungi: we may try reverting https://review.opendev.org/#/c/671445/ - with new ceph cluster in linaro-london 8vcpu probably will work fine | 21:33 |
| fungi | ahh, yep if that uses more cores then it probably explains the errors | 21:34 |
| hrw | ok, will revert. need to write commit message | 21:34 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/project-config master: Revert "Linaro London: use new bigger flavour" https://review.opendev.org/701626 | 21:35 |
| clarkb | fungi: what is our absolute quota for cpus? | 21:35 |
| clarkb | nova hasbeen known to leak quota usage | 21:36 |
| fungi | i'll check | 21:36 |
| hrw | fungi: is there a project using those ubuntu-xenial-arm64 nodes? as it would be more sane for them to switch to bionic - xenial/arm64 is full of real old stuff missing several features | 21:38 |
| fungi | according to `openstack limits show --absolute` it's maxTotalCores=128 | 21:38 |
| *** ociuhandu has quit IRC | 21:39 | |
| fungi | hrw: codesearch.o.o doesn't turn up any, but i see we have ubuntu-xenial-arm64 set to min-ready: 1 so if those are failing to come up reachable that could also explain the churn | 21:40 |
| hrw | ok | 21:40 |
| fungi | maybe min-ready is a bad idea when we can only boot ~5 nodes | 21:41 |
| clarkb | 128/5 ~= 25, are these 25 core flavors? | 21:43 |
| hrw | mailed kevinz to take a look at patch and comment | 21:43 |
| hrw | clarkb: 16c | 21:43 |
| clarkb | so ya nova must have gotten quota out of sync | 21:44 |
| fungi | also nodepool is complaining that node requests would put it at -16 cpus and openstack flavor show os.extra.large.infra says it gets vcpus=16 so the limits detail of maxTotalCores=128 and totalCoresUsed=80 leaves me wondering where the other 3 nodes worth are | 21:44 |
| fungi | the math there suggests that max-servers: 8 is correct | 21:44 |
| fungi | somehow nodepool is seeing more cpus in use than `openstack limits show --absolute` reports for totalCoresUsed | 21:45 |
| * hrw -> bed | 21:45 | |
| fungi | have a good night hrw! | 21:45 |
| hrw | you too | 21:45 |
| fungi | thanks | 21:45 |
| hrw | vote/comment https://review.opendev.org/701626 please | 21:46 |
| hrw | I pointed kevinz to irc log so he will know what was discussed | 21:46 |
| clarkb | fungi: we may have cached an older lower maxTotalCores value? | 21:46 |
| fungi | possible? | 21:47 |
| *** sshnaidm is now known as sshnaidm|afk | 21:47 | |
| fungi | how often is that refreshed? | 21:50 |
| clarkb | I think it depends on sdk, possibly never? | 21:50 |
| clarkb | Shrews may know | 21:50 |
| fungi | i guess if maxTotalCores was previously 80 and we were using a flavor with 8 vcpus (targeting max-servers: 10), but later bumped it to 128 in the provider to accommodate larger flavors, that could explain it | 21:51 |
| clarkb | ya, if we restart the launcher that should load up new quota info | 21:52 |
| clarkb | I think nl03 is the launcher responsible for that cloud | 21:53 |
| Shrews | I don't think any caching of quota is done? But I could be wrong and would have to look. | 21:54 |
| Shrews | When we've seen quota oddities in the past, it's usually been something off with the provider's database | 21:55 |
| Shrews | Something about it on their side gets... confused | 21:55 |
| clarkb | Shrews: caching of the max value not the usage value | 21:56 |
| clarkb | I think we may cache that but not sure | 21:56 |
| corvus | i think the openstack nodepool driver caches it for 5 minutes | 21:56 |
| corvus | and invalidates the cache on any error | 21:57 |
| Shrews | MAX_QUOTA_AGE = 5 * 60 # How long to keep the quota information cached | 21:57 |
| Shrews | yeah | 21:57 |
| corvus | so if it hasn't fixed itself since the beginning of this conversation, something else is up :) | 21:58 |
| fungi | i'll reevaluate assumptions | 22:02 |
| fungi | at 2020-01-08 20:59:18,113 nodepool got node request 300-0006788831 for the debian-buster-arm64 label | 22:04 |
| fungi | nl03 is still trying to satisfy it | 22:05 |
| fungi | as of 22:04:58,508 | 22:05 |
| fungi | Retrying node request 300-0006788831 | 22:06 |
| fungi | Predicted remaining provider quota: {'compute': {'ram': 58272, 'cores': -16, 'instances': 91}} | 22:06 |
| fungi | Not enough quota remaining to satisfy request 300-0006788831 | 22:06 |
| fungi | i guess it doesn't report its presumed max values | 22:06 |
| fungi | but `openstack limits show --absolute` for that project continues to report totalCoresUsed=80 and maxTotalCores=128 | 22:07 |
| openstackgerrit | Brian Haley proposed openstack/project-config master: Add periodic panel to Octavia dashboard https://review.opendev.org/701596 | 22:08 |
| fungi | so there's 48 cores unaccounted for | 22:08 |
| *** eharney has joined #openstack-infra | 22:08 | |
| fungi | also `openstack server list` gives me the same instances nodepool knows about, there are no extras showing up | 22:09 |
| *** kjackal has quit IRC | 22:09 | |
| *** slaweq has quit IRC | 22:15 | |
| *** armax has quit IRC | 22:33 | |
| openstackgerrit | James E. Blair proposed opendev/zone-zuul-ci.org master: Also delegate git. acme challenge to opendev https://review.opendev.org/701633 | 22:40 |
| corvus | clarkb, fungi: ^ | 22:40 |
| corvus | happily, acme.sh does actually produce a useful error message in that case :) | 22:40 |
| fungi | ooh! how did i miss that in reviewing the other half? | 22:41 |
| *** Goneri has joined #openstack-infra | 22:52 | |
| *** tkajinam has joined #openstack-infra | 22:56 | |
| *** tkajinam has quit IRC | 22:58 | |
| *** tkajinam has joined #openstack-infra | 23:00 | |
| openstackgerrit | Merged opendev/zone-zuul-ci.org master: Also delegate git. acme challenge to opendev https://review.opendev.org/701633 | 23:00 |
| *** slaweq has joined #openstack-infra | 23:11 | |
| *** rcernin has joined #openstack-infra | 23:13 | |
| *** slaweq has quit IRC | 23:16 | |
| *** Goneri has quit IRC | 23:16 | |
| *** armax has joined #openstack-infra | 23:19 | |
| *** eharney has quit IRC | 23:19 | |
| *** rlandy is now known as rlandy|bbl | 23:23 | |
| *** aedc_ has quit IRC | 23:40 | |
| *** dchen has joined #openstack-infra | 23:46 | |
| *** tetsuro_ has joined #openstack-infra | 23:50 | |
| *** tetsuro has quit IRC | 23:52 | |
| *** Goneri has joined #openstack-infra | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!