| *** jamesmcarthur has joined #openstack-infra | 00:00 | |
| clarkb | hrm there is a refelect in that so ya maybe its not type checked enough at build and its looking it up and failing | 00:00 |
|---|---|---|
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add mirror01.mtl01.opendev.org https://review.opendev.org/690771 | 00:00 |
| mordred | it's a bug | 00:02 |
| mordred | if you're doing this as an unlogged in user | 00:02 |
| mordred | the bug was fixed in other places | 00:02 |
| ianw | mordred: tfr -- mind checking in on the prior one too https://review.opendev.org/#/c/673724/1 ... really does the same thing, but makes the opensafs client package install explicitly first | 00:05 |
| ianw | s/first/second | 00:05 |
| *** apetrich has quit IRC | 00:06 | |
| *** dciabrin_ has quit IRC | 00:06 | |
| *** armax has quit IRC | 00:09 | |
| *** rlandy has quit IRC | 00:10 | |
| *** rfolco|rover has joined #openstack-infra | 00:10 | |
| *** gyee has quit IRC | 00:11 | |
| mordred | clarkb: https://github.com/go-gitea/gitea/pull/8653 | 00:12 |
| mordred | clarkb: I'm thinking maybe let's revert ... | 00:12 |
| mordred | OR - we could change the playbook to try the "check for root user" check to send the root user auth ... the code path for 404 would still work right, and if the root user exists, we should be able to authenticate as it | 00:13 |
| *** otsukahy has joined #openstack-infra | 00:14 | |
| mordred | of course - if we send auth and the user doesn't exist - we're likely to get a different code than 404 | 00:14 |
| *** jamesmcarthur has quit IRC | 00:16 | |
| mordred | nope. that just results in ctx.User being nil | 00:16 |
| *** dciabrin has joined #openstack-infra | 00:16 | |
| *** jamesmcarthur has joined #openstack-infra | 00:17 | |
| *** yamamoto has quit IRC | 00:19 | |
| *** otsukahy has quit IRC | 00:19 | |
| *** prometheanfire has quit IRC | 00:20 | |
| clarkb | meaning it sendsback a 404? | 00:20 |
| *** prometheanfire has joined #openstack-infra | 00:20 | |
| clarkb | I'm happy with a revert if that us the easiest thibg | 00:20 |
| mordred | I think a revert is easiest for now | 00:20 |
| mordred | and we can try again when that pr lands | 00:20 |
| mordred | if you send a user:pass as a non-existent user, you still get the 500 | 00:21 |
| *** markvoelker has joined #openstack-infra | 00:21 | |
| mordred | assuming I curl'd right: curl --basic -u mordred:foo -k -H https://localhost:3000/api/v1/users/root | 00:21 |
| clarkb | gotit | 00:21 |
| mordred | (ignore the -H) | 00:22 |
| *** jamesmcarthur_ has joined #openstack-infra | 00:26 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Revert "Update gitea to 1.9.4" https://review.opendev.org/690772 | 00:26 |
| mordred | clarkb, corvus, ianw: ^^ | 00:26 |
| *** jamesmcarthur has quit IRC | 00:26 | |
| mordred | incidentally, our testing didn't catch this because our testing doesn't run the role twice - so we dont run through the "root user exists" codepath | 00:26 |
| *** jamesmcarthur_ has quit IRC | 00:29 | |
| *** jamesmcarthur has joined #openstack-infra | 00:30 | |
| *** jamesmcarthur has quit IRC | 00:30 | |
| *** jamesmcarthur has joined #openstack-infra | 00:32 | |
| *** zhurong has quit IRC | 00:36 | |
| openstackgerrit | Merged opendev/zone-opendev.org master: Add mirror01.mtl01.inap.opendev.org https://review.opendev.org/690767 | 00:40 |
| *** jamesmcarthur has quit IRC | 00:40 | |
| *** jamesmcarthur has joined #openstack-infra | 00:40 | |
| *** otsukahy has joined #openstack-infra | 00:52 | |
| *** otsukahy has quit IRC | 00:57 | |
| *** weifan has joined #openstack-infra | 01:00 | |
| *** mriedem_afk is now known as mriedem | 01:02 | |
| *** mriedem has quit IRC | 01:02 | |
| openstackgerrit | Merged opendev/system-config master: openafs: avoid pulling in client package before kernel modules https://review.opendev.org/673724 | 01:03 |
| openstackgerrit | Merged opendev/system-config master: base-server: disable install of suggests and recommends packages https://review.opendev.org/673739 | 01:03 |
| *** weifan has quit IRC | 01:14 | |
| *** xek_ has quit IRC | 01:21 | |
| openstackgerrit | Julia Kreger proposed openstack/diskimage-builder master: Enable IPv6 DHCPv6 https://review.opendev.org/690777 | 01:23 |
| *** markvoelker has quit IRC | 01:28 | |
| *** derekh has quit IRC | 01:51 | |
| *** derekh has joined #openstack-infra | 01:51 | |
| *** spsurya has joined #openstack-infra | 01:59 | |
| *** markvoelker has joined #openstack-infra | 02:00 | |
| *** anteaya has quit IRC | 02:04 | |
| *** markvoelker has quit IRC | 02:05 | |
| *** ricolin has joined #openstack-infra | 02:12 | |
| *** tinwood has quit IRC | 02:18 | |
| *** tinwood has joined #openstack-infra | 02:20 | |
| openstackgerrit | Merged openstack/project-config master: Update ceph grafana dashboard for nova master branch https://review.opendev.org/690466 | 02:24 |
| *** jamesmcarthur has quit IRC | 02:25 | |
| *** jamesmcarthur has joined #openstack-infra | 02:30 | |
| openstackgerrit | Merged opendev/system-config master: Add mirror01.mtl01.opendev.org https://review.opendev.org/690771 | 02:41 |
| *** xinranwang has joined #openstack-infra | 03:04 | |
| *** jamesmcarthur has quit IRC | 03:10 | |
| *** jamesmcarthur has joined #openstack-infra | 03:11 | |
| *** ykarel has joined #openstack-infra | 03:11 | |
| *** jamesmcarthur has quit IRC | 03:15 | |
| *** diablo_rojo has quit IRC | 03:18 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Enable inap-mtl01 https://review.opendev.org/690714 | 03:28 |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Use opendev.org mirror for mtl01.inap region https://review.opendev.org/690788 | 03:28 |
| *** psachin has joined #openstack-infra | 03:39 | |
| ianw | clarkb: ^ ready to go i'd say | 03:39 |
| *** jamesmcarthur has joined #openstack-infra | 03:42 | |
| *** derekh has quit IRC | 03:42 | |
| *** derekh has joined #openstack-infra | 03:43 | |
| clarkb | +2'd but I cant watch it asit is nearbedtime | 03:48 |
| *** ykarel has quit IRC | 03:54 | |
| ianw | haha at this time i expected it to be a "for tomorrow" message anyway :) | 04:00 |
| *** lathiat has joined #openstack-infra | 04:01 | |
| *** yamamoto has joined #openstack-infra | 04:05 | |
| adriant | AJaeger: hey, I've just run into a weird issue between busybox/GNU find because of some old bash you added to Aodh tests in this commit: https://github.com/openstack/aodh/commit/faada69444988c66a09ade812e10fa2ab5a0adae | 04:09 |
| *** yamamoto has quit IRC | 04:10 | |
| adriant | AJaeger: is it safe to just assume whatever this was solving isn't an issue anymore and just remove the line from the tox.ini file: https://github.com/openstack/aodh/blob/master/tox.ini#L38 ? | 04:10 |
| *** ykarel has joined #openstack-infra | 04:15 | |
| *** jerryz has quit IRC | 04:20 | |
| *** jamesmcarthur has quit IRC | 04:27 | |
| *** jamesmcarthur has joined #openstack-infra | 04:31 | |
| *** udesale has joined #openstack-infra | 04:38 | |
| *** jamesmcarthur has quit IRC | 04:39 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] test arm64 mirror https://review.opendev.org/690798 | 04:44 |
| *** kjackal has joined #openstack-infra | 04:48 | |
| *** kjackal has quit IRC | 04:49 | |
| *** yamamoto has joined #openstack-infra | 04:50 | |
| *** jamesmcarthur has joined #openstack-infra | 04:52 | |
| *** tkajinam_ has joined #openstack-infra | 05:03 | |
| *** efried has quit IRC | 05:04 | |
| *** tkajinam has quit IRC | 05:05 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] test arm64 mirror https://review.opendev.org/690798 | 05:10 |
| *** efried has joined #openstack-infra | 05:11 | |
| *** slaweq has joined #openstack-infra | 05:13 | |
| *** jamesmcarthur has quit IRC | 05:13 | |
| *** surpatil has joined #openstack-infra | 05:18 | |
| *** lmiccini has joined #openstack-infra | 05:18 | |
| *** jtomasek has joined #openstack-infra | 05:18 | |
| *** jamesmcarthur has joined #openstack-infra | 05:20 | |
| *** slaweq has quit IRC | 05:29 | |
| *** jamesmcarthur has quit IRC | 05:30 | |
| openstackgerrit | Merged openstack/project-config master: Remove fedora-28 nodes https://review.opendev.org/690432 | 05:31 |
| lennyb | frickler: Is there any good instructions how to create new/custom cirros image for openstack? | 05:45 |
| *** tkajinam__ has joined #openstack-infra | 05:52 | |
| *** tkajinam_ has quit IRC | 05:54 | |
| AJaeger | adriant: yes, remove it. We test that now in infra directly | 05:57 |
| *** psachin has quit IRC | 05:58 | |
| *** dpawlik has joined #openstack-infra | 05:59 | |
| *** udesale has quit IRC | 06:01 | |
| *** markvoelker has joined #openstack-infra | 06:02 | |
| *** psachin has joined #openstack-infra | 06:02 | |
| *** dpawlik has quit IRC | 06:03 | |
| *** markvoelker has quit IRC | 06:06 | |
| *** udesale has joined #openstack-infra | 06:10 | |
| openstackgerrit | Merged openstack/project-config master: Use opendev.org mirror for mtl01.inap region https://review.opendev.org/690788 | 06:13 |
| mordred | clarkb, corvus: the gitea patch has been merged and lunny suggested backporting it to 1.9 and 1.10 - so hopefully it'll be in a release soon. maybe if it lands in release/v1.9 we can build an image from that branch until there's either a 1.9 point release with it, or a 1.10 release with it | 06:15 |
| *** dpawlik has joined #openstack-infra | 06:15 | |
| *** ykarel is now known as ykarel|afk | 06:15 | |
| *** dpawlik has quit IRC | 06:20 | |
| *** dpawlik has joined #openstack-infra | 06:23 | |
| *** slaweq has joined #openstack-infra | 06:26 | |
| *** surpatil has quit IRC | 06:30 | |
| *** iurygregory has joined #openstack-infra | 06:34 | |
| *** tkajinam__ is now known as tkajinam | 06:34 | |
| *** pkopec has joined #openstack-infra | 06:35 | |
| *** zzehring has quit IRC | 06:40 | |
| *** pcaruana has joined #openstack-infra | 06:41 | |
| *** aedc has quit IRC | 06:54 | |
| *** jaicaa has quit IRC | 07:02 | |
| *** jaicaa has joined #openstack-infra | 07:05 | |
| *** tosky has joined #openstack-infra | 07:12 | |
| *** surpatil has joined #openstack-infra | 07:12 | |
| *** kopecmartin|off is now known as kopecmartin | 07:13 | |
| *** tesseract has joined #openstack-infra | 07:18 | |
| *** ccamacho has joined #openstack-infra | 07:29 | |
| *** ykarel|afk is now known as ykarel | 07:31 | |
| openstackgerrit | Merged opendev/system-config master: Revert "Update gitea to 1.9.4" https://review.opendev.org/690772 | 07:32 |
| *** xinranwang has quit IRC | 07:33 | |
| *** gfidente has joined #openstack-infra | 07:34 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Revert "Revert "Update gitea to 1.9.4"" https://review.opendev.org/690872 | 07:34 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update gitea to v1.10 https://review.opendev.org/690873 | 07:34 |
| *** surpatil is now known as surpatil|lunch | 07:36 | |
| *** trident has quit IRC | 07:46 | |
| *** trident has joined #openstack-infra | 07:51 | |
| *** rpittau|afk is now known as rpittau | 07:52 | |
| *** apetrich has joined #openstack-infra | 07:55 | |
| *** apetrich has quit IRC | 07:57 | |
| *** apetrich has joined #openstack-infra | 08:00 | |
| *** kjackal has joined #openstack-infra | 08:02 | |
| *** amoralej|off is now known as amoralej | 08:04 | |
| *** lucasagomes has joined #openstack-infra | 08:05 | |
| *** ralonsoh has joined #openstack-infra | 08:07 | |
| *** dchen has quit IRC | 08:11 | |
| *** aedc has joined #openstack-infra | 08:12 | |
| *** kaisers has quit IRC | 08:13 | |
| openstackgerrit | Monty Taylor proposed zuul/nodepool master: Handle newer nova microversions https://review.opendev.org/690876 | 08:19 |
| *** tkajinam has quit IRC | 08:19 | |
| *** xek_ has joined #openstack-infra | 08:29 | |
| *** ykarel is now known as ykarel|lunch | 08:31 | |
| *** hwoarang has quit IRC | 08:35 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Revert "Revert "Update gitea to 1.9.4"" https://review.opendev.org/690872 | 08:36 |
| *** hwoarang has joined #openstack-infra | 08:36 | |
| *** hwoarang has quit IRC | 08:40 | |
| *** hwoarang has joined #openstack-infra | 08:48 | |
| *** xek_ has quit IRC | 09:02 | |
| *** surpatil|lunch is now known as surpatil | 09:11 | |
| *** jaosorior has joined #openstack-infra | 09:15 | |
| *** yamamoto has quit IRC | 09:16 | |
| *** yamamoto has joined #openstack-infra | 09:22 | |
| *** yamamoto has quit IRC | 09:22 | |
| *** yamamoto has joined #openstack-infra | 09:22 | |
| *** ykarel|lunch is now known as ykarel | 09:25 | |
| *** yamamoto has quit IRC | 09:26 | |
| *** e0ne has joined #openstack-infra | 09:33 | |
| openstackgerrit | Ivan Kolodyazhny proposed openstack/openstack-zuul-jobs master: Add openstack-tox-pep8 to openstack-python3-ussuri-jobs template. https://review.opendev.org/688997 | 09:54 |
| *** markvoelker has joined #openstack-infra | 10:04 | |
| *** aedc has quit IRC | 10:06 | |
| *** markvoelker has quit IRC | 10:09 | |
| *** udesale has quit IRC | 10:19 | |
| *** dpawlik has quit IRC | 10:19 | |
| *** yamamoto has joined #openstack-infra | 10:26 | |
| *** yamamoto has quit IRC | 10:30 | |
| *** otsukahy has joined #openstack-infra | 10:30 | |
| *** ykarel_ has joined #openstack-infra | 10:40 | |
| openstackgerrit | Witold Bedyk proposed openstack/project-config master: Grant permissions for Monasca UI to horizon-core https://review.opendev.org/690918 | 10:42 |
| *** yamamoto has joined #openstack-infra | 10:42 | |
| *** ykarel has quit IRC | 10:43 | |
| *** udesale has joined #openstack-infra | 10:44 | |
| *** dpawlik has joined #openstack-infra | 10:44 | |
| *** ykarel_ is now known as ykarel | 10:45 | |
| *** yamamoto has quit IRC | 10:54 | |
| *** yamamoto has joined #openstack-infra | 10:57 | |
| *** amoralej is now known as amoralej|lunch | 11:00 | |
| *** roman_g has joined #openstack-infra | 11:06 | |
| *** prometheanfire has quit IRC | 11:09 | |
| noonedeadpunk | hi:) I have kinda question regarding new upper-constraints url. the thing is, that we're using releases SHA instead of branch name to be sure about requirements used with one particular osa release. | 11:09 |
| noonedeadpunk | but, new url https://releases.openstack.org/constraints/upper/ don't understand providing it SHA | 11:10 |
| *** prometheanfire has joined #openstack-infra | 11:11 | |
| openstackgerrit | Witold Bedyk proposed openstack/project-config master: Grant permissions for Monasca UI to horizon-core https://review.opendev.org/690918 | 11:11 |
| noonedeadpunk | ofc we can use https://opendev.org/openstack/requirements/raw/5d2dc701ab2fe7e87d78e4d96eb444766eef6725/upper-constraints.txt but to be consistent with everyone, maybe it's possible to add redirect? | 11:11 |
| *** yamamoto has quit IRC | 11:12 | |
| noonedeadpunk | I just didn't found anything relevant in https://opendev.org/opendev/system-config so maybe you can point me to where these redirects are? | 11:12 |
| noonedeadpunk | oh, ok found it:) | 11:13 |
| *** otsukahy has quit IRC | 11:22 | |
| *** aedc has joined #openstack-infra | 11:27 | |
| AJaeger | noonedeadpunk: the upper-constrains URL comes from release team, best to talk with them directly | 11:28 |
| openstackgerrit | Ivan Kolodyazhny proposed openstack/openstack-zuul-jobs master: Add openstack-tox-pep8 to openstack-python3-ussuri-jobs template. https://review.opendev.org/688997 | 11:28 |
| *** roman_g has quit IRC | 11:35 | |
| openstackgerrit | Ivan Kolodyazhny proposed openstack/openstack-zuul-jobs master: Add openstack-tox-pep8 to openstack-python3-ussuri-jobs template. https://review.opendev.org/688997 | 11:36 |
| *** derekh has quit IRC | 11:37 | |
| noonedeadpunk | AJaeger: oh, I see, thanks | 11:42 |
| *** janki has joined #openstack-infra | 11:44 | |
| openstackgerrit | Witold Bedyk proposed openstack/project-config master: Grant permissions for Monasca UI to horizon-core https://review.opendev.org/690918 | 11:46 |
| *** yamamoto has joined #openstack-infra | 11:46 | |
| *** larainema has joined #openstack-infra | 11:48 | |
| *** yamamoto has quit IRC | 11:54 | |
| mtreinish | rm_work: I do not have access, I was given write access to the github repo a few months ago but haven't been added on pypi for anything yet | 11:59 |
| *** amoralej|lunch is now known as amoralej | 12:01 | |
| *** jamesmcarthur has joined #openstack-infra | 12:11 | |
| noonedeadpunk | cores, can you kindly review our initial os_murano stuff? https://review.opendev.org/#/q/topic:os_murano+status:open | 12:12 |
| noonedeadpunk | We've announced it for train, and I guess we need to backport this when it'll be merged | 12:12 |
| *** roman_g has joined #openstack-infra | 12:13 | |
| *** ykarel is now known as ykarel|afk | 12:15 | |
| *** xek_ has joined #openstack-infra | 12:16 | |
| *** psachin has quit IRC | 12:18 | |
| *** rlandy has joined #openstack-infra | 12:18 | |
| *** aedc has quit IRC | 12:21 | |
| *** jamesmcarthur has quit IRC | 12:28 | |
| *** yamamoto has joined #openstack-infra | 12:28 | |
| noonedeadpunk | And we also need to unfreeze https://review.opendev.org/#/c/686574/ | 12:30 |
| *** markvoelker has joined #openstack-infra | 12:31 | |
| *** psachin has joined #openstack-infra | 12:32 | |
| *** yamamoto has quit IRC | 12:35 | |
| *** kjackal has quit IRC | 12:39 | |
| *** jaosorior has quit IRC | 12:42 | |
| *** kjackal has joined #openstack-infra | 12:43 | |
| AJaeger | noonedeadpunk: are you sure that you're in the right channel? | 12:44 |
| noonedeadpunk | wow, no, I'm not :( | 12:44 |
| noonedeadpunk | sorry:( | 12:44 |
| roman_g | Hello team. I was checking documentation here https://docs.openstack.org/infra/manual/creators.html#add-gerrit-permissions , and couldn't find a way to add rules.pl file to the project's refs/meta/config branch. Need to set up some additional verification for voting. Also couldn't find any usage of rules.pl file in OpenStack repositories (may be my search skills are not so good). Could you guide | 12:45 |
| roman_g | me where to look for to get rules.pl file added to refs/meta/config branch and merged? | 12:45 |
| AJaeger | roman_g: rules.pl? What is that? I doubt we use that at all... | 12:46 |
| roman_g | To be more specific, need to enforce 2x Code-Review +2 from non-authors to be able to merge change. | 12:47 |
| AJaeger | roman_g: all config is in project-config/gerrit/acls/ - and that's what we support | 12:47 |
| roman_g | This one: https://gerrit-review.googlesource.com/Documentation/config-project-config.html#file-rules_pl | 12:47 |
| AJaeger | roman_g: we handle that as social contract, not by tool support currently. | 12:47 |
| roman_g | AJaeger: I know. | 12:47 |
| roman_g | File needs to be added here: https://review.opendev.org/gitweb?p=airship/docs.git;a=tree;h=refs/meta/config;hb=refs/meta/config | 12:48 |
| AJaeger | roman_g: so, I guess it needs extensive rework - including work in jeepyb. | 12:48 |
| AJaeger | roman_g: best discuss with rest of infra team later (once US west coast is awake) | 12:48 |
| *** jamesmcarthur has joined #openstack-infra | 12:48 | |
| roman_g | OK. Thank you, AJaeger. | 12:48 |
| roman_g | I hope it would just need some minor 'OpenDev Project Creator' jobs modification. | 12:49 |
| *** iokiwi has quit IRC | 12:51 | |
| *** adriant has quit IRC | 12:51 | |
| *** adriant has joined #openstack-infra | 12:52 | |
| openstackgerrit | Ivan Kolodyazhny proposed openstack/openstack-zuul-jobs master: Add openstack-tox-pep8 to openstack-python3-ussuri-jobs template. https://review.opendev.org/688997 | 12:53 |
| fungi | roman_g: we haven't supported prolog scripting for project labels up to now, and no the project creation isn't done via any ci job, it's done with custom python utilities in a the opendev/jeepyb project, triggered by configuration management | 12:53 |
| fungi | just to be clear, you're saying you're entrusting approval rights to people who can't be trusted to not approve changes nobody else has reviewed? | 12:54 |
| fungi | and also you're sure you'll never need to approve an emergency fix when there's only one core reviewer around? | 12:54 |
| fungi | an alternative (which we've used in a variety of governance-related projects for example) might be to restrict the workflow +1 vote to a different group than the code-review +2 vote | 12:57 |
| *** surpatil has quit IRC | 12:57 | |
| fungi | the script which updates refs/meta/config in gerrit is this: https://opendev.org/opendev/jeepyb/src/branch/master/jeepyb/cmd/manage_projects.py | 13:00 |
| *** goldyfruit has joined #openstack-infra | 13:00 | |
| roman_g | fungi: we have unexperienced developers and need to ensure they don't rush to merge everything they have just wrote. And we want Gerrit to enforce rule (we have seen breaches). | 13:01 |
| *** ykarel|afk is now known as ykarel | 13:02 | |
| fungi | i see. another alternative is to not grant workflow +1 (or code-review +2) access for inexperienced developers | 13:02 |
| *** eharney has joined #openstack-infra | 13:05 | |
| *** yamamoto has joined #openstack-infra | 13:10 | |
| *** jamesmcarthur has quit IRC | 13:13 | |
| *** janki has quit IRC | 13:13 | |
| *** otsukahy has joined #openstack-infra | 13:19 | |
| *** jamesmcarthur has joined #openstack-infra | 13:20 | |
| *** mriedem has joined #openstack-infra | 13:23 | |
| *** onovy has quit IRC | 13:25 | |
| *** roman_g has quit IRC | 13:26 | |
| *** psachin has quit IRC | 13:28 | |
| *** ykarel is now known as ykarel|afk | 13:29 | |
| *** spsurya has quit IRC | 13:29 | |
| *** rh-jelabarre has joined #openstack-infra | 13:30 | |
| *** yamamoto has quit IRC | 13:30 | |
| *** zzehring has joined #openstack-infra | 13:30 | |
| *** yamamoto has joined #openstack-infra | 13:31 | |
| *** yamamoto has quit IRC | 13:31 | |
| *** slaweq has quit IRC | 13:32 | |
| *** otsukahy has quit IRC | 13:34 | |
| *** e0ne has quit IRC | 13:35 | |
| *** e0ne has joined #openstack-infra | 13:36 | |
| *** onovy has joined #openstack-infra | 13:37 | |
| *** sreejithp has joined #openstack-infra | 13:39 | |
| *** sgw has joined #openstack-infra | 13:43 | |
| *** ykarel|afk is now known as ykarel | 13:48 | |
| *** amoralej is now known as amoralej|mtg | 14:00 | |
| openstackgerrit | Matt Riedemann proposed openstack/project-config master: Update ceph grafana dashboard for cinder master branch https://review.opendev.org/690980 | 14:01 |
| *** yamamoto has joined #openstack-infra | 14:03 | |
| *** jpena|off is now known as jpena | 14:05 | |
| *** jamesmcarthur has quit IRC | 14:06 | |
| *** jpena is now known as jpena|off | 14:07 | |
| *** dpawlik has quit IRC | 14:11 | |
| openstackgerrit | Ivan Kolodyazhny proposed openstack/openstack-zuul-jobs master: Update openstack-python3-ussuri-jobs-* templates to test with Python3 https://review.opendev.org/688997 | 14:12 |
| *** yamamoto has quit IRC | 14:14 | |
| *** jamesmcarthur has joined #openstack-infra | 14:17 | |
| *** ykarel has quit IRC | 14:18 | |
| *** pcaruana has quit IRC | 14:19 | |
| *** ykarel has joined #openstack-infra | 14:20 | |
| mriedem | clarkb: i've been seeing http://status.openstack.org/elastic-recheck/#1808010 a lot this week. thinking out loud, https://review.opendev.org/#/c/619319/ might have introduced that when we started setting the non-0 disk size on the tempest flavors in devstack, | 14:21 |
| mriedem | since we're using tiny cirros images, the root disk on the flavor is just 1GiB, | 14:21 |
| mriedem | before that it was disk=0 so the root disk was whatever was needed for the size of the image at spawn, | 14:21 |
| mriedem | i'm wondering if we should bump those tempest flavors up 1GiB to account for extra space needed, like the config drive | 14:22 |
| mriedem | keep seeing that growroot failure | 14:22 |
| *** larainema has quit IRC | 14:25 | |
| *** pkopec has quit IRC | 14:28 | |
| clarkb | mriedem: config drive is a separate device right? the flavor probably isnt impacted by that. However adding another half a gig probably wont hurt | 14:29 |
| *** xek_ has quit IRC | 14:30 | |
| *** xek_ has joined #openstack-infra | 14:30 | |
| mriedem | yeah that's true | 14:31 |
| openstackgerrit | Ivan Kolodyazhny proposed openstack/openstack-zuul-jobs master: Update openstack-python3-ussuri-jobs-* templates to test without Python3 https://review.opendev.org/688997 | 14:32 |
| ttx | AJaeger: you were looking into retiring old, now-unofficial repositories still present in the opensatck/ namespace (think Fuel). I think we can proceed, now that Train is behind us... Let me know if I can be of any help there | 14:35 |
| *** bnemec has quit IRC | 14:35 | |
| *** pkopec has joined #openstack-infra | 14:35 | |
| *** bnemec has joined #openstack-infra | 14:37 | |
| *** pkopec has quit IRC | 14:40 | |
| fungi | yeah, it mainly needs to be brought up with the tc to make sure it's something they want to do at this point, but their self-imposed window to opt out of tc jurisdiction over deliverables remaining in the openstack namespace since the opendev shuffle has now expired with the train release behind them | 14:43 |
| *** kopecmartin is now known as kopecmartin|off | 14:46 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Test buildset registry with k8s and docker https://review.opendev.org/689280 | 14:47 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Adjust buildset registry container name https://review.opendev.org/690992 | 14:47 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Don't sudo when saving container logs in registry test https://review.opendev.org/690993 | 14:47 |
| *** evrardjp_ is now known as evrardjp | 14:48 | |
| *** amoralej|mtg is now known as amoralej | 14:49 | |
| AJaeger | ttx, I only mentioned fuel and I reached out to SergeyLukjanov and there are some internal uses of the repos but those should "soon" move away. Slow discussion going on there... Let's wait a bit | 14:49 |
| fungi | the tc could choose to retire them now, but could also choose to give them a little more time to stop depending on those copies of their repositories | 14:51 |
| *** mattw4 has joined #openstack-infra | 14:52 | |
| fungi | but also, fuel is just one example (though perhaps the largest example) of former openstack deliverables which did not get their repos retired | 14:52 |
| *** armax has joined #openstack-infra | 14:52 | |
| ttx | AJaeger: I ran an analysis recently and here was the gap between official and openstack/ : https://etherpad.openstack.org/p/MGNTt7YkZ5 | 14:52 |
| *** xek_ has quit IRC | 14:53 | |
| *** xek_ has joined #openstack-infra | 14:53 | |
| *** roman_g has joined #openstack-infra | 14:55 | |
| *** jcoufal has joined #openstack-infra | 14:58 | |
| *** ccamacho has quit IRC | 14:59 | |
| *** Goneri has joined #openstack-infra | 15:01 | |
| *** gouthamr_ is now known as gouthamr | 15:01 | |
| ttx | looks like there are no surprises there | 15:02 |
| fungi | it's a lot less to clean up than i would have expected, really | 15:03 |
| *** yolanda has quit IRC | 15:03 | |
| *** mattw4 has quit IRC | 15:07 | |
| *** priteau has joined #openstack-infra | 15:09 | |
| *** slaweq has joined #openstack-infra | 15:10 | |
| *** yolanda has joined #openstack-infra | 15:11 | |
| *** jaosorior has joined #openstack-infra | 15:13 | |
| *** jaosorior has quit IRC | 15:16 | |
| clarkb | roman_g: fungi ya the current setup is intended to be rigid enough that we have some checks in place while still being flexible enough to give people what they need to address emergencies without involving gerrit admins | 15:18 |
| rosmaita | fungi: who should i talk to about launchpad administration? i'm looking to get the Maintainer for os-brick changed from "OpenStack Administrators" to "Cinder Drivers" | 15:18 |
| fungi | rosmaita: i think as long as cinder drivers is owned by openstack administrators then that's fine (we have that situation on some other projects too) | 15:20 |
| fungi | that at least allows someone in openstack administrators to add themselves temporarily to cinder drivers to check/fix configuration problems with the lp project | 15:20 |
| rosmaita | cinder drivers already maintains cinder & python-cinderclient, so i imagine that relationship is set up correctly | 15:21 |
| roman_g | clarkb: I see. Thank you. | 15:21 |
| clarkb | fungi: roman_g also I think if we wanted to add prolog rules we should add them globally then projects can opt into those functions in their acls as per usual | 15:21 |
| clarkb | if we do it globally we may not need to hvae jeepyb manage it | 15:22 |
| rosmaita | fungi: i just checked, cinder drivers is owned by OS Administrators | 15:22 |
| roman_g | fungi: clarkb would a patch to https://opendev.org/opendev/jeepyb/src/branch/master/jeepyb/cmd/manage_projects.py to support rules.pl be a good idea? | 15:22 |
| clarkb | (I think you can drop prolog files into review_site somewhere for them to be picked up globally) | 15:22 |
| fungi | rosmaita: i can fix os-brick in a few minutes then | 15:23 |
| fungi | thanks for checking! | 15:23 |
| rosmaita | fungi: no rush, thanks in advance | 15:23 |
| clarkb | roman_g: as I just mentioned I think my preference would be for them to be added as global rules that projects can opt into using via acl config rather than project specific rules. I believe that if we do it that way we can simply drop the prolog files into the review_site config and don't need to update tools | 15:23 |
| fungi | clarkb: ahh, so they don't have to be pushed directly into refs/meta/config? neat | 15:23 |
| clarkb | fungi: from memory no | 15:23 |
| fungi | very cool. so it just comes down to whether that's a feature the opendev sysadmins feel comfortable supporting | 15:24 |
| roman_g | clarkb: if we create global rules, then there is no way to enable/disable them per repo. Rules are enforced from top. | 15:25 |
| *** slaweq has quit IRC | 15:25 | |
| clarkb | roman_g: you add them as functions in your acl config | 15:25 |
| clarkb | the currnet function we use is a built in one which requires highest vote value at least once and no lowest vote values for submit | 15:26 |
| clarkb | you would simply say "use this other function from the prolog that is loaded" I believe that bit works the same if the prolog is project specific or global | 15:26 |
| * clarkb is trying to find docs on it though | 15:26 | |
| clarkb | aha the "global" config goes in All-Projects refs/meta/config | 15:29 |
| roman_g | https://gerrit-review.googlesource.com/Documentation/config-labels.html#label_function - this one? | 15:30 |
| clarkb | so the mechanism is the same either way, more a matter of scoping via destination | 15:30 |
| clarkb | roman_g: ya though that says the name is ignored so maybe I've misunderstood how you apply prolog to projects | 15:31 |
| clarkb | that is a bit disappointing that it is implicit and the explicit config is ignored | 15:32 |
| clarkb | we could make global config but then it would apply globally I guess and we can't name custom filters? | 15:34 |
| roman_g | that's what I thought | 15:35 |
| roman_g | I would just add directory rules/ here: https://opendev.org/openstack/project-config/src/branch/master/gerrit | 15:36 |
| roman_g | with per-project subdirs | 15:36 |
| roman_g | like there are for acl's | 15:36 |
| roman_g | and just keep rules.pl there | 15:37 |
| *** otsukahy has joined #openstack-infra | 15:38 | |
| *** gyee has joined #openstack-infra | 15:38 | |
| clarkb | that is the easy part :) the files can go just about anywhere. We'd need to map them into gerrit (existing tool that does this is manage-projects for acls) | 15:39 |
| melwitt | anybody know why there's no data in https://graphite.opendev.org for example stats_counts.zuul.tenant.openstack.pipeline.check.project.opendev_org.openstack_nova.stable_train.job there's no tempest-integrated-compute job there but it is running in real life, example: https://review.opendev.org/690711 | 15:39 |
| clarkb | we'd also need to ensure our config allows it and that the limits are not too large to cause problems, but big enough that the rules work | 15:39 |
| melwitt | I've been using graphite to know how to add jobs to a grafana dashboard for ceph and have noticed data seems missing for some things http://grafana.openstack.org/d/-iKINcImz/ceph-failure-rate?orgId=1 | 15:40 |
| roman_g | clarkb: exactly, I wanted simply to modify manage_projects.pl so that it copies rules.pl to the right place if it is present | 15:40 |
| roman_g | clarkb: default config allows rules | 15:40 |
| clarkb | roman_g: I get that, I'm pointing out it not "simple" | 15:41 |
| clarkb | roman_g: we also need to update our gerrit config and ensure the limits imposed by the config are correct for our setup | 15:41 |
| clarkb | melwitt: as a sanity check http://zuul.openstack.org/builds?job_name=tempest-integrated-compute&pipeline=check&project=openstack%2Fnova shows results (so one would expect graphite to have data too) | 15:42 |
| clarkb | roman_g: I think we should also consider whether or not this is the best solution to the problem you are trying to address | 15:42 |
| clarkb | roman_g: this was fungi's point earlier | 15:42 |
| clarkb | basically if people shouldn't be approving changes we can remove their approval rights instead | 15:42 |
| melwitt | clarkb: yeah, I don't know what's going on with graphite. it's missing jobs for train. and I don't know you to link you to a path there | 15:43 |
| clarkb | while still having the flexiblity for those that can approv echanges to do so in an emergency when others are traveling or on vacation or otherwise unable to help | 15:43 |
| clarkb | infra-root I am going to approve https://review.opendev.org/#/c/690714/2 to enable inap | 15:44 |
| corvus | clarkb, fungi: fyi, new work to replace prolog: https://docs.google.com/presentation/d/1fUjnmngWLFaexZKlN5XmL9EmOX8Nh_P0yJcThnlJ3Lk/ | 15:44 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Fixes generation of documentation for jobvars https://review.opendev.org/691008 | 15:45 |
| *** iurygregory has quit IRC | 15:46 | |
| clarkb | corvus: looks like those would be written as java and added to gerrit as plugins. Fwiw the consumption of those prolog less rules seems to reflect how I thought prolog rules were implemented. Create a named entity then in your acl file apply that named entity as the submit function | 15:49 |
| clarkb | (for that reason I'm in favor of this change) | 15:49 |
| *** tesseract has quit IRC | 15:49 | |
| clarkb | they list some other very good reasons for the change in the deck too | 15:50 |
| corvus | clarkb: yeah, i think the new model is good. the simple-submit-rules plugin is supposed to be a collection of simple things (so could probably add most things we'd want to that). it only has one rule now. but also, the lua scripting rules plugin is interesting too. | 15:50 |
| corvus | i think we get that with 2.16 | 15:50 |
| corvus | clarkb: i'm going to wait for 690714 to land, then restart zuul | 15:51 |
| clarkb | that said re tests for prolog, prolog is such an easy language to have tests for. I wonder if that is simply a lack of examples | 15:51 |
| fungi | also the switch from prolog to lua will be nice. at least the latter is a little more readable | 15:51 |
| corvus | yeah, and maybe hard to integrate with the java testing framework | 15:51 |
| clarkb | you basically define a set of facts then run your predicates and assert yes or no | 15:52 |
| clarkb | corvus: ok | 15:52 |
| clarkb | fungi: also you don't have to melt your brain to understand the "flow" | 15:52 |
| *** ykarel is now known as ykarel|away | 15:53 | |
| clarkb | roman_g: given ^ and the current work to upgrade gerrit I think a better path forward may be to rely on prolog less rules, then we can do what I described earlier. Have a set of global rules that projects opt into using | 15:53 |
| clarkb | roman_g: then we avoid adding tech debt we won't want in the near future | 15:53 |
| *** goldyfruit has quit IRC | 15:54 | |
| openstackgerrit | Merged zuul/nodepool master: Handle newer nova microversions https://review.opendev.org/690876 | 15:54 |
| *** goldyfruit has joined #openstack-infra | 15:56 | |
| *** diablo_rojo has joined #openstack-infra | 15:56 | |
| clarkb | melwitt: if you use the graphite.opendev.org browser thing on the right that job doesn't show up at all under that path | 15:58 |
| clarkb | and yet it is there at http://zuul.openstack.org/builds?job_name=tempest-integrated-compute&pipeline=check&project=openstack%2Fnova&branch=stable%2Ftrain | 15:59 |
| clarkb | /dev/mapper/main-graphite 1008G 1008G 0 100% /var/lib/graphite/storage | 15:59 |
| clarkb | I think ^ is the problem | 15:59 |
| *** diablo_rojo has quit IRC | 15:59 | |
| openstackgerrit | Merged openstack/project-config master: Enable inap-mtl01 https://review.opendev.org/690714 | 16:00 |
| sean-k-mooney | fungi: just saw the reference to https://launchpad.net/bugs/1734320 on openstack meeting by the way. | 16:00 |
| openstack | Launchpad bug 1734320 in OpenStack Compute (nova) "Eavesdropping private traffic" [Undecided,In progress] - Assigned to sean mooney (sean-k-mooney) | 16:00 |
| *** diablo_rojo has joined #openstack-infra | 16:00 | |
| melwitt | clarkb: whoa yeah | 16:00 |
| sean-k-mooney | fungi: the status of that has not changed | 16:00 |
| clarkb | corvus: with the switcharoo on garphite "paths" post zuulv3 are there old paths we can delete files for on the grpahite server? | 16:00 |
| clarkb | corvus: I wonder if we're doubling up on storage needs because of the old paths and the new paths coexisting | 16:00 |
| corvus | clarkb: almost certainly; and it's probably been long enough we aren't interested in the old data | 16:01 |
| clarkb | corvus: ++ | 16:01 |
| corvus | probably a find for timestamps is the best approach? | 16:01 |
| corvus | i'm going to restart zuul now | 16:01 |
| clarkb | corvus: that make ssense. Let me see what a find that looks for anything older than a year finds | 16:01 |
| openstackgerrit | Merged zuul/zuul-jobs master: Fixes generation of documentation for jobvars https://review.opendev.org/691008 | 16:01 |
| sean-k-mooney | fungi: at least not since febuary. the nova and neutron patch have been avaliabel and pauses more or less since then. feel free to ping me later if you want more details | 16:02 |
| *** auristor has quit IRC | 16:04 | |
| *** rpittau is now known as rpittau|afk | 16:05 | |
| fungi | sean-k-mooney: no problem, i was mostly pointing out that there's been renewed discussion on that bug report | 16:08 |
| clarkb | this is not a very fast find | 16:08 |
| * clarkb waits patiently | 16:08 | |
| *** mattw4 has joined #openstack-infra | 16:09 | |
| mwhahaha | is zuul.o.o 503ing for anyone else or is it just me | 16:09 |
| openstackgerrit | Ivan Kolodyazhny proposed openstack/openstack-zuul-jobs master: Update openstack-python3-ussuri-jobs-* templates to test without Python2 https://review.opendev.org/688997 | 16:09 |
| corvus | mwhahaha: it's restarting | 16:09 |
| mwhahaha | ah | 16:09 |
| sean-k-mooney | fungi: ah ok. i would like to get the final to patches merged eventullay but m3-release crazyness was a thing so i had to step back form it again. | 16:11 |
| sean-k-mooney | clarkb: in case you did not see my reply thanks for the tip on glean and root login | 16:12 |
| clarkb | sean-k-mooney: I did, glad you got it working. Re ipv6 it should work if your cloud does stateless config | 16:13 |
| fungi | sean-k-mooney: as for ipv6 on centos, yeah that's been a struggle, mostly to do with races between kernel handling and network-manager | 16:13 |
| clarkb | sean-k-mooney: glean doesn't know how to configure static addresses for ipv6 on centos | 16:13 |
| *** dkehn_ has joined #openstack-infra | 16:13 | |
| clarkb | fungi: that is dynamic stateless config specific and we have it working now | 16:13 |
| fungi | yep | 16:14 |
| sean-k-mooney | well my issue seams to be even running dhclient -6 does not work | 16:14 |
| *** e0ne has quit IRC | 16:14 | |
| fungi | sean-k-mooney: is that environment doing stateful or stateless dhcp6? | 16:14 |
| sean-k-mooney | but for my ubuntu 18.04 vms it just works | 16:14 |
| clarkb | sean-k-mooney: hte interface should be network manager managed (because thats the only option on rhel/centos 8 now I guess) | 16:14 |
| sean-k-mooney | stateful dhcp6 | 16:14 |
| clarkb | sean-k-mooney: there is a good chance that you have to do stateful dhcpv6 through NM | 16:14 |
| sean-k-mooney | ya maybe | 16:15 |
| sean-k-mooney | i mean i can alwasys chagne it its just my dev cloud at home | 16:15 |
| sean-k-mooney | but ill see if i can figure it out at the weekend | 16:15 |
| *** mriedem is now known as mriedem_away | 16:16 | |
| sean-k-mooney | have ye had any better look with statels rdavd | 16:16 |
| sean-k-mooney | *stateless | 16:16 |
| *** lmiccini has quit IRC | 16:16 | |
| clarkb | I don't know what NM uses under the covers | 16:16 |
| clarkb | it should be working with whatever NM uses to receive and configure RAs | 16:16 |
| sean-k-mooney | right but when using neutron stateflu dhcp6 i guess RA adversisme of ips is disable | 16:17 |
| *** lucasagomes has quit IRC | 16:17 | |
| corvus | #status log restarted all of zuul on commit 360f1e9fd32e7aa5eec7135838cb5e14f0cac6ae | 16:17 |
| openstackstatus | corvus: finished logging | 16:17 |
| sean-k-mooney | anyway now that i can log in i can play with it | 16:17 |
| *** ianychoi has joined #openstack-infra | 16:21 | |
| *** auristor has joined #openstack-infra | 16:21 | |
| *** jcoufal has quit IRC | 16:25 | |
| fungi | sean-k-mooney: i wonder, with stateful, what would be involved in getting that added to nova's network metadata (so it's available in the config drive)? might be easy to add support for that directly to glean and then not need dhclient to get involved at all | 16:28 |
| clarkb | infra-root can we get reviews on https://review.opendev.org/#/c/689248/ to clarify the state of third party ci docs | 16:29 |
| clarkb | fungi: I believe the info is already in the config drive | 16:29 |
| corvus | clarkb: done | 16:29 |
| clarkb | tyty | 16:30 |
| fungi | clarkb: oh, neat. couldn't glean just treat it the same way it does v4 in that case? | 16:30 |
| sean-k-mooney | fungi: oh so like have glean staticaly configure the static adress ya that could work | 16:31 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Adjust buildset registry container name https://review.opendev.org/690992 | 16:31 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Don't sudo when saving container logs in registry test https://review.opendev.org/690993 | 16:31 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Test buildset registry with k8s and docker https://review.opendev.org/689280 | 16:31 |
| clarkb | fungi: I think glean should configure NM to do dhcpv6 in that case, but it does not currently know how to do that | 16:31 |
| sean-k-mooney | fungi: did i imaging it or is that something that can already be done with ipv4 if you disable dhcp on the network | 16:31 |
| sean-k-mooney | clarkb: i mean that too. the network manager config had dhcp enabled | 16:32 |
| fungi | clarkb: oh, i meant telling nova to incorporate the addresses/routes into the network metadata and then have glean set it as if it were static addressing | 16:32 |
| fungi | rather than actually using dhcp | 16:33 |
| sean-k-mooney | it looks like this http://paste.openstack.org/show/785555/ | 16:33 |
| *** ykarel|away has quit IRC | 16:33 | |
| sean-k-mooney | fungi: it might already be there | 16:33 |
| sean-k-mooney | where would i check? | 16:33 |
| fungi | mount the configdrive and look in the network metadata json | 16:33 |
| clarkb | fungi: I don't think glean should do that. It should configure the network as told by config drive | 16:33 |
| clarkb | fungi: fwiw I think it would work just fine if you told NM to do dhvpcv6 | 16:34 |
| sean-k-mooney | ya its there | 16:34 |
| fungi | clarkb: right, maybe i'm not being clear. i guess i'm asking why stateful dhcp6 is any different there than static addressing | 16:34 |
| clarkb | fungi: bceause the cloud told us to use dhcpv6 | 16:34 |
| sean-k-mooney | http://paste.openstack.org/show/785556/ | 16:34 |
| * fungi tries to rephrase again | 16:35 | |
| fungi | what is the benefit to configuring the cloud to do stateful dhcp6 instead of static assignment, if the vm is capable of ingesting the static configuration anyway | 16:35 |
| sean-k-mooney | "type": "ipv6_dhcpv6-stateful" | 16:35 |
| clarkb | fungi: it is not capable of ingesting the static config | 16:35 |
| fungi | there we go | 16:35 |
| clarkb | fungi: neither static config nor dhcpv6 are implemented on red hat platofrms in glean | 16:36 |
| clarkb | (for ipv6) | 16:36 |
| fungi | so the other part of my question was "what is needed to get the static addressing usable" | 16:36 |
| sean-k-mooney | so the could is telling glean/cloud init to configre dhcpv6 in stateful mode | 16:36 |
| fungi | and if the cloud were set for static v6, couldn't glean fairly easily set those static addresses the same way as for v4 | 16:36 |
| clarkb | fungi: glean would need to be updated to handle the static address type from config drive and write out the appropriate NM /etc/sysconfig/ config to configure those addresses on those interfaces | 16:37 |
| clarkb | fungi: it needs work in glean, but it isdoable | 16:37 |
| fungi | cool, we're on the same page. that's what i was trying to get to | 16:37 |
| fungi | so if stateful dhcp6 was meant as a workaround for missing static v6 support, then it might make more sense to focus on the static v6 case | 16:37 |
| clarkb | I don't know why anyone would choose stateful dhcvp6 in neutron fwiw | 16:38 |
| clarkb | I don't think you really get much from it? | 16:38 |
| clarkb | maybe dns server config? | 16:39 |
| *** igordc has joined #openstack-infra | 16:39 | |
| sean-k-mooney | fungi: for what its woth i just chose stateful dhcpv6 for my neutron subnet because i understand its better router table usage | 16:39 |
| clarkb | benj_: inap is in the rotation again. Thank you! | 16:40 |
| clarkb | benj_: I have noticed that our maxTotalInstances quota is 10 which means we are only using 10 instances out of the configured 190 | 16:40 |
| clarkb | benj_: additionally maxTotalRamSize allows for 160 instances of 8GB ram (our flavor size) and maxTotalCores allows for 125 intsances at our flavor core count (8) | 16:42 |
| clarkb | benj_: if you have time it would be great to adjust those numbers to reflect what you are able to supply then we can update the nodepool max-servers number if necessary to match | 16:43 |
| clarkb | benj_: also let me know if I can help | 16:43 |
| fungi | #status log updated the owner for https://launchpad.net/os-brick from https://launchpad.net/~openstack-admins to https://launchpad.net/~cinder-drivers at rosmaita's request (the former already owns the latter, so no control is lost) | 16:44 |
| openstackstatus | fungi: finished logging | 16:45 |
| *** yamamoto has joined #openstack-infra | 16:45 | |
| *** whoami-rajat has joined #openstack-infra | 16:45 | |
| clarkb | fungi: fwiw I think it owuld be equally easy to add dhcpv6 support as it would be for static config (in fact dhcpv6 may be slightly easier as you are only setting flags not specific values) | 16:45 |
| clarkb | fungi: I think the red hat support in glean needs a lot of work though | 16:45 |
| sean-k-mooney | i think the same behavior as ipv4 would make sense. | 16:47 |
| sean-k-mooney | if dhcp is disabeld in the usbnet but you have the ip info staticaly configure it | 16:47 |
| clarkb | really its only not there becaus eno one has added it | 16:47 |
| sean-k-mooney | but if dhcp is enabled the configure dhcpv6 | 16:47 |
| benj_ | clarkb: I forgot the quotas have been reseted. I'm updating them. | 16:48 |
| clarkb | benj_: thank you! | 16:48 |
| sean-k-mooney | clarkb: ya i assumed that was the main reason | 16:48 |
| *** yamamoto has quit IRC | 16:49 | |
| openstackgerrit | Merged opendev/puppet-openstackci master: Add a warning to the third party CI doc https://review.opendev.org/689248 | 16:50 |
| *** tosky has quit IRC | 16:52 | |
| clarkb | benj_: I see the update | 16:53 |
| clarkb | now to see if nodepool/shade/sdk will pick up on that update without a restart | 16:53 |
| benj_ | clarkb: You should be good now. I'll be interested in knowing if it works without a restart. | 16:55 |
| clarkb | benj_: looks like nodepool caught up it is now booting many more instances | 16:55 |
| clarkb | benj_: I think we should be set now. I'll keep an eye on it though | 16:56 |
| benj_ | clarkb: same on my side. | 16:56 |
| clarkb | fungi: did the debian mirror updating get set back to normal update process? I assume we are happy with that state now if so? Any changes need to get in to close that? | 16:58 |
| *** priteau has quit IRC | 16:59 | |
| *** igordc has quit IRC | 17:01 | |
| *** igordc has joined #openstack-infra | 17:02 | |
| clarkb | #status log reenabled inap mtl01 in nodepool | 17:02 |
| openstackstatus | clarkb: finished logging | 17:02 |
| *** jaosorior has joined #openstack-infra | 17:03 | |
| fungi | clarkb: images are building successfully last i checked (modulo the day or so where we broke all image builds across the board) | 17:04 |
| *** xek_ has quit IRC | 17:05 | |
| *** xek_ has joined #openstack-infra | 17:05 | |
| fungi | we have ready debian-buster and debian-stretch images from ~4 hours ago | 17:06 |
| fungi | though the debian-buster-arm64 and debian-stretch-arm64 images have been broken for around a week, looks like | 17:07 |
| *** xek_ has quit IRC | 17:07 | |
| fungi | that may be something gone sideways on nb03 though | 17:07 |
| clarkb | I think it ran out of disk | 17:07 |
| *** xek_ has joined #openstack-infra | 17:07 | |
| clarkb | fungi: are we updating the mirror with the regularl cron and don't need to merge config fixes? or if not are those changes up for review? | 17:07 |
| clarkb | stats.timers in graphite is 894GB | 17:09 |
| clarkb | still waiting on my find to finish | 17:09 |
| fungi | cron logs indicate updates are happening. 688788 would be good to get merged | 17:10 |
| fungi | as for changes we need, mainly someone needs to work out how to get the security mirror writing to a different subpath so we can drop the symlinks | 17:10 |
| fungi | it's probably just an option in the reprepro config we need to set | 17:11 |
| *** markvoelker has quit IRC | 17:12 | |
| clarkb | approved 688788 | 17:13 |
| *** udesale has quit IRC | 17:13 | |
| *** mriedem_away is now known as mriedem | 17:14 | |
| *** eernst has joined #openstack-infra | 17:16 | |
| clarkb | benj_: we are seeing timeouts for server deletions I think that might be caused by (or possibly contribute to?) some thrashing that we are seeing there. That said in the past we've seen that when we unleash a bunch of new uncached images onto a cloud and it need a little bit to settle down | 17:16 |
| clarkb | benj_: I think I'll continue to watch it for a bit longer before deciding action needs to be taken | 17:16 |
| *** ralonsoh has quit IRC | 17:18 | |
| clarkb | `find ./ -type f -mtime 365` says there are no files more than a year old in grpahite | 17:19 |
| clarkb | it finally returned with no results | 17:19 |
| clarkb | my du trolling shows that stats.timers is the bulk of the data though so I'm digging around in there now | 17:19 |
| corvus | clarkb: when did the great opendev rename happen? | 17:21 |
| clarkb | corvus: april 19? | 17:21 |
| clarkb | mid april | 17:21 |
| clarkb | do I need to make that the cut off on the find? | 17:22 |
| corvus | clarkb: yeah, i think if we already got the zuulv3 move, that's probably the next big stats reorg | 17:22 |
| clarkb | ok lets try 176 days then (delta between today and may 1) | 17:23 |
| clarkb | fwiw I think the opensatck timers may be the big cost based on poking around with du | 17:23 |
| *** armax has quit IRC | 17:25 | |
| *** markvoelker has joined #openstack-infra | 17:26 | |
| *** jamesmcarthur has quit IRC | 17:29 | |
| *** otsukahy has quit IRC | 17:30 | |
| *** jamesmcarthur has joined #openstack-infra | 17:31 | |
| clarkb | benj_: | fault | {'message': 'Build of instance 52cf6048-c143-4e38-baab-46b0fc387100 aborted: Failed to allocate the network(s) with error No fixed IP addresses available for network: 5a52a95d-c531-4393-ad81-df5ae6ce9010, not rescheduling.', 'code': 500, 'created': '2019-10-24T17:08:05Z'} | | 17:35 |
| openstackgerrit | Merged opendev/system-config master: Further split Debian and Ubuntu reprepro configs https://review.opendev.org/688788 | 17:35 |
| clarkb | 274GB are stats.timers.openstack which means ~600GB is zuul I think | 17:35 |
| clarkb | re the network ip addr failure, the subnet is a /24 with 243 addresses in the allocation pool | 17:37 |
| clarkb | I think maybe we are thrsahing quickly enough that ip addrs aren't getting freed? | 17:37 |
| clarkb | maybe we should reduce max servers to say 100 and see if that makes it happier? | 17:37 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Store a list of held nodes per held build in hold request https://review.opendev.org/682466 | 17:39 |
| *** otsukahy has joined #openstack-infra | 17:40 | |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Set zuul_work_dir in gerrit master job https://review.opendev.org/691029 | 17:40 |
| clarkb | any idea what sends stats.timers.openstack? is that sdk? | 17:41 |
| clarkb | there is a separate stats.timers.nodepool which is much smaller (1.7GB) | 17:41 |
| corvus | clarkb: that seems likely | 17:42 |
| corvus | we pass statsd info into sdk | 17:42 |
| clarkb | http://codesearch.openstack.org/?q=openstack.api.compute&i=nope&files=&repos= seems to say probably sdk too | 17:43 |
| *** jamesmcarthur has quit IRC | 17:43 | |
| corvus | it's the only thing that can time individual api calls now that we use the shade layer in nodepool | 17:43 |
| clarkb | about 270GB of that is compute specific | 17:43 |
| clarkb | I'm not sure we need 270GB of nova api timing stats | 17:44 |
| corvus | i'm also not sure we use them? | 17:44 |
| clarkb | possible that the nodepool grafana dashboards use some of that info but I agree codeserach didn't pop anything up matching it if os | 17:44 |
| corvus | oh, we used to but we didn't update the grafana dashboards | 17:45 |
| *** jaosorior has quit IRC | 17:45 | |
| clarkb | are we able to put a retention policy on this so that we onl yhave say 4 weeks of data? | 17:45 |
| corvus | weird, i wonder why that never came up with all the fn activity... | 17:46 |
| *** otsukahy has quit IRC | 17:46 | |
| clarkb | zuul is also the other 600GB or so and we probably want to figure retention out for zuul too, but I figured the openstack stuff was easier because it is mostly admin facing | 17:46 |
| *** otsukahy has joined #openstack-infra | 17:47 | |
| *** jerryz has joined #openstack-infra | 17:48 | |
| clarkb | I need to step out for a bit but I think that helps narrow down the graphite cleanup focus | 17:48 |
| corvus | clarkb: retentions are here https://opendev.org/opendev/puppet-graphite/src/branch/master/manifests/init.pp | 17:49 |
| *** jaosorior has joined #openstack-infra | 17:50 | |
| clarkb | thanks | 17:50 |
| clarkb | I assume we could do a more specific retention for stats.timers and do less retention of those? | 17:50 |
| corvus | yep | 17:50 |
| clarkb | cool I'll work on a patch for that if no one beats me to it. It will be a bit. I've got a dr appointment at noon and need to get ready for that | 17:51 |
| *** weifan has joined #openstack-infra | 17:52 | |
| *** amoralej is now known as amoralej|off | 17:52 | |
| *** otsukahy has quit IRC | 17:52 | |
| clarkb | inap seems to be settling down now | 17:52 |
| clarkb | 53 in use | 17:52 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Remove --id option for autohold_delete/autohold_info https://review.opendev.org/691033 | 17:53 |
| openstackgerrit | Jeremy Stanley proposed opendev/storyboard master: Regular expression support for CORS and OAuth ACLs https://review.opendev.org/691034 | 17:53 |
| *** eernst has quit IRC | 17:53 | |
| *** jamesmcarthur has joined #openstack-infra | 17:54 | |
| *** ricolin has quit IRC | 17:55 | |
| openstackgerrit | Clark Boylan proposed opendev/puppet-graphite master: Reduce retention of stats.timers https://review.opendev.org/691036 | 17:56 |
| clarkb | corvus: ^ something like that maybe? | 17:56 |
| clarkb | I've got to pop out now feel free to update with better retention rules | 17:57 |
| *** jamesmcarthur has quit IRC | 17:59 | |
| *** armax has joined #openstack-infra | 18:02 | |
| *** tosky has joined #openstack-infra | 18:03 | |
| *** otsukahy has joined #openstack-infra | 18:05 | |
| *** jaosorior has quit IRC | 18:09 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Make the buildset registry port configurable https://review.opendev.org/689240 | 18:10 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Adjust buildset registry container name https://review.opendev.org/690992 | 18:10 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Don't sudo when saving container logs in registry test https://review.opendev.org/690993 | 18:10 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Test buildset registry with k8s and docker https://review.opendev.org/689280 | 18:10 |
| *** otsukahy has quit IRC | 18:10 | |
| *** jaosorior has joined #openstack-infra | 18:11 | |
| openstackgerrit | STEW TY proposed openstack/project-config master: Update Project Configs for x/ranger https://review.opendev.org/691038 | 18:15 |
| *** otsukahy has joined #openstack-infra | 18:17 | |
| *** jaosorior has quit IRC | 18:20 | |
| *** mriedem has quit IRC | 18:20 | |
| *** mriedem has joined #openstack-infra | 18:22 | |
| *** otsukahy has quit IRC | 18:22 | |
| corvus | mordred, clarkb, fungi: https://review.opendev.org/691029 should fix the failure we saw with the gerrit build when it ran against the checks plugin repo | 18:30 |
| corvus | since the new zuul pipeline config is in place, i think we're good to try again after that lands | 18:31 |
| *** otsukahy has joined #openstack-infra | 18:40 | |
| fungi | i'm stepping out to run some errands and grab late lunch/early dinner, but should be back later | 18:41 |
| *** kjackal has quit IRC | 18:42 | |
| openstackgerrit | Merged zuul/nodepool master: AWS driver: add ability to determine AMI id using filters https://review.opendev.org/683205 | 18:45 |
| *** otsukahy has quit IRC | 18:47 | |
| openstackgerrit | STEW TY proposed openstack/project-config master: Update Project Configs for x/ranger https://review.opendev.org/691038 | 18:59 |
| *** pkopec has joined #openstack-infra | 19:01 | |
| openstackgerrit | Adam Coldrick proposed opendev/storyboard-webclient master: Remove unused imagemin build step https://review.opendev.org/691050 | 19:02 |
| openstackgerrit | Adam Coldrick proposed opendev/storyboard-webclient master: Update selenium-standalone and gifsicle https://review.opendev.org/691051 | 19:02 |
| *** rascasoft has quit IRC | 19:07 | |
| openstackgerrit | Merged opendev/system-config master: Set zuul_work_dir in gerrit master job https://review.opendev.org/691029 | 19:08 |
| *** rascasoft has joined #openstack-infra | 19:09 | |
| *** weifan has quit IRC | 19:10 | |
| *** kjackal has joined #openstack-infra | 19:13 | |
| *** whoami-rajat has quit IRC | 19:15 | |
| *** otsukahy has joined #openstack-infra | 19:16 | |
| *** otsukahy has quit IRC | 19:21 | |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Remove --id option for autohold_delete/autohold_info https://review.opendev.org/691033 | 19:32 |
| rm_work | mtreinish: yeah ok, seems like lifeless is who has access to the pypi | 19:37 |
| *** weifan has joined #openstack-infra | 19:43 | |
| *** weifan has quit IRC | 19:49 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Test buildset registry with k8s and docker https://review.opendev.org/689280 | 19:50 |
| *** jtomasek has quit IRC | 19:51 | |
| *** goldyfruit_ has joined #openstack-infra | 19:57 | |
| openstackgerrit | STEW TY proposed openstack/project-config master: Update Project Configs for x/ranger https://review.opendev.org/691038 | 19:59 |
| *** goldyfruit has quit IRC | 19:59 | |
| *** tosky_ has joined #openstack-infra | 20:00 | |
| *** otsukahy has joined #openstack-infra | 20:01 | |
| *** tosky has quit IRC | 20:03 | |
| *** slaweq has joined #openstack-infra | 20:04 | |
| *** otsukahy has quit IRC | 20:07 | |
| *** iokiwi has joined #openstack-infra | 20:08 | |
| *** yamamoto has joined #openstack-infra | 20:11 | |
| *** jamesmcarthur has joined #openstack-infra | 20:14 | |
| *** otsukahy has joined #openstack-infra | 20:15 | |
| *** yamamoto has quit IRC | 20:16 | |
| *** Goneri has quit IRC | 20:18 | |
| *** otsukahy has quit IRC | 20:19 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Test buildset registry with k8s and docker https://review.opendev.org/689280 | 20:20 |
| *** weifan has joined #openstack-infra | 20:22 | |
| *** otsukahy has joined #openstack-infra | 20:22 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update gitea to v1.10 https://review.opendev.org/690873 | 20:23 |
| clarkb | infra-root https://review.opendev.org/#/c/691036/ should hopefully fix graphites lack of free disk | 20:24 |
| clarkb | if you hvae a moment to revie wthat it would be great | 20:25 |
| clarkb | inap seems to have stablized while I was out | 20:26 |
| clarkb | that is good | 20:26 |
| *** weifan has quit IRC | 20:26 | |
| benj_ | clarkb: Not sure where the network issues you saw earlier came from. But things seem to be runnning smooth now. | 20:27 |
| *** otsukahy has quit IRC | 20:27 | |
| clarkb | benj_: my hunch is that builds timed out due to demand the the images not being cached by nova computes yet | 20:28 |
| clarkb | which led to nodepool trying to build more nodes than there were IPs for (due to delay in cleaning things up fully?) | 20:28 |
| clarkb | but ya seems happy now | 20:28 |
| benj_ | clarkb: our guess too on our side. | 20:28 |
| *** weifan has joined #openstack-infra | 20:29 | |
| *** kjackal has quit IRC | 20:32 | |
| mordred | clarkb, corvus: the patch above to update gitea to 1.10 (which hasn't been released yet) actually did catch an api change - so our testing is non-terrible | 20:33 |
| clarkb | does 1.10 include your fix? | 20:33 |
| mordred | yeah. actually - release/v1.9 and release/v1.10 and master all now include my fix | 20:33 |
| *** jamesmcarthur has quit IRC | 20:34 | |
| mordred | so i pushed up https://review.opendev.org/#/c/690872/ which would get us on to what will be 1.9.5 when it releases - and then 690873 would do the same but with the 1.10 series, which is likely to release today ish? (they normally release on the 25th) | 20:35 |
| *** rh-jelabarre has quit IRC | 20:35 | |
| *** weifan has quit IRC | 20:36 | |
| *** weifan has joined #openstack-infra | 20:37 | |
| *** KeithMnemonic has quit IRC | 20:41 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Adds go jobs and roles. https://review.opendev.org/691114 | 20:45 |
| *** weifan has quit IRC | 20:46 | |
| *** weifan has joined #openstack-infra | 20:46 | |
| openstackgerrit | Merged opendev/puppet-graphite master: Reduce retention of stats.timers https://review.opendev.org/691036 | 20:51 |
| *** KeithMnemonic has joined #openstack-infra | 20:52 | |
| *** weifan has quit IRC | 21:00 | |
| *** weifan has joined #openstack-infra | 21:01 | |
| openstackgerrit | Merged zuul/zuul master: Store a list of held nodes per held build in hold request https://review.opendev.org/682466 | 21:04 |
| openstackgerrit | Merged zuul/zuul master: Remove --id option for autohold_delete/autohold_info https://review.opendev.org/691033 | 21:08 |
| *** otsukahy has joined #openstack-infra | 21:09 | |
| *** aedc has joined #openstack-infra | 21:10 | |
| *** sreejithp has quit IRC | 21:13 | |
| *** otsukahy has quit IRC | 21:13 | |
| *** markvoelker has quit IRC | 21:20 | |
| *** rfolco|rover has quit IRC | 21:25 | |
| *** e0ne has joined #openstack-infra | 21:37 | |
| *** goldyfruit_ has quit IRC | 21:38 | |
| *** armax has quit IRC | 21:38 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Test buildset registry with k8s and docker https://review.opendev.org/689280 | 21:39 |
| *** slaweq has quit IRC | 21:44 | |
| *** e0ne has quit IRC | 21:49 | |
| *** e0ne has joined #openstack-infra | 21:49 | |
| *** e0ne has quit IRC | 21:56 | |
| *** slaweq has joined #openstack-infra | 21:56 | |
| *** rcernin has quit IRC | 22:03 | |
| *** apetrich has quit IRC | 22:03 | |
| *** markvoelker has joined #openstack-infra | 22:03 | |
| *** goldyfruit has joined #openstack-infra | 22:03 | |
| *** gfidente has quit IRC | 22:06 | |
| *** igordc has quit IRC | 22:08 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: DNM Update gitea to v1.10 https://review.opendev.org/690873 | 22:11 |
| *** slaweq has quit IRC | 22:13 | |
| *** pkopec has quit IRC | 22:18 | |
| clarkb | the graphite storage-schemas.conf file has been updated on graphite.o.o | 22:23 |
| clarkb | disk usage is where it was | 22:23 |
| clarkb | I am going to restart graphite and carbon to see if that is necessary to pick up the new config | 22:23 |
| corvus | yeah, i'm not sure what needs to happen to trigger that | 22:23 |
| clarkb | it may also be necessary to manually clean up files and only new files will have the new retention? will cross that bridge if we get there | 22:23 |
| corvus | it might be on the next write? | 22:23 |
| corvus | or that | 22:23 |
| corvus | that's actually sounding familiar | 22:23 |
| clarkb | I'll start first with service restart | 22:23 |
| corvus | clarkb: https://stackoverflow.com/questions/30106553/carbon-graphite-apply-new-retention-to-old-data | 22:24 |
| clarkb | thanks | 22:26 |
| clarkb | I guess I'll run that under stats.timers.openstack.api to start | 22:26 |
| clarkb | (I didn't end up restartin gservices yet either as I didn't figure out what they were called before you shared that | 22:26 |
| *** markvoelker has quit IRC | 22:26 | |
| corvus | clarkb: i guess the best thing would be to stop, resize, start ? | 22:27 |
| clarkb | ++ | 22:27 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Test buildset registry with k8s and docker https://review.opendev.org/689280 | 22:27 |
| clarkb | carbon-cache is the service name fwiw | 22:27 |
| clarkb | and statsd | 22:28 |
| clarkb | I think I stop both of those then do resize? | 22:28 |
| clarkb | `find ./ -type f -name '*.wsp' -exec whisper-resize.py --nobackup {} 10s:8h 60s:7d 1h:90d 1d:1y \;` is my command | 22:29 |
| clarkb | going to run that in /var/lib/graphite/storage/whisper/stats/timer/openstack | 22:29 |
| clarkb | stopping carbon-cache and statsd now | 22:30 |
| openstackgerrit | Merged openstack/diskimage-builder master: Fix syntax error in selinux-fixfiles-restore https://review.opendev.org/689430 | 22:30 |
| clarkb | well that failed due to no space left on device. I killed the find | 22:31 |
| clarkb | I guess I need to make a bit of room first | 22:31 |
| *** sgw has quit IRC | 22:33 | |
| mordred | clarkb: :( | 22:34 |
| lifeless | rm_work: I do? for what? | 22:34 |
| *** otsukahy has joined #openstack-infra | 22:34 | |
| clarkb | ok I cleared up ~15MB by moving some of the openstack compute GET dirs into my homedir | 22:34 |
| rm_work | lifeless: https://pypi.org/project/python-subunit/ | 22:34 |
| clarkb | that seems to be sufficient for resizing | 22:35 |
| clarkb | and free disk is climbing now | 22:35 |
| clarkb | I can mv those dirs back into place and rerun over them when the current find is done | 22:35 |
| corvus | it is 92f (33 1/3c) outside my house | 22:35 |
| rm_work | lifeless: the last release was over a year ago... there was a fix made last december to fix a bug in py3, but no release has happened since then | 22:35 |
| *** tosky_ is now known as tosky | 22:36 | |
| clarkb | corvus: it is a beautiful day here about 65F and sunny | 22:36 |
| clarkb | corvus: I hope pg&e doesn't cut your power :/ | 22:36 |
| *** tosky has quit IRC | 22:36 | |
| corvus | clarkb: okay if i come over? :) | 22:36 |
| clarkb | corvus: sure | 22:36 |
| *** weifan has quit IRC | 22:36 | |
| clarkb | infra-root I have a root screen running the find and whisper resize | 22:36 |
| paladox | there still cutting power in CA? | 22:37 |
| corvus | paladox: yeah, they did it again up north, but there's a fire anyway (cause unknown at this point) | 22:37 |
| paladox | :( | 22:37 |
| *** weifan has joined #openstack-infra | 22:38 | |
| ianw | corvus: any thoughts on why https://review.opendev.org/690798 keeps returning NODE_FAILURE? thinking about it, might be the first time i've tried to have multiple architectures in nodes, i wonder if somehow the way arm64 is only on nl03 has some effect? | 22:38 |
| corvus | ianw: all the nodes have to be from the same provider | 22:39 |
| corvus | i'm guessing that's not going to be able to happen here | 22:39 |
| ianw | ahhhh, of course ... | 22:39 |
| corvus | ianw: i guess we'll need a duplicate job? | 22:39 |
| *** armax has joined #openstack-infra | 22:39 | |
| ianw | yeah, that's a bummer because i don't want port bridge.o.o to be arm64 compatible | 22:40 |
| corvus | ianw: or if this is just for ... oh... | 22:40 |
| fungi | attempting to simulate our multi-architecture environment in a multi-node job | 22:40 |
| ianw | i really want the standard x86 bridge node to talk to the arm64 mirror (the bit i'm interested in making sure works) | 22:40 |
| fungi | but yeah, the fact that we assume all nodes in a nodeset are in the same provider makes that impossible with single-architecture providers for each architecture | 22:41 |
| corvus | what's different about the arm64 mirror that makes it worth testing in a job like this? i guess the different arch is like a different distro? | 22:41 |
| corvus | so we want to test that all the packages are there, behave in the same way, etc? | 22:41 |
| ianw | yep, and openafs builds etc | 22:42 |
| fungi | we have some special builds of openafs, yeah | 22:42 |
| corvus | gotcha | 22:42 |
| mordred | yeah, openafs builds yay | 22:42 |
| mordred | this would be a great thing for if we coudl run the playbooks directly on the executor | 22:42 |
| corvus | welp. i acklowdge this use case with these conditions is poorly served by the current design. :| | 22:42 |
| mordred | like if the executor filled the role of bridge | 22:42 |
| ianw | yeah it just didn't click that linaro-london doesn't have any x86 nodes till you said it :) | 22:43 |
| mordred | but we'd need to drop the localhost execution restriction for that | 22:43 |
| corvus | ianw: what kind of effort do you think might be needed for bridge.yaml to be arm64 compat? | 22:43 |
| fungi | there's always the counter-argument that ideally we'd have multi-architecture providers | 22:43 |
| ianw | corvus: i dunno, i can try it ... | 22:44 |
| corvus | mordred: agreed | 22:44 |
| mordred | bridge.yaml shouldn't do *too* many things that are fancy | 22:44 |
| mordred | corvus: did we ever get any consensus around that? | 22:44 |
| corvus | ianw: i think that may be the shortest and best path to happiness | 22:44 |
| corvus | mordred: i think i need to resurrect a ml thread, but i also think we need to get zk/gearman auth/ssl completely sorted first | 22:44 |
| ianw | i'll give it a go. otherwise, i can probably test roles one-by-one | 22:44 |
| mordred | corvus: oh right. zk/gearman auth was the thing | 22:45 |
| paladox | corvus i learned over the week that ofgem has a safty net so if a engery supplier goes bust, the customer doin't need to do much. | 22:45 |
| ianw | or, indeed, just bring up a production host and basically test it live | 22:45 |
| corvus | like, probably to the point where we disallow unauth/untls zk/gear | 22:45 |
| paladox | They'll find a new engery supplier for you :) | 22:45 |
| ianw | although that's not very gitops-y | 22:45 |
| mordred | corvus: yeah | 22:45 |
| paladox | a energy supplier went bust here (small one as i've never herd of it) | 22:45 |
| ianw | i think a limiting factor for linaro-london might be we don't have space to run two mirrors at the same time in our control-plane tenant | 22:46 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Test buildset registry with k8s and docker https://review.opendev.org/689280 | 22:47 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] test arm64 mirror https://review.opendev.org/690798 | 22:47 |
| *** igordc has joined #openstack-infra | 22:47 | |
| *** rlandy has quit IRC | 22:52 | |
| *** mriedem has quit IRC | 22:54 | |
| clarkb | now over a gigabyte is free | 22:54 |
| *** weifan has quit IRC | 22:56 | |
| *** weifan has joined #openstack-infra | 22:58 | |
| *** slaweq has joined #openstack-infra | 23:00 | |
| openstackgerrit | Clark Boylan proposed zuul/zuul-website master: Specify open source projects using Zuul https://review.opendev.org/680110 | 23:02 |
| openstackgerrit | Clark Boylan proposed zuul/zuul-website master: Web improvement ideas https://review.opendev.org/680109 | 23:02 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update gitea to v1.10 https://review.opendev.org/690873 | 23:04 |
| *** tkajinam has joined #openstack-infra | 23:04 | |
| *** slaweq has quit IRC | 23:05 | |
| mordred | clarkb, corvus: I *think* with that version ^^ we should be ready to go when 1.10 is released | 23:09 |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul-website master: Update case studies on users page https://review.opendev.org/691130 | 23:12 |
| *** rcernin has joined #openstack-infra | 23:13 | |
| *** zbr has quit IRC | 23:13 | |
| *** otsukahy has quit IRC | 23:15 | |
| ianw | A start job is running for Glean for interface enp1s0 (29s / no limit) ... hrm not looking good for arm64 nodes atm :/ | 23:16 |
| ianw | the images are a bit out of date; i'm building new ones and monitoring upload | 23:16 |
| ianw | but i don't know why glean service would be stuck | 23:16 |
| *** mattw4 has quit IRC | 23:17 | |
| fungi | nb03 ran out of disk according to clarkb earlier today | 23:18 |
| clarkb | ya pretty sure that is why it hadn't built images in a while | 23:18 |
| ianw | it had also seemed to have a terminal zookeeper dropout | 23:18 |
| fungi | i noticed that the debian arm64 images were about a week stale, likely the rest are as well | 23:18 |
| ianw | yeah bionic is ~10th | 23:18 |
| openstackgerrit | Merged zuul/zuul-website master: Reduce padding to remove extraneous whitespace https://review.opendev.org/617680 | 23:21 |
| ianw | it eventually sort of came up, but it seem no networking http://paste.openstack.org/show/785566/ | 23:22 |
| mordred | ianw: well that looks like it's going to be fun to figure out | 23:25 |
| ianw | as a start, we should probably add the "systemd journal to serial console" element to see what it thinks is happening | 23:26 |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul-website master: Update case studies on users page https://review.opendev.org/691130 | 23:26 |
| *** mattw4 has joined #openstack-infra | 23:28 | |
| *** yamamoto has joined #openstack-infra | 23:30 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: arm64 images: add journal-to-console https://review.opendev.org/691132 | 23:30 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Test buildset registry with k8s and docker https://review.opendev.org/689280 | 23:31 |
| clarkb | we have a dir of whisper files for every image we've ever created | 23:33 |
| clarkb | I'm betting we have a whisper file for every instance we've ever created too | 23:33 |
| clarkb | mordred: ^ | 23:33 |
| corvus | that sounds unuseful | 23:34 |
| clarkb | ./api/object-store/PUT/MossoCloudFS_28a933f9-900d-40b8-9de3-deb24d213aa2/gentoo-17-0-systemd-1560155509/count.wsp for example | 23:34 |
| clarkb | and then we do that for every image that nodepool has created looks like | 23:34 |
| clarkb | and ya I'm not sure that is the correct axis to report on | 23:35 |
| clarkb | I'm going to bet it is a similar story for instances given the size of the compute dir | 23:35 |
| *** yamamoto has quit IRC | 23:35 | |
| *** xek_ has quit IRC | 23:41 | |
| *** slaweq has joined #openstack-infra | 23:43 | |
| openstackgerrit | Merged openstack/project-config master: arm64 images: add journal-to-console https://review.opendev.org/691132 | 23:44 |
| *** dchen has joined #openstack-infra | 23:45 | |
| *** slaweq has quit IRC | 23:48 | |
| clarkb | mordred: corvus I think https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/proxy.py#L197-L199 is the source of the problem | 23:49 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Apply review container to review-dev https://review.opendev.org/690483 | 23:49 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use explicit image paths https://review.opendev.org/690511 | 23:49 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update pip3 role to install from get-pip.py https://review.opendev.org/690766 | 23:49 |
| clarkb | what that extract function does is get bits out of the url that constitute names | 23:50 |
| mordred | clarkb: oh yeah. that's going to be super un-useful for objet-store | 23:50 |
| clarkb | I'm not positive yet but it may actually do the correct thing for compute | 23:50 |
| clarkb | but object-store because it doens't use id's but uses well names causes problems | 23:50 |
| mordred | it should be doing the correct thing for compute - making sure we didn't report on every instance was part of the original design there | 23:51 |
| mordred | yup | 23:51 |
| clarkb | mordred: ya my skim read I think that object-store is the odd one out here | 23:51 |
| mordred | I think what we need is to be able to override _report_stats_statsd in the object-store proxy | 23:51 |
| mordred | because its url pattern is just completely different | 23:52 |
| mordred | reporting stats like "container_create" or "object_create" is much more interesting than "object/x/y/z create" | 23:52 |
| clarkb | ++++++ | 23:53 |
| clarkb | melwitt: ^ I know we don't have a good answer for you yet but we are making changes that should be improving things re graphite | 23:54 |
| *** weifan has quit IRC | 23:55 | |
| clarkb | melwitt: basically openstacksdk is doing some weird things and we are trying to retain data for 5 years. So we're reducing retention and hopefully sdk will update | 23:55 |
| clarkb | and things will be happy in a little while | 23:55 |
| melwitt | clarkb: sounds cool, thanks and looking forward to seeing the improvements :) | 23:56 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!