| clarkb | ianw: fwiw I tested that image url and it was workign from here, but its large so I killed it | 00:00 |
|---|---|---|
| clarkb | ianw: but in theory it is normally accessible | 00:00 |
| ianw | yeah other jobs have worked too | 00:01 |
| ianw | i think it's just one of those days :) | 00:01 |
| openstackgerrit | Merged openstack/project-config master: nb03 : update to arm64 to inheritance, drop pip-and-virtualenv https://review.opendev.org/720641 | 00:02 |
| ianw | will tail builds of ^ | 00:02 |
| donnyd | ianw: is OE functioning correctly? | 00:03 |
| clarkb | thanks. I'm going to be around off and on for the evening, but larlgey calling it a day now | 00:03 |
| ianw | donnyd: yes thanks, doesn't seem to be a systematic issue | 00:03 |
| donnyd | ok cool. I have had few complaints with the new system and this is concerning | 00:04 |
| donnyd | I can assume one of two things - it works perfectly (unlikely), people don't want to complain because they worry there is nothing that can be done to fix it (more likley) | 00:05 |
| ianw | well i'll keep an eye and complain if i see anything :) | 00:06 |
| ianw | 2020-04-24 00:15:01.752 | E: Failed to fetch http://mirror.sjc1.vexxhost.openstack.org/ubuntu/dists/focal-backports/universe/binary-amd64/Packages 404 Not Found [IP: 38.108.68.37 80] | 00:16 |
| ianw | 2020-04-24 00:15:01.752 | E: Failed to fetch http://mirror.sjc1.vexxhost.openstack.org/ubuntu/dists/focal-security/universe/binary-amd64/Packages 404 Not Found [IP: 38.108.68.37 80] | 00:16 |
| ianw | clarkb: ^ hrm, i feel like we had workarounds to ignore these before ... i have strong dejavu | 00:17 |
| ianw | debian, fungi was invovled ... | 00:17 |
| clarkb | ya we have to disable those repos in our apt config on test nodes | 00:19 |
| clarkb | I think its a configure-mirror change? | 00:19 |
| ianw | this is during the dib build ... mordred managed to build it so something must be up | 00:20 |
| *** mlavalle has quit IRC | 00:21 | |
| ianw | they exist in https://mirror.aarnet.edu.au/pub/ubuntu/archive/dists/ | 00:21 |
| ianw | i think with debian they didn't | 00:21 |
| fungi | they exist on official mirrors with empty indices | 00:22 |
| fungi | reprepro builds its indices from the list of packages it synced | 00:22 |
| fungi | since reprepro found no packages to retrieve for those suites, it did not create any indices for them | 00:22 |
| ianw | ... ahhh ... the most compatible thing to do is to probably replicate that then, until they're populated | 00:23 |
| fungi | the reprepro manpage suggests the export subcommand is useful when "...you want to create an initial empty but fully equipped dists/codename directory." | 00:26 |
| fungi | so that might be the solution we need | 00:27 |
| ianw | fungi: --export force maybe? | 00:30 |
| fungi | under "commands" section, export [ codenames ] | 00:31 |
| fungi | though yeah, looks like maybe --export=force when mirroring could do what we want | 00:32 |
| ianw | i don't like the "inconsistent state" caveat in that command :) | 00:34 |
| fungi | yeah, sounds like maybe it doesn't do the same as the export subcommand in that case? | 00:35 |
| ianw | fungi: should i try a : k5start -t -f /etc/reprepro.keytab service/reprepro -- bash -c "reprepro --confdir /etc/reprepro/ubuntu/distributions export focal-backports" | 00:36 |
| fungi | in theory we'd only ever need to run export once, yeah | 00:36 |
| ianw | i'm going to try it ... | 00:37 |
| * fungi makes virtual thumbs-up | 00:37 | |
| clarkb | I think for confdir it ends at ubuntu no distributions | 00:38 |
| ianw | hrm, it sees a current lock and won't do it | 00:38 |
| openstackgerrit | Merged zuul/zuul-jobs master: Fix the checking helm_values_file definition https://review.opendev.org/722516 | 00:40 |
| ianw | i'll try again in a bit | 00:40 |
| fungi | is there a reprepro run in progress? | 00:40 |
| ianw | yes | 00:40 |
| ianw | http://paste.openstack.org/show/792630/ | 00:43 |
| ianw | looks like it worked | 00:43 |
| fungi | yay! so there is a solution | 00:43 |
| fungi | we just have to remember to do that whenever adding a new release i guess | 00:44 |
| ianw | there's stuff in the r/w, i'll release it | 00:44 |
| ianw | ok ... http://mirror.sjc1.vexxhost.openstack.org/ubuntu/dists/focal-backports/universe/binary-amd64/ exists | 00:46 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] add a focal test https://review.opendev.org/722544 | 00:50 |
| ianw | ... round we go again ... | 00:51 |
| ianw | fungi: yep ... see you here again in two years, same time, same place? :) | 00:52 |
| ianw | clarkb: if you have a sec -- https://review.opendev.org/#/c/721726/ is the one stopping upgrades of the functional tests to container based | 00:59 |
| clarkb | ianw is it better to do env -i or do we need other stuff in the env? | 01:02 |
| ianw | clarkb: if i was starting over right now, i might clear the env and put the path in explicitly etc ... but tbh i'd be worried about what that might do with the extant code | 01:03 |
| ianw | if it ain't broke ... :) | 01:04 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] add a focal test https://review.opendev.org/722544 | 01:25 |
| *** kevinz has joined #opendev | 02:06 | |
| ianw | https://zuul.opendev.org/t/openstack/build/8e8c7fea52224558944bb8e218271509/log/nodepool/builds/test-image-0000000001.log#327 | 02:18 |
| ianw | https://zuul.opendev.org/t/openstack/build/4d3510d6f58b42bfbdb1a5e0917a5a84/log/nodepool/builds/test-image-0000000001.log#327 | 02:18 |
| ianw | https://zuul.opendev.org/t/openstack/build/8e8c7fea52224558944bb8e218271509/log/nodepool/builds/test-image-0000000001.log#327 | 02:19 |
| ianw | donnyd: ^ that's 3 instances of seemingly not being able to get to the mirror on openedge from ~00:40 - 00:50 | 02:19 |
| ianw | donnyd: ^ don't worry, it's my fault | 02:31 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Fix args to debuntu functional tests https://review.opendev.org/722564 | 02:36 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: yum-minimal: strip env vars in chroot calls https://review.opendev.org/721726 | 02:40 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Switch functional tests to containers https://review.opendev.org/721511 | 02:40 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Restore SUSE tests to gate https://review.opendev.org/721779 | 02:40 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] add a focal test https://review.opendev.org/722544 | 02:40 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: pip-and-virtualenv : fix fedora 30 install https://review.opendev.org/716795 | 02:43 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Bionic functional tests should be voting https://review.opendev.org/716839 | 02:43 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Switch functional tests to containers https://review.opendev.org/721511 | 02:43 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Restore SUSE tests to gate https://review.opendev.org/721779 | 02:43 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] add a focal test https://review.opendev.org/722544 | 02:43 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Switch functional tests to containers https://review.opendev.org/721511 | 03:13 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Restore SUSE tests to gate https://review.opendev.org/721779 | 03:13 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] add a focal test https://review.opendev.org/722544 | 03:13 |
| openstackgerrit | Merged openstack/project-config master: Add core group for python-keystoneclient https://review.opendev.org/722333 | 03:19 |
| *** ysandeep|away is now known as ysandeep | 04:10 | |
| *** elod has quit IRC | 04:13 | |
| *** ykarel|away is now known as ykarel | 04:13 | |
| openstackgerrit | Ian Wienand proposed opendev/glean master: Drop release test https://review.opendev.org/722549 | 04:44 |
| *** roman_g has quit IRC | 04:53 | |
| *** ysandeep is now known as ysandeep|brb | 04:58 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Switch functional tests to containers https://review.opendev.org/721511 | 05:14 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Restore SUSE tests to gate https://review.opendev.org/721779 | 05:14 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add a focal test https://review.opendev.org/722544 | 05:14 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add a focal test https://review.opendev.org/722544 | 05:19 |
| *** ysandeep|brb is now known as ysandeep | 05:29 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Fix args to debuntu functional tests https://review.opendev.org/722564 | 05:31 |
| openstackgerrit | Merged openstack/diskimage-builder master: yum-minimal: strip env vars in chroot calls https://review.opendev.org/721726 | 05:34 |
| openstackgerrit | Merged openstack/diskimage-builder master: pip-and-virtualenv : fix fedora 30 install https://review.opendev.org/716795 | 05:34 |
| ianw | clarkb / mordred: i think i have a tl;dr for the current work. if you work your way down from the focal change @ https://review.opendev.org/#/c/722544/ we will have testing for all our production images under containers | 06:00 |
| ianw | there's a couple of sideways moves into glean and nodepool from some changes | 06:01 |
| *** DSpider has joined #opendev | 06:01 | |
| ianw | at that point, i feel like we can move all builds to container builders; i.e. nb04 ATM -- but not hard to bring up a couple more to spread the load | 06:02 |
| ianw | that should help corvus with TLS work and just leaves the arm64 story which is in flight | 06:02 |
| *** dpawlik has joined #opendev | 06:03 | |
| *** elod has joined #opendev | 06:20 | |
| *** elod has quit IRC | 06:25 | |
| *** elod has joined #opendev | 06:31 | |
| ianw | frickler: ^ thanks also for your reviews that have been very helpful! | 06:50 |
| *** rpittau|afk is now known as rpittau | 07:17 | |
| *** tosky has joined #opendev | 07:33 | |
| AJaeger | ianw: is https://review.opendev.org/#/c/722549 really correct? Commit message and change don't match | 07:36 |
| *** diablo_rojo has quit IRC | 07:45 | |
| *** ysandeep is now known as ysandeep|lunch | 07:57 | |
| *** ralonsoh has joined #opendev | 08:01 | |
| openstackgerrit | Andreas Jaeger proposed openstack/diskimage-builder master: Add a focal test https://review.opendev.org/722544 | 08:01 |
| yoctozepto | 23:47:08 <ianw> clarkb yoctozepto: centos-8 is slightly different in that it only has packaged pip installed -- we do not overwrite it with pip installed versions | 08:18 |
| yoctozepto | thanks, that's useful info! | 08:18 |
| *** ykarel is now known as ykarel|lunch | 09:05 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Remove translations from networking-ovn https://review.opendev.org/722643 | 09:07 |
| *** ysandeep|lunch is now known as ysandeep | 09:08 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Remove translations from networking-ovn https://review.opendev.org/722643 | 09:12 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul-jobs master: Remove hardcoded use of /home/zuul https://review.opendev.org/722649 | 09:33 |
| AJaeger | mordred: could you review, please? ^ | 09:35 |
| openstackgerrit | Merged openstack/diskimage-builder master: Bionic functional tests should be voting https://review.opendev.org/716839 | 09:41 |
| *** yoctozepto has quit IRC | 10:09 | |
| *** rpittau is now known as rpittau|bbl | 10:12 | |
| *** yoctozepto has joined #opendev | 10:18 | |
| *** elod has quit IRC | 10:24 | |
| *** elod has joined #opendev | 10:30 | |
| *** ykarel|lunch is now known as ykarel | 10:32 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: tox: allow running default envlist in tox https://review.opendev.org/721796 | 11:18 |
| openstackgerrit | Paul Albertella proposed zuul/zuul-jobs master: Add Bazel build and ensure roles https://review.opendev.org/693513 | 11:27 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: tox: allow running default envlist in tox https://review.opendev.org/721796 | 11:41 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: tox: allow running default envlist in tox https://review.opendev.org/721796 | 11:53 |
| *** sgw has quit IRC | 12:08 | |
| *** rpittau|bbl is now known as rpittau | 12:13 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: tox: allow running default envlist in tox https://review.opendev.org/721796 | 12:27 |
| openstackgerrit | Paul Albertella proposed zuul/zuul-jobs master: Add Bazel build and ensure roles https://review.opendev.org/693513 | 12:31 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: tox: allow running default envlist in tox https://review.opendev.org/721796 | 12:42 |
| *** njohnston has quit IRC | 12:43 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: tox: allow running default envlist in tox https://review.opendev.org/721796 | 12:48 |
| openstackgerrit | Akihiro Motoki proposed openstack/project-config master: Define stable cores for horizon plugins in neutron stadium https://review.opendev.org/722682 | 12:49 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: tox: allow running default envlist in tox https://review.opendev.org/721796 | 12:53 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: cabal-test: add initial haskell job https://review.opendev.org/721735 | 13:12 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: hlint: add haskell source code suggestions job https://review.opendev.org/722309 | 13:12 |
| *** ykarel is now known as ykarel|afk | 13:14 | |
| mordred | AJaeger: I left a comment in response to yours on https://review.opendev.org/#/c/722549 | 13:23 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: tox: allow running default envlist in tox https://review.opendev.org/721796 | 13:24 |
| mordred | clarkb: ianw's stack (and depends-on) looks all good to me | 13:26 |
| mordred | infra-root: I'm going to land the codesearch change | 13:30 |
| mordred | corvus: when you get up, could you re-review https://review.opendev.org/#/c/721098 ? it's green now and I updated from your review comments | 13:30 |
| openstackgerrit | Merged openstack/project-config master: Include octavia files when updating branch constraints https://review.opendev.org/722491 | 13:42 |
| openstackgerrit | Merged openstack/project-config master: Finish retiring i18n-specs https://review.opendev.org/721722 | 13:43 |
| fungi | mordred: sounds good | 13:46 |
| *** mlavalle has joined #opendev | 13:49 | |
| *** ysandeep is now known as ysandeep|afk | 13:50 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Support multi-arch image builds with docker buildx https://review.opendev.org/722339 | 13:51 |
| *** iurygregory has quit IRC | 13:55 | |
| *** iurygregory has joined #opendev | 13:56 | |
| AJaeger | mordred: Ah, thx | 13:56 |
| openstackgerrit | Merged opendev/system-config master: Split codesearch into its own playbook https://review.opendev.org/721102 | 13:57 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: tox: allow running default envlist in tox https://review.opendev.org/721796 | 13:58 |
| *** ykarel|afk is now known as ykarel | 14:07 | |
| *** smcginnis has quit IRC | 14:08 | |
| mordred | fungi, corvus, clarkb : real easy one-liner for that stack: https://review.opendev.org/#/c/721633 | 14:09 |
| mordred | corvus: I think we should be able to land both the nodepol and zuul changes today - but obviously we want to watch those | 14:11 |
| *** smcginnis has joined #opendev | 14:11 | |
| corvus | mordred: cool, we need to land that stack from ianw first right? then build new nodepool images? | 14:13 |
| AJaeger | do we need to restart nodepool once that is landed? | 14:17 |
| openstackgerrit | Douglas Mendizábal proposed openstack/project-config master: Add ansible role for managing Luna SA HSM https://review.opendev.org/721349 | 14:21 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run Zuul using Ansible and Containers https://review.opendev.org/717620 | 14:21 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Run docker-compose up automatically on web and mergers https://review.opendev.org/722799 | 14:21 |
| mordred | corvus: I don't think we're blocked on ianw rolling out zuul or nodepool changes | 14:22 |
| mordred | corvus: ^^ I updated that to not do an auto-start of the docker containers | 14:22 |
| mordred | (and then to follow up to do the start for where it's safe) - so that we can land it and not have a race | 14:23 |
| mordred | corvus: I mean - I think we shoudl land the stack from ianw - and it all looks good to me - but I think it's orthogonal to rolling out nodepool-launcher ansible | 14:24 |
| openstackgerrit | Drew Walters proposed openstack/project-config master: Add Airship subproject documentation job https://review.opendev.org/721328 | 14:30 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: tox: allow running default envlist in tox https://review.opendev.org/721796 | 14:34 |
| openstackgerrit | Merged zuul/zuul-jobs master: Remove hardcoded use of /home/zuul https://review.opendev.org/722649 | 14:35 |
| openstackgerrit | Merged zuul/zuul-jobs master: cabal-test: add initial haskell job https://review.opendev.org/721735 | 14:35 |
| corvus | mordred: what's the plan for nodepool-builder? | 14:42 |
| corvus | containers or pip? | 14:42 |
| mordred | corvus: containers - for that we do want to land ianw's changes | 14:43 |
| *** sgw has joined #opendev | 14:43 | |
| corvus | mordred: i'm pre-breakfast -- how do we land the nodepool ansible changes which use containers before we land ianw's stack? | 14:44 |
| mordred | corvus: sorry - I'm being unclear | 14:44 |
| mordred | corvus: I meant the patch that moves nodepool-launchers to ansible/containers | 14:45 |
| mordred | and the patch that does the same for zuul | 14:45 |
| corvus | mordred: oh that patch only does launchers ok | 14:45 |
| corvus | that's what i was missing :) | 14:45 |
| mordred | yah | 14:45 |
| mordred | I'm still on first coffee and you're pre-breakfast - I think we're doing pretty good | 14:45 |
| mordred | corvus: https://review.opendev.org/#/c/720527/ is the nodepool-launcher change | 14:46 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: tox: allow running default envlist in tox https://review.opendev.org/721796 | 14:46 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Support multi-arch image builds with docker buildx https://review.opendev.org/722339 | 14:48 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Support multi-arch image builds with docker buildx https://review.opendev.org/722339 | 14:50 |
| corvus | mordred, ianw: all of the dib-related changes are approved; so we should be able to proceed with that later today | 14:50 |
| mordred | corvus: woot! | 14:51 |
| mordred | corvus: I think the latest change for zuul-jobs for arm images is gonna pass | 14:51 |
| AJaeger | \o/ | 14:52 |
| corvus | mordred: nice. +2 on https://review.opendev.org/717620 -- should we bug fungi or clarkb for a +3 on that? | 14:52 |
| mordred | corvus: yes. and I *thnik* it's safe to land - but we shoudl obviously pay attention :) | 14:52 |
| AJaeger | mordred: once those are in, do you want to merge https://review.opendev.org/#/c/720719/ - to build focal images? Or wait for 717620? | 14:57 |
| * AJaeger left a +2, so you have one already | 14:58 | |
| fungi | corvus: mordred: approved it now, i skimmed it looking for the stuff i expected to be there and that there was testing, but it's a massive diff with 30 revisions so i did not perform any in-depth review | 14:58 |
| fungi | the ability to do staged restarts and quickly roll back with a revert are insurance enough for me | 14:59 |
| mordred | cool! | 15:00 |
| mordred | AJaeger: yes - I think so! | 15:00 |
| mordred | corvus: I'm uploading a focal image to rax right now for control plane - so once we roll out the nodepool-launcher-ansible patch, we can start rolling out new launchers on focal | 15:01 |
| fungi | clarkb: we hit another (smaller) oom condition today on lists.o.o. one process at 11:32:10 and then another a few seconds later at 11:32:29 | 15:01 |
| mordred | fungi: did we have the dstat running yet? | 15:02 |
| fungi | mordred: yep, see our status log for details | 15:02 |
| openstackgerrit | Merged opendev/system-config master: Split eavesdrop into its own playbook https://review.opendev.org/721098 | 15:02 |
| openstackgerrit | Merged opendev/system-config master: Stop logging puppet to syslog https://review.opendev.org/721711 | 15:02 |
| openstackgerrit | Merged opendev/system-config master: Add new etherpad to cacti https://review.opendev.org/721633 | 15:02 |
| fungi | there's a root screen session on the server with the stdout and it's also logging csv in the root homedir | 15:02 |
| mordred | cool | 15:03 |
| mordred | I look forward to understanding why the ooms keep happening | 15:04 |
| AJaeger | impressive: http://zuul.opendev.org/t/openstack/status/change/721102,9 | 15:05 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: tox: allow running default envlist in tox https://review.opendev.org/721796 | 15:05 |
| fungi | mordred: yes, it seems like it could be some periodic event though looking at the cacti graphs | 15:06 |
| fungi | but today's is an interesting specimen... "used memory" was hovering under 1gb, and then in the span of less than a 5min snmp polling interval it spikes up to consume all 4gb ram and 2gb swap | 15:09 |
| fungi | so something ran away and consumed an additional 5gb in a matter of no more than a few minutes | 15:09 |
| *** dpawlik has quit IRC | 15:10 | |
| mordred | wow | 15:13 |
| clarkb | fungi: fwiw the last time I tried to dig into this I was operating under an assumption it could be an email being processed by the mailman queue | 15:15 |
| clarkb | fungi: mailman does run quite a bit of stuff against every email that comes in | 15:15 |
| clarkb | mordred: looks like the puppet splitting got in? does that need anymore eyeballs at this point? | 15:16 |
| mordred | clarkb: don't think so, no | 15:17 |
| clarkb | cool. maybe I'll look over dstat things with some tea then | 15:17 |
| fungi | clarkb: yeah, and during one such event we got deferrals because teh perl module to so spf lookups timed out | 15:17 |
| mordred | clarkb, corvus : https://review.opendev.org/#/c/720709/ and https://review.opendev.org/#/c/720527/ are the next two patches to bug you for reviews of | 15:17 |
| AJaeger | mordred: do we need https://review.opendev.org/#/c/721627/ for accessbot updates? The dependency is merged | 15:18 |
| mordred | yes! | 15:18 |
| mordred | clarkb: ^^ | 15:18 |
| mordred | that's gonna be so neat | 15:18 |
| mordred | clarkb: oh - also - for your review queue: https://review.opendev.org/#/c/722339/ | 15:19 |
| mordred | clarkb: assuming it passes tests, we need that to build arm containers for nodepool-builder | 15:19 |
| AJaeger | mordred: dependency of https://review.opendev.org/#/c/721099/ merged - do you need that one as well (once it's fixed)? | 15:21 |
| AJaeger | oh, let me recheck first... | 15:21 |
| mordred | AJaeger: just did recheck - and yes, very much we should land that | 15:22 |
| AJaeger | mordred: left a comment with a -0.4 ;) | 15:23 |
| AJaeger | http://zuul.opendev.org/t/openstack/status/change/721102,9 is failing in deploy - infra-prod-service-etherpad | 15:26 |
| mordred | AJaeger: actually - no ... we're not going to trigger updates from irc-meetings itself - because that would require us updating the keys on eavesdrop to trust the deploy key from the irc-meetings project | 15:26 |
| clarkb | mordred: can you check the comment on https://review.opendev.org/#/c/720709/8 I worry we've overcomplicated that | 15:26 |
| mordred | AJaeger: so when people make changes to irc-meetings they'll just have to wait for the next hourly pulse | 15:26 |
| AJaeger | mordred: so, should we set allowed-projects to system-config? | 15:27 |
| AJaeger | or am I overcautious? | 15:27 |
| clarkb | mordred: fwiw that isn't a regression. It waited for the puppet pulse via cron before | 15:27 |
| AJaeger | mordred: it's final - so all good | 15:27 |
| mordred | yah | 15:27 |
| mordred | clarkb: so - corvus wasn't crazy about doing the hostnames setup | 15:28 |
| mordred | that said - we could certainly either do hostnames or just do ipv4 | 15:28 |
| AJaeger | "Timeout exception waiting for the logger. Please check connectivity to [bridge.openstack.org:19885]" on 720709 | 15:28 |
| AJaeger | oh, error is different... ignore me | 15:29 |
| AJaeger | http://zuul.opendev.org/t/openstack/status/change/721102,9 is running now for 90 mins in deploy... | 15:29 |
| mordred | #status log uploaded focal-minimal image to rax-dfw for opendev control plane use | 15:30 |
| openstackstatus | mordred: finished logging | 15:30 |
| mordred | AJaeger: yeah - it sure did trigger _Everything_ didn't it? | 15:30 |
| clarkb | fungi: looking at the dstat data we aren't capturing specific process info like devstack. I wonder if that could be useful to see in the run up to memory use what is using it | 15:30 |
| clarkb | fungi: it captures the biggest cpu user and biggest memory user for that slice | 15:30 |
| mordred | the etherpad issue is: | 15:31 |
| mordred | Failed to import the required Python library (setuptools) on etherpad01's Python /usr/bin/python. Please read module documentation and install in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter | 15:31 |
| mordred | so I think that's another one of the fallouts from the docker-compose switch | 15:31 |
| mordred | I will go remove python2 from etherpad | 15:31 |
| corvus | mordred: ++ that's what we did on meetpad | 15:31 |
| mordred | ooh! | 15:32 |
| mordred | wait | 15:32 |
| mordred | so - on etherpad that will cause a different issue - bup is using python2 | 15:32 |
| clarkb | mordred: corvus I think we just want to use python3 for ansible there | 15:32 |
| mordred | so I think we might either need to install python2-setuptools - or set ansible_python_interpreter on that host | 15:32 |
| clarkb | that is what we've been doing on the other hosts | 15:32 |
| mordred | yeah. well - on meetpad it was an easy fix to remove the python2 we weren't using | 15:33 |
| mordred | and ansible found python2 just fine | 15:33 |
| mordred | but in this case I think we need the override - because bup | 15:33 |
| mordred | (but also - we should probably figure out a python2 bup story - is it on python3 on focal I hope?) | 15:33 |
| mordred | uhoh | 15:34 |
| mordred | there is no bup in focal | 15:34 |
| corvus | it's still under development: https://github.com/bup/bup | 15:35 |
| corvus | https://packages.debian.org/search?searchon=names&keywords=bup | 15:36 |
| corvus | oldstable and unstable, but nothing in stable | 15:36 |
| mordred | looks like it fell out of buster, yeah | 15:36 |
| corvus | did it have a moment where it wasn't in debian? | 15:36 |
| mordred | also - looking at the code, it doesn't look like they support python3 | 15:36 |
| mordred | https://github.com/bup/bup/blob/master/config/configure#L61-L72 | 15:37 |
| clarkb | fwiw I've been really happy with borg. I don't think we need to switch to borg yet (or at all), but there are other tools out there should bup no longer be viable | 15:39 |
| corvus | mordred: https://groups.google.com/forum/#!topic/bup-list/QvU-_0oEiFw | 15:39 |
| corvus | mordred: last message | 15:40 |
| clarkb | one of my favorite features is it comes with a fuse driver that allows you to mount and navigate backups | 15:40 |
| corvus | clarkb: does it have append-only behavior? | 15:41 |
| AJaeger | so, https://review.opendev.org/721099 for yaml2ical passes, anybody to +2A, please? | 15:42 |
| clarkb | corvus: yes, it is configurable (I don't know if that means it is toggleable though) | 15:42 |
| clarkb | fungi: 24-04 11:27:58 is when we see the memory use start to climb | 15:43 |
| corvus | clarkb: hopefully not on the client side :) "haxor: pls rewrite old backups with compromised datas, kthx" | 15:43 |
| clarkb | corvus: https://borgbackup.readthedocs.io/en/stable/usage/notes.html#append-only-mode | 15:44 |
| clarkb | I don't use append only with my local backups fwiw | 15:44 |
| corvus | clarkb: yeah, it's useful if you want to be able to restore from a security compromise | 15:45 |
| clarkb | corvus: I think you'd make the borg config in the repo writeable by non remote users reading that doc | 15:45 |
| mordred | we seem to have lost gerritbot | 15:45 |
| clarkb | mordred: it does the channel rotation thing so it may not be completely gone | 15:46 |
| mordred | ah - nod | 15:46 |
| mordred | remote: https://review.opendev.org/722917 Use python3 for ansible on etherpad | 15:46 |
| mordred | remote: https://review.opendev.org/722918 Remove two unused ansible vars files | 15:46 |
| mordred | clarkb corvus ^^ that should fix the etherpad ansible issue - the other is just something I noticed while doing it | 15:47 |
| fungi | clarkb: i'm running dstat the same way as the csv log devstack generates, at least according to the devstack source. i can take a closer look shortly but need to go do our weekly grocery pick-up first. will be back in a few | 15:48 |
| *** ysandeep|afk is now known as ysandeep|away | 15:49 | |
| mordred | that said - the current prod deploy is pushing 2 hours | 15:50 |
| clarkb | fungi: devsatck uses --top-mem --top-cpu-adv and --top-io-adv | 15:50 |
| mordred | but I think it's mostly delays between zuul job runs | 15:50 |
| clarkb | fungi: I think this may be part of the console output and not csv output? either way it might show us what is drastically starting to consume memory at that point in time | 15:50 |
| mordred | the individual playbook times aren't particulaly slow | 15:50 |
| mordred | http://paste.openstack.org/show/792677/ | 15:50 |
| mordred | base took 17 minutes, which isn't super unreasonable | 15:51 |
| corvus | what's the delay between runs? | 15:51 |
| mordred | so - pulling one at random - mirror-update took 55 seconds to run - and the next playbook ran 5 minutes later | 15:52 |
| corvus | mordred: remember that we do have the situation where we are running deploy jobs out of order | 15:53 |
| mordred | so if we wind up running ALL of the playbooks and that's a reasonably realistic delay - that's a good 100 minutes in interstitual delay | 15:53 |
| mordred | corvus: totally - I'm only looking at playbook log timestamps | 15:53 |
| corvus | we could have run a different job in those 5 minutes | 15:53 |
| mordred | nope - I'm sorting by log files | 15:53 |
| corvus | nope | 15:53 |
| corvus | i'm talking about different changes | 15:53 |
| corvus | mordred: or do you mean you would have seen the log on bridge? | 15:54 |
| mordred | yes | 15:54 |
| corvus | ok, sorry, i got it now | 15:54 |
| mordred | I'm only looking at bridge log times | 15:54 |
| corvus | mordred: what was the delaybetween zookeeper and zuul-preview? | 15:54 |
| mordred | -rw-r--r-- 1 root root 73056 Apr 24 15:51 service-zookeeper.yaml.log | 15:54 |
| mordred | -rw-r--r-- 1 root root 140 Apr 24 15:54 service-zuul-preview.yaml.log | 15:54 |
| mordred | so not as bad | 15:55 |
| mordred | that's good - means it's variable | 15:55 |
| corvus | well that doesn't tell us the interstitial time | 15:55 |
| mordred | good point - one sec | 15:55 |
| mordred | zookeeper ended at 15:51:25 - zuul-preview started at 15:54:10 | 15:56 |
| mordred | from a bridge playbook pov | 15:56 |
| mordred | lemme see if I can get a better grep that captures both of those things | 15:56 |
| corvus | mordred: i didn't see a delay in zuul status page; i'm guessing if we looked at the executor logs, we'd see job post/cleanup tasks going right up to the start of the next one | 15:56 |
| corvus | 2020-04-24 15:52:57,726 INFO zuul.AnsibleJob: [e: cc2f89f4d6ba4028af8b28180166fee7] [build: a61ccc41a81e4e4ab7d33aac70e45a6c] Job execution took: 338.891 seconds | 15:57 |
| corvus | that was when the executor was done with it | 15:58 |
| mordred | nod - so we're looking at setup/cleanup times | 15:58 |
| corvus | yeah, i think so, with probably a little bit of scheduler time (< 1 minute here i think) | 15:58 |
| corvus | mordred: can we revisit the mutex? what are the actual critical sections? | 15:59 |
| corvus | we ought to be able to have more parallelism here, but i feel like we actually have less than we did with puppet | 15:59 |
| mordred | puppet ran completely serial | 16:00 |
| mordred | in run_all | 16:00 |
| clarkb | fungi: the IncomingRunner qrunner for openstack is currently using 900MB of resident memory roughly | 16:00 |
| mordred | but - I agree - I think we can introduce more parallelism | 16:00 |
| mordred | and totally think we should revisit the mutex | 16:00 |
| mordred | corvus: the big issue _now_ though will be that each change is going to push git repo states | 16:01 |
| mordred | so I think it'll take some deeper thinking to unwind the mutex | 16:01 |
| corvus | mordred: if we could secure the "only run jobs for one change at a time" issue, could we do a "push git repo job", then have all the others depend on that? | 16:01 |
| mordred | corvus: yes - although we'd want to put _all_ of the required projects for any of them into the push git repo jobs job | 16:02 |
| mordred | (so that we're always pushing both system-config and project-config and any other repos like irc-meetings that have data - but I don't think that's problematic to do at all) | 16:03 |
| mordred | and I actually think it's a great idea | 16:03 |
| mordred | because then for any given change we can go back to letting job dependencies handle sequencing and let the others run willy nilly | 16:03 |
| corvus | okay, i'll look at what it would take to use manager:dependent window:1 in deploy | 16:04 |
| mordred | corvus: in fact- all of them already depend on install-ansible - it needs to _always_ run - so that could also be the "sync repos" job | 16:04 |
| mordred | it would make logical sense "sync the repos and install any ansible config updates" | 16:05 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: tox: allow running default envlist in tox https://review.opendev.org/721796 | 16:10 |
| openstackgerrit | Brian Haley proposed openstack/project-config master: Retire openstack/networking-ovn repository https://review.opendev.org/721565 | 16:10 |
| openstackgerrit | Merged openstack/diskimage-builder master: Switch functional tests to containers https://review.opendev.org/721511 | 16:14 |
| openstackgerrit | Brian Haley proposed openstack/project-config master: Remove the networking-ovn grafana dashboard https://review.opendev.org/722925 | 16:15 |
| openstackgerrit | Monty Taylor proposed openstack/project-config master: Set sync repos flag in all project-config infra-prod jobs https://review.opendev.org/722926 | 16:15 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Only sync the git repos in install-ansible https://review.opendev.org/722927 | 16:15 |
| mordred | corvus: ^^ that should do it from a job config perspective | 16:16 |
| mordred | corvus: once we land both of those and get window:1 working, I think we can drop the mutex | 16:17 |
| corvus | mordred: cool, we should probably wip those for now | 16:17 |
| corvus | since i think if we run a job for an out of order change it'd be bad | 16:18 |
| mordred | corvus: agree | 16:19 |
| mordred | corvus: I -2'd the project-config change | 16:19 |
| openstackgerrit | Merged openstack/diskimage-builder master: Restore SUSE tests to gate https://review.opendev.org/721779 | 16:20 |
| openstackgerrit | Merged openstack/diskimage-builder master: Add a focal test https://review.opendev.org/722544 | 16:20 |
| mordred | corvus: also - hahaha - I just noticed ... in the "run zuul with ansible" change ... | 16:20 |
| mordred | I didn't actually add a job to run zuul with ansible | 16:20 |
| mordred | so - you know - it's neat and all, but it sure isn't going to do anything - followup coing | 16:21 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: tox: allow running default envlist in tox https://review.opendev.org/721796 | 16:23 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Actually run service-zuul playbook in prod https://review.opendev.org/722933 | 16:27 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add service-zookeeper to nightly jobs https://review.opendev.org/722934 | 16:27 |
| mordred | corvus, fungi : ^^ | 16:27 |
| mordred | infra-root: new codesearch playbook ran successfully | 16:30 |
| openstackgerrit | Merged opendev/system-config master: Run Zuul using Ansible and Containers https://review.opendev.org/717620 | 16:31 |
| clarkb | exciting | 16:32 |
| *** ykarel is now known as ykarel|away | 16:33 | |
| mordred | clarkb: well - almost | 16:36 |
| mordred | see 722933 :) | 16:36 |
| mordred | VERY close to being exciting though | 16:36 |
| mordred | clarkb: also - the scrollback between corvus and I about mutex and windows is worth reading | 16:37 |
| clarkb | mordred: should that run after gerrit so that new projects are created in gerrit before zuul tries to consume them | 16:37 |
| mordred | clarkb: maybe after manage-projects | 16:37 |
| fungi | clarkb: yeah, i focused on the csv part you had mentioned, but if that's not actually what captures the things we want i'm happy to adjust | 16:38 |
| clarkb | mordred: oh yup we split those now | 16:38 |
| clarkb | fungi: dstat -tcmndrylpg --top-cpu-adv --top-mem --swap I think you have to run that in the foreground | 16:39 |
| clarkb | fungi: unfortauntely it doesn't show us a pid for the top-mem process just a process name | 16:39 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Actually run service-zuul playbook in prod https://review.opendev.org/722933 | 16:40 |
| mordred | clarkb: fixed | 16:40 |
| mordred | clarkb: and thanks - good catch! | 16:40 |
| fungi | clarkb: yeah, the script for devstack runs two dstat processes, one in the background writing to a csv and the other in the foreground... i suppose i could run in two different screen windows | 16:41 |
| clarkb | fungi: I don't know that we need the csv version | 16:41 |
| *** rpittau is now known as rpittau|afk | 16:41 | |
| clarkb | fungi: it is good that it shows us when memory use changes (even when cacti is unable to snmp) but it doesn't show us much more than that | 16:42 |
| corvus | mordred: i think we missed something with the deploy pipeline: we can end up deploying the change-as-written not the change-as-merged. | 16:42 |
| mordred | corvus: oh? | 16:42 |
| corvus | mordred: a change-merged trigger is fine for promote, because we're promoting the artifact-as-merged-and-built; but even in that case, the code on disk is the change-as-written | 16:43 |
| corvus | well, okay, it's a hybrid | 16:43 |
| corvus | it's kind of racy | 16:43 |
| mordred | oh - oy | 16:43 |
| corvus | it's the change-as-written merged onto the repo tip. which in most cases should be the thing that merged right before it. but may not be if 2 things merge really fast? | 16:44 |
| corvus | i'm not positive about this, but trying to explain it has helped me see the weak spots. let me dig more. | 16:44 |
| mordred | corvus: I agree | 16:44 |
| mordred | corvus: I think the words you said have caused me to panic less | 16:45 |
| mordred | because while it's racy, *most* of the time it's likely about the same thing for us | 16:45 |
| mordred | but - definitely could also be incorrect | 16:45 |
| mordred | also - given how we're doing things, the chances that 2 changes land in parallel and have a substantive impact on each others playbook run git state without being a merge-conflict or a depends-on is pretty low | 16:46 |
| mordred | (so like, the zuul playbook change might not include the nodepoo playbook change because of a race - but if they were semantically related they would have been stacked or depends-on'd) | 16:47 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add service-zookeeper to nightly jobs https://review.opendev.org/722934 | 16:59 |
| mordred | corvus, fungi : could I get one of you to +A https://review.opendev.org/722933 real quick? | 17:00 |
| openstackgerrit | Roman Gorshunov proposed openstack/project-config master: Retire airship-in-a-bottle https://review.opendev.org/720160 | 17:03 |
| openstackgerrit | Roman Gorshunov proposed openstack/project-config master: Retire airship-in-a-bottle https://review.opendev.org/720160 | 17:04 |
| *** mlavalle has quit IRC | 17:04 | |
| *** mlavalle has joined #opendev | 17:04 | |
| openstackgerrit | Merged opendev/system-config master: Use python3 for ansible on etherpad https://review.opendev.org/722917 | 17:09 |
| mordred | corvus: I don't know if it's useful - but we have an instance of a job running "out of order" in deploy right now | 17:12 |
| openstackgerrit | Merged opendev/system-config master: Remove two unused ansible vars files https://review.opendev.org/722918 | 17:17 |
| *** logan- has quit IRC | 17:30 | |
| openstackgerrit | Merged opendev/system-config master: Actually run service-zuul playbook in prod https://review.opendev.org/722933 | 17:34 |
| *** elod has quit IRC | 17:42 | |
| *** logan- has joined #opendev | 17:42 | |
| *** elod has joined #opendev | 17:48 | |
| *** ralonsoh has quit IRC | 17:52 | |
| openstackgerrit | Merged opendev/system-config master: Add service-zookeeper to nightly jobs https://review.opendev.org/722934 | 17:58 |
| openstackgerrit | Merged openstack/project-config master: Retire airship-in-a-bottle: end project gating https://review.opendev.org/721978 | 19:04 |
| openstackgerrit | Merged openstack/project-config master: Retire openstack/networking-ovn repository https://review.opendev.org/721565 | 19:04 |
| openstackgerrit | Merged openstack/project-config master: Remove the networking-ovn grafana dashboard https://review.opendev.org/722925 | 19:07 |
| openstackgerrit | Merged openstack/project-config master: Update Neutron Grafana dashboard https://review.opendev.org/718392 | 19:07 |
| AJaeger | clarkb, corvus, mordred, want to merge https://review.opendev.org/721099 and https://review.opendev.org/720719 https://review.opendev.org/721627 ? | 19:13 |
| mordred | AJaeger: ++ | 19:17 |
| mordred | clarkb: feel like one more fun review? https://review.opendev.org/#/c/720527/ | 19:28 |
| clarkb | Ya I'll pick up more reviews after lunch | 19:29 |
| mordred | cool | 19:33 |
| mordred | infra-root: since we missed triggering the service-zuul playbook b/c of job config - I'm going to run it manually real quick | 19:33 |
| fungi | wfm | 19:36 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use the sync-project-config role in service-zuul https://review.opendev.org/723022 | 19:43 |
| mordred | noticed that ^^ | 19:44 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: The package is libjemalloc1 https://review.opendev.org/723023 | 19:46 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: The package is libjemalloc1 https://review.opendev.org/723023 | 19:48 |
| mordred | infra-root: ^^ two quick fixes from the run of service-zuul - including the fact that we didnt' test the zuul-executor role because of a typo in the hostname | 19:48 |
| yoctozepto | is gerrit healthy? it rejects an up-to-date change with message that it cannot merge it https://review.opendev.org/723021 | 19:49 |
| clarkb | yoctozepto: note both zuul and gerrit report the same thing | 19:50 |
| yoctozepto | clarkb: yeah, is zuul healthy then? :D | 19:50 |
| clarkb | I expect they are both healthy if they independently reach the same conclusion | 19:51 |
| yoctozepto | any hints? | 19:51 |
| yoctozepto | could it be that it picks up the cross-ref as breaker? | 19:52 |
| AJaeger | yoctozepto: did you try rebasing lockly? | 19:52 |
| yoctozepto | it is on top of the current head | 19:52 |
| yoctozepto | yes, this is right off the HEAD | 19:52 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: hlint: add haskell source code suggestions job https://review.opendev.org/722309 | 19:52 |
| yoctozepto | even gerrit confirms ;-) | 19:52 |
| yoctozepto | see Parent(s) | 19:52 |
| clarkb | ya I'm not sure yet why they dislike it | 19:52 |
| *** mgagne has joined #opendev | 19:53 | |
| AJaeger | why is Zuul reporting *twice* on the change? | 19:53 |
| corvus | different tenants | 19:53 |
| AJaeger | corvus: kolla-ansible is only once in project-config/zuul/main.yaml - so there should be only one tenant | 19:54 |
| yoctozepto | lorem ipsum todo fixme did not help :O | 19:54 |
| AJaeger | yoctozepto: Zuul does not like Latin ;) | 19:55 |
| fungi | lorem ipsum isn't technically latin... perhaps zuul appreciates good latin ;) | 19:55 |
| clarkb | scheduler says it has a merge conflict | 19:56 |
| clarkb | now to find the merger I guess | 19:56 |
| AJaeger | if you look at https://review.opendev.org/#/q/owner:radoslaw.piliszek%2540gmail.com+status:open it does not show a merge conflict | 19:56 |
| yoctozepto | yeah, I suspect Zuul | 19:56 |
| yoctozepto | but cannot tell | 19:57 |
| clarkb | Unable to reset repo <Repo 0x7f9968183f60 /var/lib/zuul/git/opendev.org/openstack/kolla-ansible> on zm02 | 19:57 |
| mordred | clarkb: maybe something went poorly with teh service-zuul playbook? | 19:57 |
| fungi | i've confirmed 723021 is definitely parented directly on the current master ref | 19:58 |
| mgagne | mordred: A new release of glean was done a couple days ago. I'm having issue with glean on Ubuntu Bionic where resolvers (although configured) aren't used until systemd-resolved service is restarted. Did you find any similar issue? | 19:58 |
| clarkb | mgagne: ya it looks like permissions issues | 19:58 |
| clarkb | mgagne: we don't use systemd-resolvd | 19:58 |
| clarkb | mgagne: but the release of glean was made to fix systemd-resolved for gentoo | 19:58 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: hlint: add haskell source code suggestions job https://review.opendev.org/722309 | 19:58 |
| mgagne | well, it broke it for bionic :P | 19:58 |
| mordred | clarkb: ok - I think there was a change due to ansible - although I still don't know why that broke something | 19:59 |
| mgagne | how can one disable systemd-resolved ? | 19:59 |
| corvus | mordred, clarkb, yoctozepto: I594ceff9073f7ca14221d4f37716c76b462fc3d9 has merged, but the merger proc on zm02 is 2 months old, so it's certainly not running there | 19:59 |
| clarkb | mordred: stderr: 'Permission denied (publickey). | 19:59 |
| clarkb | its trying to fetch refs and failing to do so via ssh I think | 20:00 |
| mordred | poo. did we write out the wrong ssh key? | 20:00 |
| corvus | AJaeger: ah it's in 2 pipelines -- check and check-arm64 | 20:00 |
| corvus | mordred, clarkb, yoctozepto: let's take the merger issue to #opendev or #opendev-meeting | 20:01 |
| corvus | er, derp this is opendev | 20:01 |
| corvus | sorry :) | 20:01 |
| yoctozepto | :-) | 20:01 |
| mordred | let's take it to meeting though | 20:01 |
| clarkb | mordred: I think you added the zuul user | 20:01 |
| *** roman_g has joined #opendev | 20:01 | |
| yoctozepto | ok, folks, thanks for your attention, /me going to bed now, please touch this change so that it can test overnight if it's not a problem :-) | 20:03 |
| *** roman_g has quit IRC | 20:06 | |
| *** gouthamr has joined #opendev | 20:09 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Realign with previous uid/gid values https://review.opendev.org/723029 | 20:17 |
| -openstackstatus- NOTICE: The Zuul project gating service is reporting new patches in merge conflict erroneously due to a configuration error, fix in progress | 20:19 | |
| *** avass has joined #opendev | 20:24 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use the right key for zuul ssh key https://review.opendev.org/723032 | 20:25 |
| *** rchurch has joined #opendev | 20:54 | |
| mgagne | reverting to a previous release of glean fixed my issue | 21:03 |
| clarkb | mgagne: ok so likely fallout from what prometheanfire changed | 21:04 |
| clarkb | mgagne: can you describe the issue in some detail so that we can sync up with prometheanfire about it? email or storyboard or even just here probably works | 21:05 |
| mgagne | prometheanfire, clarkb: Since release 1.18.0, DNS is no longer working on Ubuntu Bionic. In fact, it seems resolvers used to be configured in /etc/resolv.conf before the release. After the release, they got moved to /etc/systemd/resolved.conf but won't be considered/used until systemd-resolved service is restarted manually after the boot. | 21:08 |
| clarkb | mgagne: aha and bionic doesn't run resolved by default iirc | 21:09 |
| clarkb | fwiw my suggestion was to write to /etc/resolv.conf since that gets symlinked to systemd things if systemd things are used | 21:09 |
| clarkb | I guess that wasn't how prometheanfire fixed it. Unfortunately I didn't have time to read the glean changes | 21:09 |
| mgagne | The service is active though: active (running). But still need to restart it to pickup the new nameservers. | 21:10 |
| clarkb | oh maybe a race in writing the config then | 21:10 |
| mgagne | yeah, I tried restarting it in rc.local like a caveman but it seems it's too early to restart it somehow, glean is still doing its stuff. | 21:11 |
| fungi | i'm surprised systemd hasn't just set rc.local on fire | 21:15 |
| mgagne | I think it no longer exists by default, you need to create it and set the right permissions to get it executed. | 21:16 |
| mgagne | but yeah, still supported... | 21:16 |
| mgagne | and yes, systemctl is-enabled systemd-resolved returns "enabled" on Bionic | 21:17 |
| prometheanfire | mgagne: so sounds like glean needs to reload resolved if already started | 21:19 |
| mgagne | prometheanfire: I agree | 21:20 |
| clarkb | prometheanfire: mgagne usually we set it up so that glean runs before those other services instead | 21:21 |
| prometheanfire | clarkb: ya, not sure why it seems to be run before glean runs | 21:23 |
| prometheanfire | mgagne: can you try adding a Before=systemd-resolved.service line to the top of glean? | 21:25 |
| prometheanfire | the glean service file that is | 21:25 |
| mgagne | in "/usr/lib/systemd/system/glean@.service" right? | 21:26 |
| prometheanfire | looks right | 21:26 |
| mgagne | ok, I'll try. gonna time some time to get image built though. | 21:27 |
| prometheanfire | clarkb: mgagne I think the problem may be having both wants= and before=network-pre.target | 21:28 |
| clarkb | prometheanfire: I think thats ok | 21:29 |
| prometheanfire | or not | 21:29 |
| prometheanfire | Note that requirement dependencies do not influence the order in which services are started or stopped. This has to be configured independently with the After= or Before= options. If unit foo.service pulls in unit bar.service as configured with Wants= and no ordering is configured with After= or Before=, then both units will be started simultaneously and without any delay between them if | 21:29 |
| prometheanfire | foo.service is activated. | 21:29 |
| clarkb | wants means only run if networking is being start | 21:29 |
| clarkb | before means we run before it | 21:29 |
| prometheanfire | right | 21:29 |
| prometheanfire | I think before should fix it | 21:29 |
| prometheanfire | mgagne: you could remove the conditionpreexists line to do it live | 21:30 |
| mgagne | and clean some config files too since they will already be configured | 21:31 |
| prometheanfire | doesn't ubuntu 18.04+ have networkd installed by default | 21:31 |
| prometheanfire | mgagne: ya | 21:31 |
| clarkb | prometheanfire: no ubuntu is using some other thing theybuilt for it | 21:33 |
| clarkb | prometheanfire: it uses yaml and is not systemd related | 21:33 |
| prometheanfire | netplan | 21:33 |
| prometheanfire | for a while networkd looked to be the one way | 21:33 |
| prometheanfire | then redhat | 21:33 |
| fungi | i thought bionic was using networkmangler | 21:34 |
| prometheanfire | maybe | 21:34 |
| clarkb | no red hat is all networkmanager | 21:34 |
| clarkb | ubuntu is yaml thing | 21:34 |
| fungi | ahh | 21:34 |
| clarkb | I only know this because networkmanager can't actually ivp6 properly | 21:35 |
| prometheanfire | man, that reminds me of mangler (ventrilo client), packaged a long time ago | 21:35 |
| clarkb | and we spent extensive effort fixing it in red hat distros | 21:35 |
| fungi | well, or it avoids doing ipv6 autoconfig if it thinks the kernel is taking care of it | 21:35 |
| clarkb | fungi: and ipv4 config in that case too :) | 21:36 |
| clarkb | which is the problem | 21:36 |
| clarkb | tehre was some actual fix that I don't remember now | 21:36 |
| * prometheanfire suggests we just install networkd on all images, disable the distro-specific network thing and call it a day :D | 21:37 | |
| clarkb | prometheanfire: we are trying to do less special stuff | 21:38 |
| clarkb | prometheanfire: we are even ripping out all of the preinstall of python stuff | 21:38 |
| * prometheanfire just grumbles | 21:38 | |
| clarkb | prometheanfire: glean isgetting installing in a python3 -mvenv virtualenv going forward | 21:38 |
| clarkb | really if you don't use systemd everything works great | 21:39 |
| clarkb | :P | 21:39 |
| prometheanfire | openrc works :P | 21:39 |
| mgagne | prometheanfire: using Before=systemd-resolved.service didn't work | 22:03 |
| prometheanfire | wtf | 22:05 |
| mgagne | http://paste.openstack.org/show/792695/ | 22:16 |
| mgagne | it doesn't seem using Before worked. here with systemd file http://paste.openstack.org/show/792696/ | 22:17 |
| *** rkukura has joined #opendev | 22:23 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Fix zuul.conf jinja2 template https://review.opendev.org/723046 | 22:37 |
| clarkb | mordred: ^ fyi trying to track thsoe fixes there | 22:38 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add utility playbooks for starting and stopping executors https://review.opendev.org/723048 | 22:41 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add nodepool node key https://review.opendev.org/723049 | 22:41 |
| mordred | clarkb: ^^ same | 22:41 |
| prometheanfire | mgagne: I'll look at creating a image and testing it | 22:42 |
| prometheanfire | mgagne: have a dib invocation for me to test? | 22:43 |
| mgagne | prometheanfire: thanks for looking into it. I've reverted back to previous release for now. | 22:43 |
| prometheanfire | mgagne: how did you call dib for testing? | 22:43 |
| mgagne | it's all baked into nodepool. | 22:43 |
| prometheanfire | ah, nodepool invocation, thanks | 22:43 |
| mgagne | ubuntu-minimal,simple-init,growroot,devuser,nodepool-base,nodepool-jenkins-slave are the elements. nodepool-base and nodepool-jenkins-slave are custom elements. nodepool-base requires openssh-server,pip-and-virtualenv. The other one installs a bunch of development libs. | 22:46 |
| mgagne | we are also using an old version of DIB: diskimage-builder==2.11.0 Hopefully it doesn't have too much impact. | 22:47 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add nodepool node key https://review.opendev.org/723049 | 22:54 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add nodepool node key https://review.opendev.org/723049 | 22:54 |
| -openstackstatus- NOTICE: the This Zuul outage was taken as an opportunity to perform an impromptu maintenance for changing our service deployment model; any merge failures received from Zuul between 19:40 and 20:20 UTC were likely in error and those changes should be rechecked; any patches uploaded between 20:55 and 22:45 UTC were missed entirely by Zuul and should also be rechecked to get fresh test results | 22:56 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Remove an extra backslash https://review.opendev.org/723052 | 23:04 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: The package is libjemalloc1 https://review.opendev.org/723023 | 23:06 |
| *** tosky has quit IRC | 23:14 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Fix zuul.conf jinja2 template https://review.opendev.org/723046 | 23:18 |
| clarkb | mordred: fungi corvus ^ that was rebased on mordred fix for zl01 and now it will collect zuul.conf files in system-config-run-zuul | 23:19 |
| prometheanfire | mgagne: hmm, I just built an image and it can ping out just fine | 23:19 |
| * prometheanfire forgot to install openssh-server though | 23:20 | |
| prometheanfire | but once installed (via virsh console) everything seems normal | 23:21 |
| prometheanfire | called via... | 23:22 |
| prometheanfire | DIB_DEV_USER_PASSWORD=bazfoobar DIB_DEV_USER_USERNAME=gentooo DIB_DEV_USER_PWDLESS_SUDO=yes DIB_DEV_USER_AUTHORIZED_KEYS=/root/.ssh/authorized_keys DIB_RELEASE=bionic disk-image-create -a amd64 -t qcow2 --image-size 3 ubuntu-minimal simple-init growroot devuser vm block-device-mbr bootloader | 23:22 |
| prometheanfire | so cannot reproduce | 23:23 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!