| ianw | fungi: yeah, i saw that ... i don't know if debootstrap speed has been a limiting factor :) the cross-arch stuff was interesting | 00:00 |
|---|---|---|
| clarkb | fwiw the biggest technical hurdles to use the vanilla images are going to be rax networking and dealing with git repos | 00:01 |
| fungi | i rather like the fakechroot support, and ability to build on mounts with the nodev option | 00:01 |
| ianw | i think that has been shoehorned into dib in some way, i think dan radez managed a cross-build | 00:01 |
| ianw | fungi: but for our following steps, running all our scripts in the chroot, we're still going to have to mount in /dev /proc etc? | 00:02 |
| clarkb | We could go back to non minimal builds and layer git repos on top of the upstream vanilla images if they supported rax networking | 00:02 |
| *** diablo_rojo has joined #openstack-infra | 00:02 | |
| ianw | fungi: that's one big area of unreliability in the failure case; leaving all those mounts behind on unexpected exit. we do have exit traps but can get confused at points | 00:02 |
| fungi | ianw: yeah, that's also where the cross-arch builds are less useful for us unless we add an emulation/virt layer | 00:03 |
| fungi | (at some point we need to run processes within the scope of that built filesystem before we image it) | 00:03 |
| ianw | even binary translation it might still run faster than gentoo native rebuilding everything ;) | 00:04 |
| clarkb | if I ever find time it was the yocto project (I think) that had a builder that didn't even need root because it used mkfs.ext4 and .btrfs etc support to write a chroot subtree straight into the fs | 00:04 |
| fungi | if _everything_ you need to put on the image is represented by debian packages, then mmdebstrap has some obvious wins | 00:04 |
| clarkb | this doesn't work with xfs though | 00:04 |
| clarkb | (which is another problem with the upstream images, we're stuck with their partitioning and fs choices) | 00:04 |
| clarkb | I'm not sure there are arm64 images we can use out of the box? | 00:06 |
| ianw | ubuntu does publish them | 00:06 |
| clarkb | in any case the rax networking "problem" is problem the first thing to figure out if this is somethign we want ot seriously consider | 00:07 |
| fungi | on an unrelated note, have others been following the [placement] zuul job dependencies thread on openstack-discuss? http://lists.openstack.org/pipermail/openstack-discuss/2019-February/003142.html | 00:08 |
| fungi | it's starting to veer into the "let's optimize job configuration for the benefit of everyone" danger zones | 00:09 |
| fungi | or at least it's no longer clear to me whether they're just debating the merits of doing this for the placement team's jobs | 00:10 |
| openstackgerrit | John Studarus proposed openstack-infra/irc-meetings master: Feb 2019 election updates https://review.openstack.org/639223 | 00:11 |
| openstackgerrit | John Studarus proposed openstack-infra/irc-meetings master: new meeting time per vote today https://review.openstack.org/639224 | 00:12 |
| fungi | oh, and zaneb had an interesting suggestion on a project-team-guide change of mine... is there any reason we never added workflow -2 as a sticky gate-blocking vote for core reviewers to use in place of a procedural code-review -2"? | 00:12 |
| clarkb | fungi: I glanced at it and honestly am getting a bit tired about the "node cost" argument for the ~.05% case | 00:12 |
| clarkb | I've explained multiple times with hard data where our resource consumption use actually is | 00:12 |
| clarkb | and very few people seem actually willing to optomize those resource hogs | 00:12 |
| fungi | clarkb: yes, but people want to believe there's some easy solution which avoids them needing to tackle those actual problems | 00:12 |
| fungi | or that's been my takeaway | 00:13 |
| fungi | i have refrained from chiming in on that thread for similar reasons of exhaustion | 00:13 |
| clarkb | re the gerrit thing, only the lowest vote could historically be blocking. Unsure if that has hcanged | 00:13 |
| clarkb | so your -1 workflow would no longer block | 00:13 |
| fungi | ahh, right | 00:13 |
| fungi | i forgot about that | 00:13 |
| clarkb | there may be a new function you can aplpy to the category that says an negative is a block | 00:14 |
| *** wolverineav has quit IRC | 00:15 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: executor: add log_stream_port and log_stream_file settings https://review.openstack.org/535538 | 00:15 |
| clarkb | http://paste.openstack.org/show/746083/ just ran that cpu resource utilization data collection if anyone does want to tackle it | 00:16 |
| *** eandersson has quit IRC | 00:16 | |
| *** eandersson_ has joined #openstack-infra | 00:17 | |
| fungi | clarkb: https://review.openstack.org/Documentation/config-labels.html doesn't seem to indicate any option for that in our deployment at any rate | 00:20 |
| cgoncalves | ianw, I'm installing pip_and_virtualenv from package (https://review.openstack.org/#/c/638581/2/diskimage-create/diskimage-create.sh@341) so yeah, I think that would work | 00:20 |
| clarkb | fungi: ya I don't see one either. So that is probably why. We want the WIP vote to block | 00:20 |
| *** wolverineav has joined #openstack-infra | 00:21 | |
| clarkb | fungi: WIP -1 could be used as a procedural blocker too? | 00:21 |
| clarkb | fungi: we could also set it to copy min score | 00:21 |
| clarkb | so you'd have to explicitly unwip it if that is what the procedural votes nee | 00:21 |
| openstackgerrit | Merged openstack-infra/system-config master: Set health service's vhost hostname explicitly https://review.openstack.org/639217 | 00:21 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Add new health01 server to our inventory https://review.openstack.org/639229 | 00:23 |
| fungi | yeah, problem is if we want one score which can carry over and have wip be reset on a new patchset, then we either need yet another label for it or maybe something something prolog rules something | 00:23 |
| clarkb | ianw fungi ^ you reviewed the first change this is the follow up I promised to actually puppet a new server | 00:23 |
| *** tosky has quit IRC | 00:23 | |
| clarkb | fungi: ya or just get people to manually unset WIP | 00:23 |
| fungi | sure | 00:24 |
| clarkb | also a really easy resource saver if we want it is to delete the py35 jobs | 00:25 |
| clarkb | but that doesn't even show up in the top 20 jobs list so is less than 1.19% of total usage | 00:26 |
| fungi | heh | 00:26 |
| * clarkb runs numbers to see if it is top 100 | 00:27 | |
| clarkb | openstack-tox-py35: 10149578.19s, 0.87% | 00:27 |
| clarkb | it is 26th | 00:28 |
| clarkb | 100th is 0.12% | 00:28 |
| clarkb | and 25th is the 1% threshold | 00:28 |
| fungi | wow, that's a great retort to the folks worried about removing py35 jobs before we add py37 | 00:28 |
| clarkb | fungi: I already made that back in december fwiw | 00:29 |
| *** hwoarang has quit IRC | 00:29 | |
| clarkb | when I told coreycb I was fine with adding py37 jobs because we don't run all that many py3x jobs anyway | 00:29 |
| fungi | the other argument of course has less to do with node utilization and more to do with job instability leading to more rechecks the more jobs a project runs | 00:29 |
| fungi | but again, that's mostly the realm of integration jobs | 00:30 |
| clarkb | ya there are other things to considerfor sure. And our python3 jobs do seemto be less stable than python2 | 00:30 |
| mordred | clarkb, fungi: new gerrit has a WIP plugin finally I think | 00:30 |
| fungi | mordred: oh, right! | 00:30 |
| fungi | Shrews has been vindicated, after all these long years | 00:30 |
| mordred | so maybe that would be the answer - and then workflow -1 can be a core-only procedural blocker | 00:30 |
| fungi | does the wip plugin also block merging? i guess it probably does | 00:31 |
| mordred | I'd hope so | 00:31 |
| *** markvoelker has joined #openstack-infra | 00:31 | |
| *** hwoarang has joined #openstack-infra | 00:32 | |
| clarkb | fungi ianw thanks for the reviews. I'll probably approve it tomorrow morning so that I can watch the puppetry | 00:34 |
| ianw | ++ | 00:35 |
| clarkb | ok I responded to the thread with a link to both elastic-recheck (figure out what fails) and the zuul resource usage paste (total aggregate utilization) | 00:43 |
| fungi | thanks! | 00:44 |
| clarkb | really quickly before I got make food for the kids. I've noticed that I've had to run unattended upgrades by hand post launch.py to install a bunch of things (and then reboot) | 00:45 |
| clarkb | I think this is something we must've lost with the changes to the launch script. Would anyone be opposed to me updating the launch script to run unattended upgrades as part of the bootstrapping as well as a reboot? | 00:46 |
| *** ijw has quit IRC | 00:46 | |
| clarkb | put another way is there a reason we wouldn't want to automate that? | 00:47 |
| *** ijw has joined #openstack-infra | 00:48 | |
| clarkb | also unrelated to that what do we think about alphebatizing zone files? or maybe sorting by record type? | 00:48 |
| clarkb | we'll still git conflict on the serial, but not needing to also fix conflicts in the zone "body" would be nice | 00:48 |
| *** ijw has quit IRC | 00:50 | |
| *** ijw has joined #openstack-infra | 00:50 | |
| *** gregoryo has joined #openstack-infra | 00:54 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Update UC Meeting time. https://review.openstack.org/639176 | 00:58 |
| ianw | clarkb: hrm, i guess that's just the equivalent of an "apt-get upgrade" at launch, right? that seems like something that should be done | 01:03 |
| *** markvoelker has quit IRC | 01:04 | |
| *** ijw has quit IRC | 01:04 | |
| *** Swami has quit IRC | 01:04 | |
| *** ijw has joined #openstack-infra | 01:07 | |
| clarkb | ya with our rules for unattendedness included | 01:09 |
| *** ijw has quit IRC | 01:10 | |
| *** ijw has joined #openstack-infra | 01:11 | |
| openstackgerrit | Merged openstack-infra/zuul master: scheduler: add job's variants to the rpc job_list method https://review.openstack.org/573473 | 01:11 |
| *** wolverineav has quit IRC | 01:18 | |
| *** eharney has quit IRC | 01:19 | |
| *** jaosorior has quit IRC | 01:20 | |
| *** hwoarang has quit IRC | 01:20 | |
| *** jamesmcarthur has joined #openstack-infra | 01:23 | |
| *** hwoarang has joined #openstack-infra | 01:24 | |
| *** ijw has quit IRC | 01:24 | |
| *** wolverineav has joined #openstack-infra | 01:24 | |
| *** yamamoto has joined #openstack-infra | 01:24 | |
| *** ijw has joined #openstack-infra | 01:25 | |
| *** yamamoto has quit IRC | 01:29 | |
| *** jaosorior has joined #openstack-infra | 01:32 | |
| *** mriedem is now known as mriedem_afk | 01:39 | |
| *** ijw has quit IRC | 01:39 | |
| *** jamesmcarthur has quit IRC | 01:49 | |
| *** hwoarang has quit IRC | 01:49 | |
| *** hwoarang has joined #openstack-infra | 01:51 | |
| *** hongbin has joined #openstack-infra | 01:55 | |
| *** ijw has joined #openstack-infra | 01:56 | |
| *** whoami-rajat has joined #openstack-infra | 02:01 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: executor: add zuul_release_nodeset Ansible action https://review.openstack.org/639002 | 02:05 |
| *** wolverineav has quit IRC | 02:05 | |
| tonyb | Is there somethign I'm missing on https://review.openstack.org/#/c/633049/ can it be +W'd ? | 02:08 |
| *** wolverineav has joined #openstack-infra | 02:08 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] rhel8 beta support https://review.openstack.org/623137 | 02:11 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [wip] shortcut pip-and-virtualenv for later redhat distros https://review.openstack.org/639220 | 02:11 |
| *** ijw has quit IRC | 02:16 | |
| *** calbers has quit IRC | 02:17 | |
| *** ijw has joined #openstack-infra | 02:17 | |
| *** ijw has quit IRC | 02:18 | |
| *** ijw has joined #openstack-infra | 02:19 | |
| *** jklare has quit IRC | 02:22 | |
| *** mriedem_afk has quit IRC | 02:24 | |
| *** jamesmcarthur has joined #openstack-infra | 02:25 | |
| *** wolverineav has quit IRC | 02:26 | |
| *** wolverineav has joined #openstack-infra | 02:27 | |
| *** mriedem has joined #openstack-infra | 02:29 | |
| *** wolverineav has quit IRC | 02:32 | |
| ianw | tonyb: i think it looks ok ... always scary merging :) personally i'd love a comment on what magic strings are matching in increasingly complex operations like that | 02:32 |
| *** ijw has quit IRC | 02:35 | |
| tonyb | ianw: Okay I'll try to get elod (ping) to add an inline comment to clear this out | 02:36 |
| tonyb | ianw: Thanks | 02:36 |
| ianw | tonyb: oh, i'm not saying no based on that, just my preference :) | 02:38 |
| tonyb | ianw: okay | 02:40 |
| *** ijw has joined #openstack-infra | 02:40 | |
| ianw | ok, i feel like i did due diligence :) i'm around should the worst happen and need a revert | 02:40 |
| tonyb | ianw: \o/ Thanks | 02:41 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Fix Editable Git install package name parsing https://review.openstack.org/633049 | 02:49 |
| *** jklare has joined #openstack-infra | 02:50 | |
| ianw | cgoncalves: oh cool, so without too much fuss looks like we've got good nodepool testing on the rhel8 image too -> http://logs.openstack.org/13/639013/3/check/nodepool-functional-py35-redhat-src/fd7aa34/ | 02:55 |
| *** psachin has joined #openstack-infra | 02:55 | |
| *** yamamoto has joined #openstack-infra | 03:00 | |
| *** markvoelker has joined #openstack-infra | 03:01 | |
| *** wolverineav has joined #openstack-infra | 03:02 | |
| *** ijw has quit IRC | 03:03 | |
| *** ijw has joined #openstack-infra | 03:06 | |
| *** wolverineav has quit IRC | 03:07 | |
| *** ijw has quit IRC | 03:11 | |
| *** mriedem has quit IRC | 03:12 | |
| *** roman_g has quit IRC | 03:19 | |
| *** ykarel has joined #openstack-infra | 03:19 | |
| *** jamesmcarthur has quit IRC | 03:25 | |
| *** jamesmcarthur has joined #openstack-infra | 03:28 | |
| *** rlandy has quit IRC | 03:28 | |
| *** hongbin has quit IRC | 03:33 | |
| *** markvoelker has quit IRC | 03:35 | |
| *** ramishra has joined #openstack-infra | 03:36 | |
| *** jamesmcarthur has quit IRC | 03:36 | |
| *** jamesmcarthur has joined #openstack-infra | 03:40 | |
| *** raukadah is now known as chandankumar | 03:41 | |
| *** ijw has joined #openstack-infra | 03:43 | |
| *** hwoarang has quit IRC | 03:44 | |
| *** jamesmcarthur has quit IRC | 03:45 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:46 | |
| *** hwoarang has joined #openstack-infra | 03:46 | |
| *** sdake has joined #openstack-infra | 03:49 | |
| *** sdake has quit IRC | 03:49 | |
| *** ijw_ has joined #openstack-infra | 03:54 | |
| *** udesale has joined #openstack-infra | 03:54 | |
| *** ijw has quit IRC | 03:56 | |
| *** yamamoto has quit IRC | 03:57 | |
| *** yamamoto has joined #openstack-infra | 03:58 | |
| *** gyee has quit IRC | 03:58 | |
| *** yamamoto has quit IRC | 04:00 | |
| *** sdake has joined #openstack-infra | 04:06 | |
| *** jamesmcarthur has joined #openstack-infra | 04:07 | |
| *** sdake has quit IRC | 04:10 | |
| *** sdake has joined #openstack-infra | 04:12 | |
| *** sdake has quit IRC | 04:13 | |
| *** sdake has joined #openstack-infra | 04:15 | |
| *** jamesmcarthur has quit IRC | 04:23 | |
| *** sdake has quit IRC | 04:23 | |
| *** hwoarang has quit IRC | 04:23 | |
| *** yamamoto has joined #openstack-infra | 04:25 | |
| *** hwoarang has joined #openstack-infra | 04:25 | |
| *** yamamoto has quit IRC | 04:26 | |
| *** yamamoto has joined #openstack-infra | 04:30 | |
| *** markvoelker has joined #openstack-infra | 04:32 | |
| *** wolverineav has joined #openstack-infra | 04:33 | |
| *** dave-mccowan has quit IRC | 04:44 | |
| *** markvoelker has quit IRC | 05:05 | |
| *** wolverineav has quit IRC | 05:09 | |
| *** wolverineav has joined #openstack-infra | 05:11 | |
| *** janki has joined #openstack-infra | 05:26 | |
| *** ramishra has quit IRC | 05:32 | |
| *** bhavikdbavishi has quit IRC | 05:35 | |
| *** ramishra has joined #openstack-infra | 05:38 | |
| *** ijw_ has quit IRC | 05:57 | |
| *** wolverineav has quit IRC | 06:01 | |
| *** wolverineav has joined #openstack-infra | 06:02 | |
| *** markvoelker has joined #openstack-infra | 06:03 | |
| *** wolverineav has quit IRC | 06:07 | |
| *** wolverineav has joined #openstack-infra | 06:14 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Add API endpoint to get frozen jobs https://review.openstack.org/607077 | 06:33 |
| *** markvoelker has quit IRC | 06:35 | |
| *** jchhatbar has joined #openstack-infra | 06:43 | |
| *** janki has quit IRC | 06:45 | |
| *** wolverineav has quit IRC | 07:06 | |
| *** wolverineav has joined #openstack-infra | 07:06 | |
| *** wolverineav has quit IRC | 07:13 | |
| *** pgaxatte has joined #openstack-infra | 07:21 | |
| *** kopecmartin|off is now known as kopecmartin | 07:23 | |
| *** jtomasek has joined #openstack-infra | 07:25 | |
| *** dpawlik has joined #openstack-infra | 07:30 | |
| *** ramishra has quit IRC | 07:31 | |
| *** ramishra has joined #openstack-infra | 07:32 | |
| *** aojea has joined #openstack-infra | 07:35 | |
| *** logan- has quit IRC | 07:35 | |
| *** logan- has joined #openstack-infra | 07:37 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Add API endpoint to get frozen jobs https://review.openstack.org/607077 | 07:44 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Get executor job params https://review.openstack.org/607078 | 07:44 |
| *** wolverineav has joined #openstack-infra | 07:47 | |
| *** wolverineav has quit IRC | 07:52 | |
| *** ykarel is now known as ykarel|lunch | 07:56 | |
| *** rcernin has quit IRC | 07:56 | |
| *** kjackal has joined #openstack-infra | 08:01 | |
| *** ginopc has joined #openstack-infra | 08:01 | |
| *** ccamacho has joined #openstack-infra | 08:02 | |
| *** quiquell is now known as quiquell|brb | 08:03 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Add depends-on support to frozen jobs API https://review.openstack.org/639022 | 08:06 |
| *** rascasoft has joined #openstack-infra | 08:08 | |
| *** tkajinam has quit IRC | 08:16 | |
| *** shardy has joined #openstack-infra | 08:16 | |
| *** bhavikdbavishi has joined #openstack-infra | 08:17 | |
| *** helenafm has joined #openstack-infra | 08:20 | |
| *** quiquell|brb is now known as quiquell | 08:23 | |
| *** wolverineav has joined #openstack-infra | 08:29 | |
| *** markvoelker has joined #openstack-infra | 08:32 | |
| *** roman_g has joined #openstack-infra | 08:32 | |
| *** wolverineav has quit IRC | 08:33 | |
| *** pcaruana has joined #openstack-infra | 08:35 | |
| *** electrofelix has joined #openstack-infra | 08:37 | |
| *** tosky has joined #openstack-infra | 08:39 | |
| *** lathiat_ is now known as lathiat | 08:39 | |
| *** jpena|off is now known as jpena | 08:42 | |
| *** wolverineav has joined #openstack-infra | 08:44 | |
| *** dtantsur|afk is now known as dtantsur | 08:46 | |
| *** wolverineav has quit IRC | 08:49 | |
| *** jpich has joined #openstack-infra | 08:53 | |
| *** bhavikdbavishi has quit IRC | 08:54 | |
| *** shardy has quit IRC | 08:54 | |
| *** ccamacho has quit IRC | 08:58 | |
| *** priteau has joined #openstack-infra | 08:59 | |
| *** markvoelker has quit IRC | 09:06 | |
| *** ykarel|lunch is now known as ykarel | 09:10 | |
| *** jchhatbar has quit IRC | 09:16 | |
| *** jchhatbar has joined #openstack-infra | 09:16 | |
| *** gregoryo has quit IRC | 09:17 | |
| *** jchhatbar has quit IRC | 09:22 | |
| *** jchhatbar has joined #openstack-infra | 09:23 | |
| *** dklyle has quit IRC | 09:25 | |
| *** e0ne has joined #openstack-infra | 09:25 | |
| *** calbers has joined #openstack-infra | 09:26 | |
| *** benj_- has joined #openstack-infra | 09:26 | |
| *** dklyle has joined #openstack-infra | 09:27 | |
| *** elod_ has joined #openstack-infra | 09:29 | |
| *** auristor has quit IRC | 09:30 | |
| *** FlorianFa has quit IRC | 09:30 | |
| *** elod has quit IRC | 09:30 | |
| *** benj_ has quit IRC | 09:30 | |
| *** tbarron has quit IRC | 09:30 | |
| *** elod_ is now known as elod | 09:30 | |
| *** otherwiseguy has quit IRC | 09:32 | |
| *** otherwiseguy has joined #openstack-infra | 09:32 | |
| *** ekultails has quit IRC | 09:35 | |
| *** jamesdenton has quit IRC | 09:35 | |
| *** ekultails has joined #openstack-infra | 09:35 | |
| *** FlorianFa has joined #openstack-infra | 09:35 | |
| *** jamesdenton has joined #openstack-infra | 09:36 | |
| *** lucasagomes has joined #openstack-infra | 09:37 | |
| *** ccamacho has joined #openstack-infra | 09:39 | |
| *** wolverineav has joined #openstack-infra | 09:41 | |
| *** yboaron_ has joined #openstack-infra | 09:45 | |
| *** roman_g has quit IRC | 09:45 | |
| *** derekh has joined #openstack-infra | 09:46 | |
| *** yboaron_ has quit IRC | 09:53 | |
| *** yboaron_ has joined #openstack-infra | 09:53 | |
| *** yamamoto has quit IRC | 09:53 | |
| *** priteau has quit IRC | 09:57 | |
| *** priteau has joined #openstack-infra | 09:58 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Turn on the future parser for ask.o.o https://review.openstack.org/585206 | 10:02 |
| *** markvoelker has joined #openstack-infra | 10:03 | |
| *** bhavikdbavishi has joined #openstack-infra | 10:04 | |
| *** dpawlik has quit IRC | 10:05 | |
| *** priteau has quit IRC | 10:06 | |
| *** yamamoto has joined #openstack-infra | 10:10 | |
| *** ricolin has joined #openstack-infra | 10:10 | |
| mordred | ianw: how are we managing to test a rhel image? I guess we don't actually need to apply a license to it for what we're doing? | 10:11 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Separate out executor server from runner https://review.openstack.org/607079 | 10:14 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: zuul-runner: implement prep-workspace https://review.openstack.org/607082 | 10:14 |
| *** wolverineav has quit IRC | 10:14 | |
| mordred | ianw: also - I posted 2 small nits on your dib patch. cool work there! | 10:15 |
| *** yamamoto has quit IRC | 10:21 | |
| *** yamamoto has joined #openstack-infra | 10:26 | |
| *** markvoelker has quit IRC | 10:36 | |
| *** luizbag has joined #openstack-infra | 10:52 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul master: Get executor job params https://review.openstack.org/607078 | 10:54 |
| *** bhavikdbavishi has quit IRC | 10:56 | |
| *** bhavikdbavishi has joined #openstack-infra | 10:56 | |
| *** gfidente has joined #openstack-infra | 11:02 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul master: Get executor job params https://review.openstack.org/607078 | 11:08 |
| *** bhavikdbavishi1 has joined #openstack-infra | 11:12 | |
| *** bhavikdbavishi has quit IRC | 11:16 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 11:16 | |
| *** udesale has quit IRC | 11:29 | |
| *** Adri2000 has quit IRC | 11:33 | |
| *** Adri2000 has joined #openstack-infra | 11:34 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul master: Get executor job params https://review.openstack.org/607078 | 11:35 |
| *** dpawlik has joined #openstack-infra | 11:37 | |
| *** kjackal has quit IRC | 11:39 | |
| *** psachin has quit IRC | 11:40 | |
| *** wolverineav has joined #openstack-infra | 11:41 | |
| *** jamesmcarthur has joined #openstack-infra | 11:44 | |
| *** wolverineav has quit IRC | 11:46 | |
| *** bhavikdbavishi has quit IRC | 11:46 | |
| *** ccamacho has quit IRC | 11:50 | |
| *** jpena is now known as jpena|lunch | 11:57 | |
| fungi | maintenance to move the backend databases openstackid.org uses to vexxhost is underway | 12:02 |
| fungi | should be effectively hitless | 12:02 |
| *** quiquell is now known as quiquell|lunch | 12:04 | |
| JpMaxMan | fungi: +1 :) | 12:05 |
| openstackgerrit | Merged openstack-infra/system-config master: Updated openstackid prod node https://review.openstack.org/639190 | 12:06 |
| fungi | that's ^ the configuration management adjustment | 12:06 |
| fungi | i have the associated hiera edits ready to hit save and can kick a configuration management pulse for the openstackid server as soon as the db is live at the remote end | 12:07 |
| openstackgerrit | Slawek Kaplonski proposed openstack-infra/project-config master: Move openstack-tox-lower-constraints to UT jobs graph https://review.openstack.org/639321 | 12:07 |
| *** yamamoto has quit IRC | 12:10 | |
| *** rfolco|rover has quit IRC | 12:11 | |
| *** rfolco|rover has joined #openstack-infra | 12:11 | |
| *** ccamacho has joined #openstack-infra | 12:12 | |
| *** janki has joined #openstack-infra | 12:14 | |
| *** jchhatbar has quit IRC | 12:15 | |
| *** quiquell|lunch is now known as quiquell | 12:15 | |
| *** yamamoto has joined #openstack-infra | 12:15 | |
| *** roman_g has joined #openstack-infra | 12:23 | |
| fungi | using kick.sh to apply config changes to openstackid01 now | 12:23 |
| *** janki has quit IRC | 12:24 | |
| *** janki has joined #openstack-infra | 12:24 | |
| *** aojeagarcia has joined #openstack-infra | 12:25 | |
| *** rf0lc0 has joined #openstack-infra | 12:26 | |
| *** shachar has joined #openstack-infra | 12:26 | |
| *** rkukura_ has joined #openstack-infra | 12:26 | |
| *** manfred has joined #openstack-infra | 12:27 | |
| *** logan_ has joined #openstack-infra | 12:27 | |
| *** logan_ is now known as Guest11647 | 12:28 | |
| *** dmellado_ has joined #openstack-infra | 12:28 | |
| *** kjackal has joined #openstack-infra | 12:29 | |
| *** janki has quit IRC | 12:29 | |
| *** beekneemech has joined #openstack-infra | 12:29 | |
| *** pgaxatte1 has joined #openstack-infra | 12:30 | |
| *** Shrews_ has joined #openstack-infra | 12:30 | |
| *** diablo_rojo_ has joined #openstack-infra | 12:30 | |
| *** bgmccollum has quit IRC | 12:30 | |
| *** armax has quit IRC | 12:30 | |
| *** gouthamr has quit IRC | 12:30 | |
| *** electrofelix has quit IRC | 12:30 | |
| *** kaiokmo has quit IRC | 12:30 | |
| *** dayou has quit IRC | 12:30 | |
| *** jento has quit IRC | 12:30 | |
| *** kaisers has quit IRC | 12:30 | |
| *** cgoncalves has quit IRC | 12:30 | |
| *** dpawlik has quit IRC | 12:30 | |
| *** yboaron_ has quit IRC | 12:30 | |
| *** owalsh has quit IRC | 12:30 | |
| *** Shrews has quit IRC | 12:30 | |
| *** jlvillal has quit IRC | 12:30 | |
| *** gtmanfred has quit IRC | 12:30 | |
| *** manfred is now known as gtmanfred | 12:30 | |
| *** pgaxatte has quit IRC | 12:30 | |
| *** dosaboy has quit IRC | 12:30 | |
| *** dmellado has quit IRC | 12:30 | |
| *** zzzeek has quit IRC | 12:30 | |
| *** rfolco|rover has quit IRC | 12:30 | |
| *** gfidente has quit IRC | 12:30 | |
| *** logan- has quit IRC | 12:30 | |
| *** aojea has quit IRC | 12:30 | |
| *** ramishra has quit IRC | 12:30 | |
| *** diablo_rojo has quit IRC | 12:30 | |
| *** rkukura has quit IRC | 12:30 | |
| *** bnemec has quit IRC | 12:30 | |
| *** snapiri has quit IRC | 12:30 | |
| *** pcrews has quit IRC | 12:30 | |
| *** openstackgerrit has quit IRC | 12:30 | |
| *** rkukura_ is now known as rkukura | 12:30 | |
| *** dmellado_ is now known as dmellado | 12:30 | |
| *** yboaron_ has joined #openstack-infra | 12:30 | |
| *** owalsh_ has joined #openstack-infra | 12:31 | |
| *** gfidente has joined #openstack-infra | 12:31 | |
| *** dpawlik has joined #openstack-infra | 12:31 | |
| *** zzzeek has joined #openstack-infra | 12:31 | |
| *** Guest11647 is now known as logan- | 12:31 | |
| *** jlvillal has joined #openstack-infra | 12:31 | |
| *** dayou has joined #openstack-infra | 12:31 | |
| *** kaisers has joined #openstack-infra | 12:32 | |
| *** cgoncalves has joined #openstack-infra | 12:32 | |
| *** gouthamr has joined #openstack-infra | 12:33 | |
| *** markvoelker has joined #openstack-infra | 12:33 | |
| *** rf0lc0 is now known as rfolco|rover | 12:33 | |
| *** jamesmcarthur has quit IRC | 12:34 | |
| *** jamesmcarthur has joined #openstack-infra | 12:34 | |
| *** electrofelix has joined #openstack-infra | 12:35 | |
| *** ramishra_ has joined #openstack-infra | 12:38 | |
| fungi | everything seems to have gone fine with the openstackid.org database cut-over | 12:43 |
| *** kjackal has quit IRC | 12:49 | |
| *** dpawlik has quit IRC | 12:50 | |
| *** mriedem has joined #openstack-infra | 12:50 | |
| *** auristor has joined #openstack-infra | 12:51 | |
| *** jamesmcarthur has quit IRC | 12:52 | |
| *** ccamacho has quit IRC | 12:52 | |
| *** jamesmcarthur has joined #openstack-infra | 12:57 | |
| *** gfidente has quit IRC | 13:04 | |
| *** ccamacho has joined #openstack-infra | 13:05 | |
| *** markvoelker has quit IRC | 13:06 | |
| *** Adri2000 has quit IRC | 13:08 | |
| *** sdake has joined #openstack-infra | 13:15 | |
| *** dpawlik has joined #openstack-infra | 13:19 | |
| *** owalsh_ is now known as owalsh | 13:19 | |
| *** trown|outtypewww is now known as trown | 13:24 | |
| *** agopi has quit IRC | 13:29 | |
| *** kjackal has joined #openstack-infra | 13:30 | |
| *** jpena|lunch is now known as jpena | 13:33 | |
| *** rlandy has joined #openstack-infra | 13:33 | |
| *** jamesmcarthur has quit IRC | 13:35 | |
| *** dtantsur is now known as dtantsur|afk | 13:38 | |
| *** jamesmcarthur has joined #openstack-infra | 13:39 | |
| *** jamesmcarthur has quit IRC | 13:40 | |
| *** jamesmcarthur has joined #openstack-infra | 13:40 | |
| *** yamamoto has quit IRC | 13:40 | |
| *** yamamoto has joined #openstack-infra | 13:40 | |
| *** dosaboy has joined #openstack-infra | 13:51 | |
| *** dave-mccowan has joined #openstack-infra | 13:53 | |
| guilhermesp | corvus: hey! I'm from vexxhost. Just to let you know that we are working on that cloud-init issue in sjc1. I'll get back to you as soon as I fix the issue! | 13:55 |
| *** agopi has joined #openstack-infra | 13:58 | |
| *** beekneemech is now known as bnemec | 13:59 | |
| fungi | guilhermesp: thanks for the update! | 14:00 |
| guilhermesp | fungi: np! We are close to get this fixed! | 14:00 |
| *** ginopc has quit IRC | 14:11 | |
| *** ykarel is now known as ykarel|away | 14:13 | |
| *** jcoufal has joined #openstack-infra | 14:18 | |
| *** kgiusti has joined #openstack-infra | 14:21 | |
| *** sdake has quit IRC | 14:21 | |
| *** gfidente has joined #openstack-infra | 14:24 | |
| *** ginopc has joined #openstack-infra | 14:29 | |
| *** sdake has joined #openstack-infra | 14:33 | |
| *** dave-mccowan has quit IRC | 14:34 | |
| *** bgmccollum has joined #openstack-infra | 14:37 | |
| *** eharney has joined #openstack-infra | 14:46 | |
| *** ykarel|away is now known as ykarel | 14:46 | |
| *** sdake has quit IRC | 14:55 | |
| *** jento has joined #openstack-infra | 14:58 | |
| *** ykarel_ has joined #openstack-infra | 14:59 | |
| *** sdake has joined #openstack-infra | 15:00 | |
| *** ykarel has quit IRC | 15:00 | |
| *** ijw has joined #openstack-infra | 15:05 | |
| *** markvoelker has joined #openstack-infra | 15:07 | |
| *** kjackal has quit IRC | 15:07 | |
| *** sreejithp has joined #openstack-infra | 15:08 | |
| *** kjackal has joined #openstack-infra | 15:09 | |
| *** jamesmcarthur has quit IRC | 15:11 | |
| *** jamesmcarthur has joined #openstack-infra | 15:11 | |
| *** eharney has quit IRC | 15:13 | |
| *** eharney has joined #openstack-infra | 15:15 | |
| *** jamesmcarthur has quit IRC | 15:15 | |
| *** e0ne has quit IRC | 15:16 | |
| *** owalsh_ has joined #openstack-infra | 15:24 | |
| *** owalsh has quit IRC | 15:25 | |
| chandankumar | AJaeger: clarkb we have got the ack from telemetry team https://review.openstack.org/#/c/636093/ please have a look when free! | 15:26 |
| *** ijw has quit IRC | 15:26 | |
| *** owalsh_ is now known as owalsh | 15:26 | |
| *** jamesmcarthur has joined #openstack-infra | 15:27 | |
| *** jamesmcarthur has quit IRC | 15:30 | |
| *** ykarel_ is now known as ykarel|away | 15:32 | |
| *** yamamoto has quit IRC | 15:37 | |
| *** markvoelker has quit IRC | 15:37 | |
| *** yamamoto has joined #openstack-infra | 15:38 | |
| *** yamamoto has quit IRC | 15:38 | |
| *** yamamoto has joined #openstack-infra | 15:39 | |
| *** yamamoto has quit IRC | 15:39 | |
| *** jamesmcarthur has joined #openstack-infra | 15:39 | |
| clarkb | fungi: thank you for taking care of that this morning | 15:43 |
| fungi | there was really nothing to do. it was approving a simple change to the global site manifest, making a few hostvar edits on bridge.o.o and running kick.sh | 15:46 |
| fungi | went very smoothly | 15:46 |
| fungi | i could probably have skipped the last step, but wanted slightly faster feedback to make sure it was working as expected | 15:47 |
| *** bhavikdbavishi has joined #openstack-infra | 15:49 | |
| *** ccamacho has quit IRC | 15:51 | |
| *** ricolin has quit IRC | 15:52 | |
| *** e0ne has joined #openstack-infra | 15:53 | |
| *** ccamacho has joined #openstack-infra | 15:55 | |
| clarkb | infra-root https://review.openstack.org/#/c/585206/3 is the last futureparser change. Anyone else care to review it really quick? I can watch it go in | 15:56 |
| clarkb | then with that out of the way we should be ready to start doing puppet4 upgrades fairly broadly (etherpad-dev01 is the only puppet4 host so far) | 15:56 |
| cmurphy | \o/ | 15:59 |
| *** sdake has quit IRC | 15:59 | |
| *** sdake has joined #openstack-infra | 16:01 | |
| *** sdake has quit IRC | 16:02 | |
| *** ijw has joined #openstack-infra | 16:03 | |
| corvus | clarkb, fungi, mordred: can you look over the list of tasks in https://etherpad.openstack.org/p/Q3MZJNcmbK and let me know if that looks right? | 16:05 |
| corvus | if it does, i'll move it into storyboard so we can ask folks to sign up | 16:05 |
| corvus | (that's mostly drawn from the spec, but updated slightly) | 16:05 |
| mordred | corvus: yes. I believe that looks good | 16:07 |
| tosky | did anyone notice that the notification bot disappeared? | 16:07 |
| clarkb | corvus: do we want to point git.foo to files and not the gitea appropriate location? | 16:07 |
| clarkb | tosky: the gerritbot? it has too many channels to connect to so connects in a least recently used manner round robin as necessary | 16:08 |
| clarkb | tosky: if there are events to send to your channel it should join and send them | 16:08 |
| corvus | clarkb: i was thinking that git.foo becomes a vhost on files, which redirects to location | 16:08 |
| clarkb | corvus: gotcha | 16:08 |
| clarkb | oh I see the htaccess goes to files | 16:08 |
| *** yboaron_ has quit IRC | 16:09 | |
| tosky | clarkb: I didn't see any notification for sahara reviews (#openstack-sahara) and devstack and grenade (#openstack-qa) in the last few hours | 16:09 |
| corvus | clarkb: yeah, based on this: http://files.openstack.org/project/git-test/www/.htaccess | 16:09 |
| clarkb | tosky: 2019-02-26 16:10:11,647 INFO gerritbot: Sending "Merged openstack-infra/system-config master: Add new health01 server to our inventory https://review.openstack.org/639229" to #openstack-infra | 16:10 |
| fungi | we may need to refresh the .zuul.yaml changes if we do them before we even announce | 16:10 |
| clarkb | tosky: as you can probably tell that message hasn't arrived here | 16:10 |
| clarkb | tosky: so possible it ended up in a netsplit | 16:10 |
| tosky | oh, right | 16:11 |
| tosky | that's possible | 16:11 |
| fungi | 2019-02-26 12:30:30 <-- openstackgerrit (~openstack@review01.openstack.org) has quit (Ping timeout: 255 seconds) | 16:11 |
| *** dpawlik has quit IRC | 16:11 | |
| clarkb | corvus: the list looks good | 16:11 |
| tosky | I see few people leaving at the same time with the same ping timeout | 16:12 |
| fungi | likely a netslipt or irc server outage, yeah | 16:12 |
| *** Adri2000 has joined #openstack-infra | 16:13 | |
| fungi | there's a good chance it won't recover on its own. i can restart gerritbot unless someone thinks i shouldn't | 16:14 |
| clarkb | fungi: probably the best bet at this point | 16:14 |
| fungi | done | 16:15 |
| fungi | it just joined #openstack nova and announced https://review.openstack.org/639159 | 16:16 |
| fungi | so seems to be back to normal | 16:16 |
| fungi | #status log restarted the gerritbot service on review01 to resolve its 12:30:30 ping timeout | 16:17 |
| openstackstatus | fungi: finished logging | 16:17 |
| *** sdake has joined #openstack-infra | 16:18 | |
| *** yamamoto has joined #openstack-infra | 16:19 | |
| corvus | clarkb, fungi, mordred, ianw: i put the opendev-gerrit tasks in storyboard: https://storyboard.openstack.org/#!/story/2004627 | 16:23 |
| *** yamamoto has quit IRC | 16:24 | |
| fungi | corvus: thanks! it looks like we're missing a step to actually generate the "prepared changes" we need to merge, but i guess it can really just be part of that step to generate and propose those with the script we make | 16:25 |
| *** sdake has quit IRC | 16:25 | |
| corvus | fungi: yeah, i think that needs to happen right before the outage, so i'll stick it in the outage plan | 16:26 |
| *** sdake has joined #openstack-infra | 16:27 | |
| fungi | agreed, as i mentioned above i expect particularly the .zuul.yaml files to drift pretty quickly if we try to do it ahead of time | 16:27 |
| clarkb | the logstash pipeline is still up and running this morning (re fears that jobs would continue to make it fall over) | 16:27 |
| corvus | fungi: yeah. i updated that to "Run the script generate zuul.yaml and gitreview change and force-merge them" | 16:27 |
| clarkb | it is possible that those exceptionally sized error log files were related to a specific hcange or two (should still address it if possible but may not be a persistent thing) | 16:28 |
| corvus | fungi: yeah. i updated that to "Run the script to generate zuul.yaml and gitreview changes and force-merge them" | 16:28 |
| *** dave-mccowan has joined #openstack-infra | 16:28 | |
| fungi | wfm, thanks! | 16:28 |
| *** quiquell is now known as quiquell|off | 16:28 | |
| *** pcaruana has quit IRC | 16:32 | |
| *** markvoelker has joined #openstack-infra | 16:35 | |
| *** sdake has quit IRC | 16:35 | |
| *** sdake has joined #openstack-infra | 16:36 | |
| *** openstackgerrit has joined #openstack-infra | 16:37 | |
| openstackgerrit | Merged openstack-infra/system-config master: Turn on the future parser for ask.o.o https://review.openstack.org/585206 | 16:37 |
| corvus | openstack.org is down | 16:38 |
| corvus | which is causing the openstack theming on lists.openstack.org to be broken. in case anyone was wondering. | 16:38 |
| *** sdake has quit IRC | 16:40 | |
| *** ykarel|away has quit IRC | 16:41 | |
| *** sdake_ has joined #openstack-infra | 16:41 | |
| mordred | corvus: well poop | 16:41 |
| *** ykarel has joined #openstack-infra | 16:42 | |
| *** sdake_ has quit IRC | 16:45 | |
| *** sdake has joined #openstack-infra | 16:46 | |
| *** pgaxatte1 has quit IRC | 16:50 | |
| *** sdake has quit IRC | 16:50 | |
| fungi | is it still down? | 16:51 |
| *** sdake has joined #openstack-infra | 16:51 | |
| fungi | seems up. i wonder if they missed copying some specific assets | 16:51 |
| fungi | yeah, confirmed the theming issue by force-refreshing my browser. i guess i had some of those files cached | 16:55 |
| fungi | i can't tell which files are missing though. tried following all the css urls in the page source and i get content for them | 16:55 |
| clarkb | are the headers preventing the other domains from loading them? | 16:56 |
| *** gfidente is now known as gfidente|afk | 16:56 | |
| *** kjackal has quit IRC | 16:56 | |
| fungi | oh, perhaps | 16:57 |
| *** kjackal has joined #openstack-infra | 16:57 | |
| fungi | jamesmcarthur: ^ you can see this on http://lists.openstack.org/pipermail/openstack-discuss/ but may need to force refresh | 16:57 |
| *** helenafm has quit IRC | 16:58 | |
| clarkb | I don't see headers that would prevent that, possible the css changed? | 16:58 |
| clarkb | and so no longer applies? | 16:59 |
| clarkb | oh wait I misread firefox debugger. I get 0 bytes back? | 16:59 |
| corvus | clarkb: yeah, that's what i was seeing | 17:00 |
| corvus | http://openstack.org/ is still down | 17:00 |
| corvus | https://www.openstack.org/ is up | 17:00 |
| fungi | OH! | 17:00 |
| *** wolverineav has joined #openstack-infra | 17:00 | |
| *** sdake has quit IRC | 17:00 | |
| *** ccamacho has quit IRC | 17:01 | |
| jamesmcarthur | ah ha | 17:02 |
| *** sdake_ has joined #openstack-infra | 17:03 | |
| jamesmcarthur | Give us a few. Thanks for the heads up corvus! | 17:03 |
| corvus | jamesmcarthur: np! | 17:03 |
| *** markvoelker has quit IRC | 17:07 | |
| openstackgerrit | Tobias Urdin proposed openstack-infra/zuul-jobs master: Use PDK to build puppet module https://review.openstack.org/627534 | 17:12 |
| *** gyee has joined #openstack-infra | 17:13 | |
| jamesmcarthur | ok - corvus: this should be fixed. openstack.org was still pointed to teh old RAX servers, so when I shut them down, things went ka boom | 17:16 |
| jamesmcarthur | All set now. | 17:16 |
| *** sshnaidm is now known as sshnaidm|afk | 17:17 | |
| corvus | jamesmcarthur: looks good, thanks! | 17:17 |
| jamesmcarthur | np, thank you again for raising the flag | 17:17 |
| *** jpich has quit IRC | 17:17 | |
| clarkb | hrm there are type errors on ask future parser puppeting | 17:19 |
| clarkb | I'll work out a fix after some breakfast | 17:19 |
| clarkb | Feb 26 17:16:13 ask puppet-user[3086]: This Type-Name has no effect. A value-producing expression without other effect may only be placed last in a block/sequence at /etc/puppet/modules/postgresql/manifests/server/database.pp:73:3 on node ask.openstack.org | 17:20 |
| clarkb | if anyone is curious | 17:20 |
| clarkb | and health01 server failed to install python things because it couldn't find pip | 17:21 |
| clarkb | I wonder if that is a bootstrapping thing and next run will fix it | 17:21 |
| *** e0ne has quit IRC | 17:24 | |
| fungi | yeah, sounds like an ordering issue | 17:24 |
| clarkb | oh the puppet thing is a postgresql module issue | 17:26 |
| clarkb | that just got more interesting | 17:26 |
| fungi | we also seem to generate exim panic errors during bootstrapping | 17:26 |
| fungi | User 0 set for local_delivery transport is on the never_users list | 17:26 |
| fungi | cron is going to e-mail us daily until someone deletes the paniclog | 17:27 |
| clarkb | in the health case pip is installed so I think this is a race between installing pip which that manifest does and the path being checked to find if pip is installed | 17:29 |
| clarkb | we can likely fix this with a fully rooted path to pip | 17:30 |
| openstackgerrit | Tobias Urdin proposed openstack-infra/project-config master: Remove bindep role for Puppet build job https://review.openstack.org/627566 | 17:34 |
| *** aojeagarcia__ has joined #openstack-infra | 17:37 | |
| openstackgerrit | Merged openstack-infra/storyboard master: Add an interface specifying the API for storage backends https://review.openstack.org/633364 | 17:39 |
| *** aojeagarcia has quit IRC | 17:40 | |
| *** yamamoto has joined #openstack-infra | 17:41 | |
| *** jmorgan1 has joined #openstack-infra | 17:45 | |
| clarkb | https://github.com/puppetlabs/puppetlabs-postgresql/commit/289de739c3c92e58652529456a4c8cff5fb2eecd#diff-dc590b12ae31eaf043f7f56780b16b88 is the fix for the askbot thing | 17:45 |
| *** yamamoto has quit IRC | 17:46 | |
| fungi | oh, huh | 17:46 |
| clarkb | and you thought whitespace wasn't significant in puppet :P | 17:46 |
| clarkb | I certainly had that impression | 17:46 |
| * fungi shakes fist at inscrutable whitespace gods | 17:47 | |
| fungi | also, am i the only one bothered by calling it "whitespace" when the spaces in my terminal are all black? | 17:48 |
| clarkb | there are big scary warnings in the readme for that module upgrading from v2 to v3 but nothing for v3 to v4 | 17:49 |
| fungi | best not to think about it or i'll just end up pondering the mysteries of the universe | 17:49 |
| clarkb | we may be able to update teh version and call it a day | 17:49 |
| corvus | it's redspace in gerrit | 17:49 |
| * fungi thinks redspace is somewhere you really don't want to accidentally end up when your starship falls through a wormhole | 17:50 | |
| cmurphy | if you're not already aware ianw already has a change up to upgrade puppetlabs-postgresql https://review.openstack.org/558995 | 17:52 |
| *** kjackal has quit IRC | 17:56 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Request child nodes of paused jobs at higher priority https://review.openstack.org/638796 | 17:56 |
| *** trown is now known as trown|lunch | 17:57 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Update postgres module to 4.3.0 https://review.openstack.org/639404 | 17:57 |
| clarkb | cmurphy: oh I wasn't aware | 17:57 |
| clarkb | ah ianw has found an even better version to jump to | 17:58 |
| clarkb | I've rechecked it so that we get up to date tests but will approve if those come back happy | 17:58 |
| clarkb | and now to abandon my change | 17:58 |
| *** wolverineav has quit IRC | 18:02 | |
| *** wolverineav has joined #openstack-infra | 18:03 | |
| *** wolverineav has quit IRC | 18:03 | |
| clarkb | cmurphy: https://github.com/puppetlabs/puppetlabs-postgresql/blob/4.9.0/CHANGELOG.md#bugfixes the usernames and passwords are now converted to strings bugfix there. Do you expect that to be important for puppet4? | 18:07 |
| clarkb | https://github.com/puppetlabs/puppetlabs-postgresql/commit/e69308b4c1baf9adf02c4e20cff3616f221b9439 | 18:11 |
| clarkb | I'm guessing we are ok if the inputs we use are strings (which I believe we do) | 18:11 |
| *** yamamoto has joined #openstack-infra | 18:13 | |
| *** kopecmartin is now known as kopecmartin|off | 18:14 | |
| *** jpena is now known as jpena|off | 18:15 | |
| *** chandankumar is now known as raukadah | 18:16 | |
| *** yamamoto has quit IRC | 18:18 | |
| *** wolverineav has joined #openstack-infra | 18:18 | |
| clarkb | The pip installs run refreshonly so they didn't rerun on the last puppet pass. I've rm'd the two git repos in /opt to force it to reclone and reinstall finding pip | 18:21 |
| *** wolverineav has quit IRC | 18:22 | |
| *** wolverineav has joined #openstack-infra | 18:22 | |
| *** wolverineav has quit IRC | 18:23 | |
| *** derekh has quit IRC | 18:25 | |
| *** wolverineav has joined #openstack-infra | 18:26 | |
| clarkb | I think the puppet python module isn't doing quite what we want there (installing pip and making it avaialble to the currently running pupept run) | 18:26 |
| clarkb | but fully rooted paths should still fix that | 18:26 |
| *** panda|ruck is now known as panda|ruck|off | 18:28 | |
| *** sdake_ has quit IRC | 18:30 | |
| *** sdake has joined #openstack-infra | 18:33 | |
| *** sdake has quit IRC | 18:35 | |
| cmurphy | clarkb: i think that won't matter for puppet3 vs 4, it's internal ruby typing not puppet dsl typing | 18:36 |
| cmurphy | should only make a difference if your postgresql username or password was an integer | 18:37 |
| *** sdake has joined #openstack-infra | 18:39 | |
| clarkb | zuul still says +1 on that 4.8.0 change. I'll try to approve it after our meeting (but won't do it now because it would be distracting if that happened during the meeting) | 18:40 |
| *** sdake has quit IRC | 18:46 | |
| *** sdake_ has joined #openstack-infra | 18:46 | |
| *** aojeagarcia__ has quit IRC | 18:46 | |
| *** rfolco|rover has quit IRC | 18:46 | |
| *** sean-k-mooney has joined #openstack-infra | 18:49 | |
| sean-k-mooney | o/ | 18:50 |
| fungi | howdy | 18:50 |
| *** sdake_ has quit IRC | 18:50 | |
| *** Swami has joined #openstack-infra | 18:50 | |
| sean-k-mooney | i noticed two jobs fail due to missing cffi.h so tox could not compile the wheel is that a random one off or is this a know gate issue? | 18:51 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Put health servers in the puppet health group https://review.openstack.org/639415 | 18:51 |
| clarkb | infra-root ^ is a needed fix for the puppeting of the new health01 server | 18:51 |
| clarkb | I'm setting up the hieradata now | 18:51 |
| *** sdake has joined #openstack-infra | 18:52 | |
| fungi | sean-k-mooney: my guess is some project added a (perhaps transitive) dependency on cffi and lacks a bindep.txt entry for a package with the libffi headers | 18:53 |
| fungi | sean-k-mooney: got a link to the failure? | 18:54 |
| clarkb | fungi: sean-k-mooney: it could also be a package updated with an sdist but not a wheel | 18:54 |
| clarkb | in many cases we avoid needing libffi-dev because the python packages publish wheels | 18:54 |
| clarkb | but occasionally there is a no wheel package published | 18:54 |
| sean-k-mooney | yep on sech it was the neutorn-fuctional an py27 varint that failed | 18:54 |
| fungi | oh, good point. in that case it could be a short-term failure as it raced our wheel builder | 18:55 |
| clarkb | infra team meeting strats in about 5 minutes | 18:55 |
| sean-k-mooney | http://logs.openstack.org/23/620123/9/gate/neutron-functional-python27/ff056e2/job-output.txt.gz#_2019-02-26_15_13_00_408424 | 18:55 |
| *** sdake has quit IRC | 18:55 | |
| *** sdake_ has joined #openstack-infra | 18:56 | |
| sean-k-mooney | fungi: ill give it an hour or so an ill recheck | 18:56 |
| *** sdake_ has quit IRC | 18:56 | |
| sean-k-mooney | assuming that is enough time for the wheels to be rebuilt? | 18:56 |
| *** ginopc has quit IRC | 18:57 | |
| fungi | huh, that log says ot retrieved http://mirror.iad.rax.openstack.org/pypifiles/packages/64/7c/27367b38e6cc3e1f49f193deb761fe75cda9f95da37b67b422e62281fcac/cffi-1.12.2.tar.gz (as a dependency of cryptography, as a dependency of ansible) | 18:57 |
| sean-k-mooney | Ansible Version: 2.5.15 | 18:58 |
| fungi | not the wheel | 18:58 |
| sean-k-mooney | did it get bumped recently? | 18:58 |
| *** sdake has joined #openstack-infra | 18:58 | |
| sean-k-mooney | fungi: way so it was building the bdis_wheel locally | 18:58 |
| clarkb | no ansible hasn't been bumped recently. What was likely bumped is cryptography | 18:58 |
| clarkb | the sdist for cryptography requires cffi to link to libssl | 18:59 |
| clarkb | the wheel doesn't iirc | 18:59 |
| clarkb | or at least ships with enough headers to make it work | 18:59 |
| sean-k-mooney | cryptography had its last release on januar 22 so i dont think its that | 18:59 |
| fungi | https://pypi.org/project/cffi/#history | 19:00 |
| fungi | new release 4 hours ago | 19:00 |
| sean-k-mooney | ah that would be about right | 19:00 |
| *** sdake has quit IRC | 19:00 | |
| ianw | mordred: re rhel8 -- yeah it's the beta which is available. this is why i'm vacillating on merging it as it; it has a short life span and the will need to split into a rhel path (subscrption manager all that jazz) and centos 8 | 19:00 |
| clarkb | oh huh I guess cryptography relies on cffi post install because cffi does compile on demand? | 19:01 |
| sean-k-mooney | the build started about 4 hours ago too | 19:01 |
| fungi | http://mirror.iad.rax.openstack.org/wheel/ubuntu-16.04-x86_64/cffi/ is where our wheels would be | 19:01 |
| sean-k-mooney | cffi is the forinf fucntion interface. i assumed it was a runtime dep to call in to c code but never played with it | 19:01 |
| *** sdake_ has joined #openstack-infra | 19:02 | |
| sean-k-mooney | fungi: ya 1.12.2 is not there yet | 19:02 |
| fungi | i don't see a 1.12.2 there yet | 19:02 |
| fungi | right, so waiting on our wheel builder job, seems like | 19:02 |
| *** bhavikdbavishi has quit IRC | 19:02 | |
| sean-k-mooney | cool ill check back there perorically thanks :) | 19:02 |
| *** wolverineav has quit IRC | 19:03 | |
| fungi | if i get a sec i'll see how often that job runs | 19:03 |
| fungi | and figure out when it will run again | 19:04 |
| sean-k-mooney | the neutorn jobs and posibly other are going to keep exploding untill that syncs. is it worth my time adding cffi to bindeps on neutorn to avoid that? | 19:04 |
| *** wolverineav has joined #openstack-infra | 19:04 | |
| *** markvoelker has joined #openstack-infra | 19:04 | |
| fungi | though strictly speaking, any project which relies on cffi ought to include a bindep.txt entry for libffi packages for the benefit of documenting local development environment setup | 19:04 |
| fungi | because cffi doesn't publish their own manylinux wheel, so anyone who wants to run these same tests locally is going to need a libffi headers | 19:05 |
| *** sdake_ has quit IRC | 19:06 | |
| *** sdake has joined #openstack-infra | 19:06 | |
| sean-k-mooney | well this was alway a dep you had to install to get devstack to work in the past so most developer proably get it becasue of that | 19:06 |
| fungi | yep, but a newcomer who wants to start by running unit tests is going to bang their head on it | 19:07 |
| sean-k-mooney | ya ok ill submit a patch to neuton to add it as a bindep | 19:07 |
| sean-k-mooney | its need by nova too so i might as well add it there too. | 19:08 |
| clarkb | fwiw cffi does publish linux wheels | 19:08 |
| *** ramishra_ has quit IRC | 19:09 | |
| fungi | oh, indeed i wonder why cffi-1.12.2-cp27-cp27m-manylinux1_x86_64.whl didn't get used | 19:09 |
| clarkb | it may not have existed at that time | 19:09 |
| fungi | ahh, right maybe they didn't upload them all at once | 19:09 |
| fungi | sean-k-mooney: so given that, it's probably safe to recheck already and probably not necessary to propose any bindep.txt additions after all | 19:10 |
| *** sdake has quit IRC | 19:10 | |
| sean-k-mooney | ok | 19:10 |
| sean-k-mooney | ill wait a bit | 19:10 |
| sean-k-mooney | im just going to grab dinner and ill recheck after | 19:10 |
| *** sdake has joined #openstack-infra | 19:11 | |
| fungi | definitely let us know if you continue to run into it | 19:11 |
| *** rfolco has joined #openstack-infra | 19:12 | |
| *** trown|lunch is now known as trown | 19:18 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Support requests for specific providers https://review.openstack.org/639418 | 19:20 |
| *** dayou has quit IRC | 19:20 | |
| *** sdake has quit IRC | 19:20 | |
| *** gouthamr has quit IRC | 19:20 | |
| *** sdake_ has joined #openstack-infra | 19:20 | |
| *** gouthamr_ has joined #openstack-infra | 19:20 | |
| *** mriedem has quit IRC | 19:21 | |
| *** dayou has joined #openstack-infra | 19:21 | |
| *** mriedem has joined #openstack-infra | 19:22 | |
| *** sdake_ has quit IRC | 19:25 | |
| *** sdake has joined #openstack-infra | 19:26 | |
| *** e0ne has joined #openstack-infra | 19:26 | |
| *** rfolco is now known as rfolco|rover | 19:28 | |
| *** ykarel has quit IRC | 19:33 | |
| *** yamamoto has joined #openstack-infra | 19:33 | |
| *** sdake has quit IRC | 19:35 | |
| *** sdake_ has joined #openstack-infra | 19:36 | |
| *** yamamoto has quit IRC | 19:37 | |
| *** markvoelker has quit IRC | 19:38 | |
| openstackgerrit | Merged openstack-infra/system-config master: Put health servers in the puppet health group https://review.openstack.org/639415 | 19:40 |
| *** wolverineav has quit IRC | 19:42 | |
| *** sdake_ has quit IRC | 19:45 | |
| *** yamamoto has joined #openstack-infra | 19:45 | |
| *** e0ne has quit IRC | 19:46 | |
| *** e0ne has joined #openstack-infra | 19:46 | |
| openstackgerrit | Merged openstack-infra/system-config master: Update puppetlabs-postgresql to 4.8.0 https://review.openstack.org/558995 | 19:46 |
| *** sdake has joined #openstack-infra | 19:46 | |
| *** ekultails has quit IRC | 19:47 | |
| *** yamamoto has quit IRC | 19:50 | |
| *** sdake has quit IRC | 19:55 | |
| *** wolverineav has joined #openstack-infra | 19:56 | |
| *** sdake has joined #openstack-infra | 19:57 | |
| *** derekh has joined #openstack-infra | 19:58 | |
| *** wolverineav has quit IRC | 20:01 | |
| clarkb | ianw: I see we raced on the update of the git:// stuff is jianweizhang working on that or did it autocomplete weird for ianw :) | 20:02 |
| clarkb | ianw: if you want that task feel free to take me off of it. I'll grab somethign else in the list | 20:02 |
| *** sdake has quit IRC | 20:03 | |
| *** hurgleburgler has joined #openstack-infra | 20:03 | |
| *** wolverineav has joined #openstack-infra | 20:04 | |
| *** ekultails has joined #openstack-infra | 20:06 | |
| *** luizbag has quit IRC | 20:07 | |
| ianw | clarkb: oh, yeah it seems jianweizhang is the only person to match on "ianw + enter" : ) | 20:08 |
| ianw | no strong feelings, happy to help out with reviews etc | 20:08 |
| *** wolverineav has quit IRC | 20:09 | |
| *** derekh has quit IRC | 20:09 | |
| ianw | i need to go wait for a guy to run a cable to new house ... should hopefully be back tethered in a little | 20:13 |
| *** jamesmcarthur has quit IRC | 20:15 | |
| *** jamesmcarthur has joined #openstack-infra | 20:16 | |
| *** yamamoto has joined #openstack-infra | 20:16 | |
| *** jamesmcarthur has quit IRC | 20:20 | |
| *** kjackal has joined #openstack-infra | 20:21 | |
| *** yamamoto has quit IRC | 20:21 | |
| clarkb | ianw: corvus: I'm trying to read up on afs upgrade paths and many mailing posts assert it is ok to have mixed filer versions. So I think we are safe to put the RW volume on one host, upgrade the other host(s), then move the RW to the now upgraded host and upgrade the other host(s) | 20:24 |
| *** calebb has joined #openstack-infra | 20:25 | |
| *** wolke23 has joined #openstack-infra | 20:26 | |
| clarkb | On the db host side of things http://docs.openafs.org/AdminGuide/HDRWQ406.html documents why keeping the IPs stable is desirable if I am reading that document right | 20:26 |
| fungi | so afs\d+ can follow our normal upgrade replacement process but afsdb\d+ are better off as in-place upgrades? | 20:27 |
| clarkb | fungi: I think we may still want afs\d+ to be inplace. We can just do them without outage by failing over back and forth | 20:28 |
| *** wolke has quit IRC | 20:28 | |
| fungi | ahh, okay | 20:28 |
| corvus | clarkb: remember that moving a volume in our network topology takes forever | 20:28 |
| corvus | well, a large volume | 20:28 |
| clarkb | corvus: we have two nodes in dfw now and only small volumes are in ord iirc | 20:28 |
| fungi | unless we can detach the block devices for it and attach them to another fileserver, i guess, but... no idea how complex that is for afs | 20:29 |
| clarkb | corvus: I guess worst case we'd have RO access during the outage if we don't fail back and forth | 20:29 |
| clarkb | On the db cluster side of things I have found mention that people usually upgrade those last because they are the most complicated. Implying that you can have newer file servers with older db servers while you wait to do that upgrade | 20:31 |
| *** e0ne has quit IRC | 20:31 | |
| clarkb | What I still haven't sorted out is if the openafs 1.6.7 on trusty db is going to be mad with openafs 1.6.15 on xenial if we do those one at a time | 20:31 |
| clarkb | ianw I know you've dug into the details of openafs recently a couple times, do you have any idea where I might find info like ^ | 20:32 |
| clarkb | I'm going to grab lunch now. But will continue to try and dig around the documentation for the info we need around that. Help appreciated :) | 20:35 |
| *** diablo_rojo_ has quit IRC | 20:35 | |
| clarkb | also I think I've got health01.opendev.org working | 20:35 |
| clarkb | I guess I should go ahead and push up the zone file updates now | 20:36 |
| *** diablo_rojo has joined #openstack-infra | 20:36 | |
| fungi | i got up a bit early, so will probably disappear soon to make early dinner and then be back to help prep for tc election setup (and have tc office hour after that's finished) | 20:36 |
| fungi | but i can likely at least review some afs upgrade activities | 20:37 |
| *** wolverineav has joined #openstack-infra | 20:37 | |
| openstackgerrit | Merged openstack-infra/zuul master: Log to job output when running Ansible setup https://review.openstack.org/637813 | 20:39 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zone-opendev.org master: Organize the zone file by record name https://review.openstack.org/639433 | 20:41 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zone-opendev.org master: Add health01.opendev.org https://review.openstack.org/639434 | 20:41 |
| clarkb | infra-root let me know what you think about 639433. Is that organization worthwhile? | 20:41 |
| *** gouthamr_ is now known as gouthamr | 20:42 | |
| *** wolverineav has quit IRC | 20:42 | |
| clarkb | heh and of course I forget the serial update on the second change | 20:42 |
| mordred | clarkb: lgtm | 20:42 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zone-opendev.org master: Add health01.opendev.org https://review.openstack.org/639434 | 20:42 |
| *** takamatsu_ has quit IRC | 20:48 | |
| *** takamatsu_ has joined #openstack-infra | 20:50 | |
| corvus | clarkb: https://status.openstack.org/ | 20:50 |
| corvus | clarkb: does the ssl situation there seem weird to you? | 20:50 |
| corvus | clarkb: like, maybe, dns entries for status.openstack.org have been updated to point to openstack.org? | 20:51 |
| clarkb | err_cert_common_name_invalid hrm | 20:52 |
| clarkb | yes IP addrs match | 20:53 |
| clarkb | no more jimmy here | 20:53 |
| corvus | i am happy that we are moving to a dns system based around version control. | 20:54 |
| clarkb | I've askedabout it on the foundation side | 20:56 |
| corvus | clarkb: while we're waiting -- is oshealth something that should be in opendev? or is it a bit openstack-specific? | 20:57 |
| clarkb | corvus: The service is fairly generic. In its current use it is a bit openstack specific but I think we could have the other projects use it too (its just grabbing aubunit data and showing it to you | 20:57 |
| clarkb | I think all the zuul python unittests are in there for example | 20:58 |
| corvus | clarkb: i think my concerns are (a) the name (b) whether things about project/branch/test/job names are openstacky... in fact, did it ever get updated to handle the fact that job names are no longer unique per project? | 20:58 |
| clarkb | on the last thing yes I believe that was fixed | 20:59 |
| clarkb | on a) we would have to update it like other services. for b) i dont know if it makes assumptions around things like stable branches | 20:59 |
| corvus | sorry, for (a) i mean, the project is named "openstack-health" | 21:00 |
| clarkb | oh the code itself. | 21:00 |
| corvus | so are its maintainers even interested in supporting more than openstack | 21:00 |
| corvus | yah | 21:00 |
| clarkb | gmann: ^ do you know? | 21:02 |
| *** spsurya has quit IRC | 21:03 | |
| clarkb | I sort of saw it as a generic subunit results tracker but if there are openstack assumptions (and Im sure there is some) then that is non zero effprt to change | 21:03 |
| corvus | clarkb: i put a -1 on https://review.openstack.org/639434 until we can talk about it more, mostly because i don't think we want to make first and second-class citizens in opendev. it's hard for me to make that evaluation right now. | 21:05 |
| corvus | hopefully the dns issue will get fixed soon (it has a short ttl) and we can poke at it. | 21:06 |
| clarkb | I've not yet heardback on dns | 21:07 |
| *** jamesmcarthur has joined #openstack-infra | 21:11 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-openstack_health master: Fully root pip path https://review.openstack.org/639442 | 21:16 |
| clarkb | ^^ is a fix that will be helpful if we end up deciding that a redeploy of health under openstack.org is better | 21:17 |
| pabelanger | ianw: do you have any idea what is happening here? http://logs.openstack.org/99/639399/2/check/windmill-src-ubuntu-bionic/5365f15/logs/nb01/var/log/nodepool/builds/ubuntu-bionic-0000000131.log | 21:17 |
| clarkb | corvus: jimmy is fixing dns now | 21:17 |
| pabelanger | seems diskimage-builder no longer working for some readon | 21:18 |
| pabelanger | reason* | 21:18 |
| pabelanger | oh | 21:18 |
| pabelanger | ah | 21:18 |
| pabelanger | I might know | 21:18 |
| clarkb | 2019-02-26 21:06:44.143 | sudo: no tty present and no askpass program specified | 21:18 |
| clarkb | I think its saying it cannot sudo | 21:19 |
| pabelanger | yah, the patch modified sudoers | 21:19 |
| pabelanger | so, think I clobbered something on the images | 21:20 |
| pabelanger | sorry for noise | 21:20 |
| clarkb | corvus: dns should be fixed (at least I get the server locally now you may still have to wiat for cache expiry) | 21:20 |
| *** wolverineav has joined #openstack-infra | 21:21 | |
| *** wolverineav has quit IRC | 21:21 | |
| jamesmcarthur | corvus: et. al. - My apologies. This is squarely on me. I combined status.o.o into our list of domain updates this morning. | 21:21 |
| *** wolverineav has joined #openstack-infra | 21:21 | |
| jamesmcarthur | Terribly sorry for any outage or confusion. | 21:21 |
| clarkb | corvus: http://status.openstack.org/openstack-health/#/g/project/openstack-infra~2Fzuul is how health renders things for zuul. It does group things by branch and other fields but I don't see it making assumptiosn about those branches (I could be missing something) | 21:22 |
| clarkb | probably the biggest problem that jumps out to me is the big headr that says visualize test results for opensatck CI jobs | 21:22 |
| *** manjeets_ has joined #openstack-infra | 21:22 | |
| corvus | clarkb: it makes up its own job names | 21:22 |
| clarkb | corvus: oh interesting: tox-py35-zuul | 21:23 |
| corvus | ya | 21:23 |
| ianw | clarkb: sorry, back now ... i think we should be able to mix afs versions, i don't think there's any on-wire differences | 21:23 |
| *** manjeets has quit IRC | 21:24 | |
| clarkb | I guess that was the dedup method chosen | 21:24 |
| *** derekh has joined #openstack-infra | 21:24 | |
| *** derekh has quit IRC | 21:24 | |
| openstackgerrit | ladjadj proposed openstack-infra/grafyaml master: Add capability to use grafana folderid parameter https://review.openstack.org/637520 | 21:24 |
| clarkb | the logs link is correct at least | 21:25 |
| openstackgerrit | ladjadj proposed openstack-infra/grafyaml master: Add capability to use grafana folderid parameter https://review.openstack.org/637520 | 21:25 |
| openstackgerrit | Chris Dent proposed openstack-infra/project-config master: Set up placement project to use storyboard https://review.openstack.org/639445 | 21:25 |
| *** yamamoto has joined #openstack-infra | 21:25 | |
| corvus | clarkb: does http://status.openstack.org/openstack-health/#/tests work for you? | 21:26 |
| clarkb | if that is loading the /tests api endpoint (I used this endpoint to test the new server) it does take a while to get the json back | 21:27 |
| *** zbr|out has quit IRC | 21:27 | |
| corvus | okay, i got the page | 21:27 |
| corvus | i don't know what it means | 21:27 |
| *** e0ne has joined #openstack-infra | 21:28 | |
| corvus | it's mostly uuids, then a bunch of project names | 21:28 |
| clarkb | I'm still waiting for it to render | 21:28 |
| clarkb | my guess is that the uuids are the uuids that tempest supports in its tests for use by refstack | 21:29 |
| *** yamamoto has quit IRC | 21:30 | |
| corvus | what causes tox-py35-zuul to get the suffix, but not zuul-tox-remote? | 21:30 |
| fungi | okay, back and catching up | 21:32 |
| mordred | corvus: maybe zuul-tox-remote is only used by one project? | 21:32 |
| corvus | mordred: maybe. how does it know? | 21:33 |
| mordred | maybe it's based on what has reported? so if there is only a single project associated with a job in the db it doesn't suffix? | 21:33 |
| *** kgiusti has left #openstack-infra | 21:33 | |
| *** eharney has quit IRC | 21:33 | |
| *** manjeets_ is now known as manjeets | 21:35 | |
| clarkb | I'm not seeing anything in subunit2sql that jumps out at me as encoding that from the subunit somewhere | 21:35 |
| *** markvoelker has joined #openstack-infra | 21:35 | |
| clarkb | ianw: cmurphy fyi http://paste.openstack.org/show/746309/ the postgres change on ask applied and postgres things were updated and service restarted. Askbot is still responding to http so I think its fine but I'll dig up some diffs of those files really quick to be extra sure | 21:37 |
| corvus | clarkb, mordred: http://git.openstack.org/cgit/openstack-infra/puppet-subunit2sql/commit/?id=2a7070995e47791a31f86073bb448f8b2c259a04 | 21:37 |
| *** jcoufal has quit IRC | 21:37 | |
| clarkb | ianw: cmurphy /etc/postgresql/9.3/main/pg_ident.conf was just commented out file changed to "managed by puppet" comment so that one seems fine | 21:38 |
| clarkb | oh its in the gearman worker itself interesting | 21:39 |
| mordred | clarkb, corvus: the logic is "if project name not already in job name, add suffix" | 21:39 |
| corvus | clarkb, mordred: yeah, so apparently it doesn't get the 'zuul' suffix because it already has a 'zuul' prefix | 21:39 |
| corvus | ya | 21:39 |
| *** eandersson_ is now known as eandersson | 21:40 | |
| clarkb | /usr/local/bin/validate_postgresql_connection.sh was updated to chagne how it did a loop | 21:40 |
| clarkb | I think those both look like safe edits and the postgres stuff is ifne | 21:40 |
| mordred | I feel like this is an openstack specific tool and if we wanted it to be a general zuul/opendev tool it would need a non-zero amount of work | 21:40 |
| corvus | yeah, i have serious concerns that the data model as currently implemented is really going to scale to more projects | 21:41 |
| *** whoami-rajat has quit IRC | 21:41 | |
| clarkb | the concern being collisions in the input data? I agree that is less than ideal (properly tracking project as a field in the db seems saferbut also effort) | 21:42 |
| clarkb | my desktop just locked up | 21:43 |
| corvus | the fact that it makes up its own names for the jobs is not ideal either. doesn't really set us up for a unified service offering | 21:43 |
| clarkb | cant even switvh to vty1 | 21:43 |
| *** jtomasek has quit IRC | 21:43 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [dnm] letsencrypt prototype implementation https://review.openstack.org/636759 | 21:44 |
| openstackgerrit | Merged openstack-infra/zone-opendev.org master: Organize the zone file by record name https://review.openstack.org/639433 | 21:44 |
| mordred | clarkb: it's feeling to me that everything on status.o.o is pretty openstack-specific, including e-r. I think elasticsearch log indexing itself is a sane opendev thing to offer - but the e-r classification also seems pretty openstack project specific | 21:44 |
| clarkb | and the reason for that is so when viewed on the job axis you dont get results for all projects in one spot? | 21:45 |
| mordred | clarkb: so maybe it's best to consider status.o.o to be a collection of openstack tools | 21:45 |
| corvus | clarkb: reason for what? | 21:45 |
| clarkb | corvus: for deduping at all | 21:45 |
| fungi | clarkb: technically there was no need to bump the serial on 639433 since you weren't making any record changes and therefore had no need for it to propagate to the slaves anyway, but it doesn't hurt | 21:45 |
| clarkb | or is it undeduping? | 21:45 |
| clarkb | mordred: Im not sure I agree re e-r | 21:45 |
| corvus | clarkb: i can not speak to the reasoning. i'll just let http://git.openstack.org/cgit/openstack-infra/puppet-subunit2sql/commit/?id=2a7070995e47791a31f86073bb448f8b2c259a04 speak for itself. | 21:46 |
| *** wolverineav has quit IRC | 21:46 | |
| mordred | clarkb: I think if we were to offer e-r generally, we'd want a collection of elastic recheck patterns per-tenant - and a dashboard per tenant | 21:46 |
| clarkb | mordred: the queries are arbitrary. The one bit that is quite openstack specific is the list of gate jobs that failed without a match | 21:46 |
| clarkb | mordred: thats fair | 21:46 |
| mordred | clarkb: and I think it coudl certainly be nice to develop that as a generic offering that could be offered to each tenant | 21:47 |
| corvus | yeah, so maybe with a little bit of work we can make e-r more general. that doesn't sound like over-reaching. | 21:47 |
| mordred | ++ | 21:47 |
| mordred | but still a thing with multiple instances - so that groups can manage their own queries and so that the dashboard is at least tenant-aware - or that there are per-tenant dashboards | 21:47 |
| *** wolverineav has joined #openstack-infra | 21:48 | |
| clarkb | ya the managing your own queries aspect is probably the biggest one | 21:48 |
| *** wolverineav has quit IRC | 21:48 | |
| mordred | yah | 21:48 |
| *** wolverineav has joined #openstack-infra | 21:48 | |
| corvus | multiple instances is probably pretty easy to do with e-r, and the shortest way to get from here to there. | 21:48 |
| corvus | and relatively low-cost since it's basically a cron-job. | 21:49 |
| clarkb | corvus: yes though right now all the queries are laoded from the one repo. So need to change that then run different instances and that should do it | 21:49 |
| clarkb | I dont expect that to be too difficult to change | 21:49 |
| corvus | we could totally do that and have it write out to "recheck.opendev.org/openstack" or whatever. | 21:50 |
| fungi | also the subunit2sql db presents a bit of a scaling problem of its own | 21:50 |
| clarkb | So I think I'm hearing I should build a new health server and put it in openstackland | 21:50 |
| mordred | clarkb: yeah. I think for now that's the best bet | 21:51 |
| fungi | that makes sense to me | 21:51 |
| clarkb | can I get reviews on 639442 as part of that? | 21:51 |
| mordred | clarkb: and then when we figure out recheck.opendev.org we can do that | 21:51 |
| fungi | similar reasons why the openstackid servers are on openstack.org hostnames not opendev.org | 21:51 |
| clarkb | fixes a bootstrapping issue in puppet | 21:51 |
| fungi | it took me a bit of rethink to agree that we should only put things in the opendev.org domain if we expect to be able to have them be generic opendev services | 21:52 |
| corvus | i had to quit/restart my browser because it was caching the invalid status redirect, but after doing so, it works again. fyi in case anyone else reports that problem. | 21:53 |
| fungi | since the goal is to build a cohesive service offering under the opendev "brand" and not just wholesale move all our services there regardless of whether they might be project-specific | 21:54 |
| *** kjackal has quit IRC | 21:54 | |
| clarkb | fungi: ya. fwiw I had considered this but had also thought of the health dashboard as more generic subunit visualizer. | 21:55 |
| corvus | i think it should be, but i think it is not. | 21:56 |
| *** yamamoto has joined #openstack-infra | 21:56 | |
| clarkb | ya | 21:57 |
| fungi | probably "just" needs a different data model | 21:58 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Revert "Add new health01 server to our inventory" https://review.openstack.org/639451 | 21:59 |
| clarkb | infra-root ^ is the cleanup of the opendev server in our inventory | 22:00 |
| clarkb | and on the earlier change that should fix a thing now I'm confused http://logs.openstack.org/42/639442/1/check/puppet-beaker-rspec-infra/aa6fee5/job-output.txt.gz#_2019-02-26_21_38_12_232250 | 22:00 |
| clarkb | the test suite fails on the error we saw in the real deploy | 22:00 |
| clarkb | but only when I flip it around so that the real deploy should work | 22:00 |
| fungi | bizarre | 22:01 |
| clarkb | probably because our test images put pip at /usr/local/bin/pip for us | 22:01 |
| clarkb | so puppet is nooping there | 22:01 |
| clarkb | but its not in /usr/bin/pip | 22:01 |
| clarkb | I'm going to think about a better fix to that now | 22:01 |
| fungi | or it's puppet having nondeterministic ordering of parallel operations? | 22:01 |
| *** yamamoto has quit IRC | 22:01 | |
| clarkb | no puppet 3 and 4 should be ordered | 22:01 |
| clarkb | they made it implicitly top to bottom halfway through 3 iirc | 22:02 |
| fungi | oh, okay | 22:02 |
| *** e0ne has quit IRC | 22:03 | |
| *** rcernin has joined #openstack-infra | 22:05 | |
| clarkb | mordred: do you know where the {{ target }} variable comes from in system-config/playbooks/set-hostnames.yaml ? | 22:06 |
| clarkb | mordred: I think that is what I want but can't find ansible docs saying it is special | 22:06 |
| mordred | clarkb: looking | 22:07 |
| clarkb | (I'm adding a playbook that will run unattended-upgrade on our nodes after boot with launch-node.oy) | 22:07 |
| ianw | for those waiting on my cable update; the guy they sent isn't qualified to work on my sort of pole, so now we need requisition the advanced pole team | 22:07 |
| mordred | clarkb: it's from -e given to it in launch-node.py | 22:07 |
| clarkb | ah | 22:07 |
| clarkb | ya I see it now | 22:08 |
| mordred | clarkb: so the ansible_cmd in launch-node always adds -e target={name} | 22:08 |
| clarkb | so that is what I want | 22:08 |
| mordred | clarkb: yeah. you should just be able to add another playbook to the list there | 22:08 |
| *** wolverineav has quit IRC | 22:08 | |
| corvus | ianw: that's amazing :) | 22:09 |
| *** markvoelker has quit IRC | 22:09 | |
| corvus | mordred: and we should definitely call it 'launch-node.oy' :) | 22:09 |
| corvus | er, that was clarkb's typo ^ | 22:10 |
| *** wolverineav has joined #openstack-infra | 22:10 | |
| clarkb | oy | 22:10 |
| ianw | .oy.vey seems sort of appropriate | 22:10 |
| mordred | ianw: that's making me want to write a new programming language | 22:12 |
| ianw | pabelanger: do you want to look at this easy grafyaml -> https://review.openstack.org/#/c/637520 ... just the first thing that's not been basic syntax matching updates in a while | 22:12 |
| *** wolverineav has quit IRC | 22:14 | |
| clarkb | ansible style question when: ansible_facts['os_family'] == "Debian" or when: ansible_os_family == "Debian" ? | 22:15 |
| clarkb | both seem to be valid? | 22:15 |
| ianw | clarkb: looking back through puppet-graphite, it seems i got everything xenialish merged and then just got distracted. any problems if i start up graphite01? | 22:16 |
| corvus | mordred: ^ | 22:16 |
| clarkb | ianw: I don't expect so | 22:16 |
| fungi | clarkb: roll a d20 and consult the style chart? | 22:16 |
| clarkb | the conditionals docs page uses the first option. So maybe i just use the first option too | 22:16 |
| mordred | clarkb: I think the first option is the new way forward | 22:16 |
| ianw | are we doing this as graphite01.opendev.org ? | 22:17 |
| mordred | clarkb: ansible_os_family is a form they're moving away from | 22:17 |
| corvus | ianw: ++ | 22:17 |
| fungi | ianw: graphite seems opendevish to me, yes | 22:17 |
| *** wolverineav has joined #openstack-infra | 22:17 | |
| fungi | we've got zuul and nodepool sending data directly to it, after all | 22:18 |
| *** slaweq has quit IRC | 22:18 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Apply package updates before we reboot in launch-node https://review.openstack.org/639454 | 22:18 |
| ianw | fungi: i'm feeling that a detach + reattach of the volumes is the easiest, rather than rsyncing | 22:18 |
| clarkb | mordred: ^ cool I picked correctly in that change then | 22:18 |
| ianw | but let's get the host up first ... | 22:18 |
| fungi | ianw: i only added rsync to my comment there because i was too lazy to check whether we'd put that data all on a cinder volume | 22:19 |
| fungi | graphs and whisper files? | 22:19 |
| clarkb | I can test 639454 before we merge it (I'll test with launch of health01.openstack.org) | 22:19 |
| * clarkb starts that process then will figure out why puppet is sad about pip | 22:20 | |
| ianw | fungi: yeah, it's all the whisper db files that are isolated under /var/lib/graphite/storage which is a lvm on a cinder | 22:21 |
| ianw | well two 1tb cinder volumes | 22:22 |
| *** slaweq has joined #openstack-infra | 22:26 | |
| *** sdake has joined #openstack-infra | 22:27 | |
| mordred | clarkb: lgtm \o/ | 22:29 |
| *** slaweq has quit IRC | 22:30 | |
| *** sdake has quit IRC | 22:30 | |
| clarkb | I'm pretty sure the issue with pip on health is in the test nodes python-pip is "installed" but /usr/bin/pip is not present | 22:32 |
| clarkb | so the python module doesn't try to install python-pip | 22:32 |
| *** dave-mccowan has quit IRC | 22:33 | |
| clarkb | when we run the actual server we install python-pip via puppet and nothing installs /usr/local/bin/pip | 22:33 |
| *** sdake has joined #openstack-infra | 22:33 | |
| clarkb | but the problme there is we don't know pip exists in $PATH at that point because pupept has already cached that data? | 22:33 |
| clarkb | I'm double checking the exec PATH handling now | 22:33 |
| clarkb | ya its got /usr/bin in the path so it must be caching things | 22:34 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zone-opendev.org master: Add graphite01.opendev.org https://review.openstack.org/639461 | 22:39 |
| *** sdake has quit IRC | 22:40 | |
| *** sdake_ has joined #openstack-infra | 22:42 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-openstack_health master: Fully root pip path https://review.openstack.org/639442 | 22:43 |
| clarkb | I think ^ will test the theory I've got about why this fails | 22:43 |
| clarkb | its a bit hacky and not sure we really want to merge that if it fixes things | 22:43 |
| clarkb | ianw: ^ you deal with a lot of the pip stuff and our images. Any ideas? | 22:43 |
| clarkb | mordred: that script seemed to work. I have a newly booted server with packages updated | 22:44 |
| *** slaweq has joined #openstack-infra | 22:44 | |
| *** sdake_ has quit IRC | 22:45 | |
| *** ijw has quit IRC | 22:46 | |
| *** sdake has joined #openstack-infra | 22:46 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Add health01.openstack.org server https://review.openstack.org/639464 | 22:47 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Support requests for specific providers https://review.openstack.org/639418 | 22:47 |
| clarkb | infra-root ^ I think we can go ahead and move forward with that rather than fix puppet for the bootstrap case since this has gotten more complicated than I initially expected | 22:47 |
| clarkb | ianw: +2 on your graphite01 dns change, I didn't approve as not sure how far into bootstrapping you want to get before getting it into dns | 22:48 |
| *** jamesmcarthur has quit IRC | 22:49 | |
| *** slaweq has quit IRC | 22:49 | |
| ianw | clarkb: hrm, no i'm not sure what's going on with that puppet pip /usr/local/bin /usr/bin split :/ | 22:49 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add graphite01.opendev.org https://review.openstack.org/639466 | 22:52 |
| *** sdake has quit IRC | 22:56 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add graphite01.opendev.org https://review.openstack.org/639466 | 22:56 |
| *** sdake has joined #openstack-infra | 22:58 | |
| openstackgerrit | Merged openstack-infra/zone-opendev.org master: Add graphite01.opendev.org https://review.openstack.org/639461 | 22:58 |
| clarkb | ianw: you may want ot manually run the package updates on new graphite https://review.openstack.org/#/c/639454/ since I'm assuming you haven't run with that too | 22:58 |
| clarkb | infra-root 639454 is ready for review. I am satisfied that it worked with the launch of health01.openstack.org | 22:58 |
| clarkb | has anyone else noticed the worst thing about opendev.org is we can't currently use o.o as a fqdn suffix? | 22:59 |
| *** agopi has quit IRC | 22:59 | |
| corvus | i thought that was a feature :) | 22:59 |
| clarkb | heh its just so much more typing when you need to be specific :) | 23:00 |
| *** sdake has quit IRC | 23:00 | |
| *** sdake has joined #openstack-infra | 23:02 | |
| *** jamesmcarthur has joined #openstack-infra | 23:02 | |
| *** tkajinam has joined #openstack-infra | 23:02 | |
| ianw | Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] | 23:02 |
| ianw | now that's interesting ... | 23:02 |
| ianw | ansible i guess has updated the apt files on this xenial host to bionic | 23:03 |
| clarkb | ianw: or is it really bionic? did you set the image name? I wonder if we default to bionic in the launch script now | 23:04 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Request dependent job nodes from paused parent provider https://review.openstack.org/639467 | 23:04 |
| clarkb | that may be worth a note in the readme if that is the case | 23:04 |
| ianw | clarkb: yeah, i specified the xenial image, and it's got VERSION="16.04.5 LTS (Xenial Xerus)" | 23:04 |
| clarkb | huh the server I launched has got deb http://security.ubuntu.com/ubuntu xenial-security main universe | 23:05 |
| clarkb | VERSION_ID="16.04" | 23:05 |
| clarkb | which isn't a little different (that is the os-release file) | 23:05 |
| *** sdake has quit IRC | 23:06 | |
| *** sdake_ has joined #openstack-infra | 23:06 | |
| ianw | Found image Ubuntu 16.04 LTS (Xenial Xerus) (PVHVM) | 23:06 |
| ianw | --- Running initial configuration on host 104.130.138.88 --- | 23:06 |
| clarkb | oh wait that is version id | 23:06 |
| clarkb | VERSION="16.04.6 LTS (Xenial Xerus)" | 23:06 |
| clarkb | that is incredibly odd | 23:06 |
| clarkb | ianw: I wonder if that server got added into a group that makes it bionic as a side effect when we thought it was going to be bionic + docker? | 23:07 |
| ianw | it definitely decided on bionic sources -> http://paste.openstack.org/show/746373/ | 23:09 |
| *** jamesmcarthur has quit IRC | 23:09 | |
| *** sreejithp has quit IRC | 23:10 | |
| ianw | src: 'sources.list.{{ ansible_facts.lsb.codename }}.{{ ansible_facts.architecture }}' | 23:10 |
| clarkb | src: 'sources.list.{{ ansible_facts.lsb.codename }}.{{ ansible_facts.architecture }}' is how it decides which file to use from the ansible | 23:10 |
| clarkb | implying the ansible_fact came back as bionic for some reason | 23:10 |
| ianw | how would it get that fact wrong? | 23:10 |
| *** sdake_ has quit IRC | 23:10 | |
| clarkb | b5bc5aab-fd74-4afe-ba49-5b166fda8b51 is the image id in dfw that I booted with | 23:10 |
| clarkb | looks like new graphtie isn't in dfw (which is fine but that may explain the difference at least) | 23:11 |
| *** sdake has joined #openstack-infra | 23:11 | |
| clarkb | (I kept new health in dfw because that is where the large mysql db for the subunit data is) | 23:11 |
| ianw | export OS_REGION_NAME=DFW | 23:12 |
| ianw | i intended it to be there ... | 23:12 |
| clarkb | I don't see it under a show | 23:12 |
| * clarkb does list | 23:12 | |
| clarkb | oh because I'm a derp | 23:13 |
| clarkb | opendev not openstack ? | 23:13 |
| clarkb | b5bc5aab-fd74-4afe-ba49-5b166fda8b51 is your image which is the same as mine | 23:13 |
| clarkb | so how did the facts end up wrong | 23:13 |
| ianw | 75014a17-d51e-4b20-ab92-9b3ee5776ae8 is the id | 23:13 |
| clarkb | did launch node somehow use bridge's facts for the remote node? | 23:14 |
| clarkb | bridge is bionic | 23:14 |
| *** rascasoft has quit IRC | 23:15 | |
| ianw | oh, hrm, i did actually create one first that *was* bionic, but then i deleted it | 23:16 |
| ianw | maybe the facts were cached somehow? | 23:16 |
| clarkb | oh could it be the fact cache in that case? | 23:16 |
| clarkb | ya | 23:17 |
| *** yamamoto has joined #openstack-infra | 23:17 | |
| clarkb | mordred: ^ I think you understand the ansible fact cache better than anyone else. Can we disable it for launch node? and or confirm this is the problem | 23:17 |
| ianw | yeah, that probably explains it. a rerun will probably fix it | 23:17 |
| ianw | but yeah, launch node ignoring any caches would probably be ideal | 23:18 |
| clarkb | ianw: dpkg/apt are usually really unhappy about trying to do a downgrade | 23:18 |
| clarkb | so if any package were pulled in from the bionic repos it might be easier to rebuild? (not sure) | 23:18 |
| corvus | guilhermesp: do you think we retry creating servers in sjc1 now? | 23:19 |
| *** sdake has quit IRC | 23:20 | |
| *** dave-mccowan has joined #openstack-infra | 23:21 | |
| clarkb | ianw: looks like we want to run ansible-playbook with --flush-cache | 23:21 |
| clarkb | ianw: https://docs.ansible.com/ansible/2.4/ansible-playbook.html#cmdoption-ansible-playbook-flush-cache | 23:21 |
| *** yamamoto has quit IRC | 23:21 | |
| *** gfidente|afk has quit IRC | 23:22 | |
| *** sdake has joined #openstack-infra | 23:23 | |
| *** sdake has quit IRC | 23:25 | |
| *** sdake has joined #openstack-infra | 23:27 | |
| *** sdake has quit IRC | 23:30 | |
| *** sdake has joined #openstack-infra | 23:31 | |
| *** sdake has quit IRC | 23:35 | |
| ianw | interesting, it's got itself into this un-ansible-able state now because /usr/bin/python isn't installed | 23:35 |
| ianw | "msg": "SSH Error: data could not be sent to remote host \"2001:4800:7818:103:be76:4eff:fe00:beac\". Make sure this host can be reached over ssh", | 23:35 |
| ianw | this error is a bit of a red herring | 23:35 |
| ianw | cause it actually tries SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/59038bd598 2001:4800:7818:103:be76:4eff:fe00:beac '/bin/sh -c '"'"'/usr/bin/python && sleep 0'"'"'' | 23:36 |
| corvus | we usually need to set ansible_python_interpreter on bionic hosts | 23:36 |
| ianw | yeah; this is now a xenial/bionic frankenstein | 23:37 |
| clarkb | Its alive! | 23:37 |
| ianw | i think though, if i just rebuild it as xenial, and run the base playbook against it without cached facts, we should be ok and retain the same ip addrs | 23:37 |
| clarkb | I would expect that to work | 23:37 |
| clarkb | also set-hostname is the other one we run | 23:38 |
| *** sdake has joined #openstack-infra | 23:38 | |
| ianw | yeah, basically just a manual launch-node but against the current inventory | 23:38 |
| ianw | well, create an inventory for the currently existing host, i mean | 23:38 |
| ianw | oh, keys ... | 23:39 |
| ianw | maybe it's just easier to update dns at this point | 23:40 |
| *** sdake has quit IRC | 23:40 | |
| ianw | yeah, i'll delete it, and try a launch node with the flush-cache option to boot | 23:40 |
| clarkb | k | 23:40 |
| *** sdake has joined #openstack-infra | 23:42 | |
| *** sdake has quit IRC | 23:50 | |
| *** jesusaur has quit IRC | 23:51 | |
| *** sdake has joined #openstack-infra | 23:51 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-openstack_health master: Fully root pip path https://review.openstack.org/639442 | 23:52 |
| *** ijw has joined #openstack-infra | 23:53 | |
| fungi | i think i must have skimmed too eagerly... did we cache bionic facts from an aborted replacement of graphite and then end up incorrectly configuring a followup xenial build as partly bionic as a result? | 23:53 |
| * fungi only gets a few minutes between batches of ballots | 23:54 | |
| clarkb | fungi: basically ya | 23:54 |
| clarkb | and made a frankenstein | 23:55 |
| *** yamamoto has joined #openstack-infra | 23:55 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zone-opendev.org master: Add graphite01.opendev.org, take 2 https://review.openstack.org/639469 | 23:55 |
| clarkb | http://logs.openstack.org/64/639464/1/check/infra-puppet-apply-3-ubuntu-xenial/4a295ac/applytest/puppetapplytest08.final.out.FAILED is going to fail my addition of the new health01 server to the inventory | 23:55 |
| clarkb | I have no idea how that would've failed we didn't change anything in the drupal or drush moduels as far as I can tell? | 23:55 |
| *** sdake has quit IRC | 23:55 | |
| clarkb | oh the upstream is 502ing | 23:57 |
| clarkb | I was just able to clone it locally so maybe I just need a recheck | 23:57 |
| *** sdake has joined #openstack-infra | 23:57 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add --flush-cache to launch.py ansible https://review.openstack.org/639470 | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!