| corvus | clarkb: lgtm | 00:00 |
|---|---|---|
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Replicate to the standalone gitea severs https://review.openstack.org/640431 | 00:01 |
| clarkb | corvus: mordred ^ there is the update and I put more info in the commit message about it | 00:02 |
| clarkb | url = ssh://git@github.com:22/${name}.git was the full config line on review-dev ftr. Our puppetry appends the {name}.git suffix | 00:03 |
| dmsimard | ianw: find anything ? | 00:04 |
| ianw | dmsimard: umm, sorry had to duck out, i dunno where the xvdb came from | 00:05 |
| *** hwoarang has quit IRC | 00:06 | |
| ianw | it's booted into 4.15.0-45-generic | 00:07 |
| ianw | i notice that bridge is running 4.15.0-22, but has 44 & 45 installed, presumably through updates | 00:07 |
| *** hwoarang has joined #openstack-infra | 00:08 | |
| *** sdake has joined #openstack-infra | 00:08 | |
| clarkb | ianw: yes likely just pending a reboot there | 00:09 |
| ianw | yeah, i wonder though if the snapshot is in a weird state | 00:10 |
| ianw | dmsimard: i may have screwed it up in emergency mode, i'm just going to try rebuilding it and examining it again, knowing that i'm looking for swapped partitions | 00:10 |
| corvus | clarkb: i tested your replication url syntax locally with docker-compose and gerrit and gitea, and it seems to work | 00:11 |
| corvus | clarkb: i'm pretty confident that should address frickler's comment, and mordred previously approved, so i'm going to +3 | 00:12 |
| *** sdake has quit IRC | 00:12 | |
| corvus | (we still have the manual gerrit restart step to do to actually kick that off. | 00:12 |
| *** sdake has joined #openstack-infra | 00:12 | |
| clarkb | corvus: yup and review-dev is running the exact same version of gerrit and the plugin so that should be a good other test | 00:13 |
| clarkb | and ya we'll need to restart gerrit. Remember when we did that ~weekly? | 00:13 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient master: Rebuild node-sass bindings when running the webclient dev server https://review.openstack.org/634615 | 00:14 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient master: Make tags section more consistent https://review.openstack.org/633613 | 00:14 |
| *** wolverineav has joined #openstack-infra | 00:16 | |
| corvus | clarkb, fungi: i've +3d https://review.openstack.org/636775 since the git playbook is now < 30m | 00:16 |
| pabelanger | odd, I cannot seem to connect to https://zuul.o.o on this public network, but https://zuul.opendev.org works | 00:17 |
| corvus | dmsimard: http://zuul.openstack.org/config-errors | 00:19 |
| clarkb | pabelanger: they are the same server and you should get the same A or AAAA record | 00:19 |
| pabelanger | clarkb: yah, dig looks to be the same | 00:20 |
| pabelanger | http://paste.openstack.org/show/747263/ | 00:20 |
| clarkb | I can confirm both work from here so it isn't a universal problem | 00:21 |
| pabelanger | I'm not going to spend too much time on it, suspect something on firewall is filtering something | 00:21 |
| *** jcoufal has joined #openstack-infra | 00:22 | |
| *** Adri2000 has quit IRC | 00:22 | |
| *** Adri2000 has joined #openstack-infra | 00:25 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Use a buildset registry in the container jobs https://review.openstack.org/640900 | 00:28 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-preview master: DNM: break dockerfile https://review.openstack.org/640901 | 00:30 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: DNM: test broken zuul-preview dockerfile https://review.openstack.org/640903 | 00:31 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-preview master: DNM: fix dockerfile https://review.openstack.org/640904 | 00:31 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: DNM: test fixed zuul-preview dockerfile https://review.openstack.org/640905 | 00:32 |
| openstackgerrit | Josef Wells proposed openstack-infra/zuul master: quickstart: mount playbooks volume for web image https://review.openstack.org/640906 | 00:33 |
| clarkb | corvus: the current run of ansible should docker compose up with up to date images | 00:33 |
| corvus | clarkb: cool, though i'm not sure if we have any outstanding changes at this point | 00:33 |
| clarkb | corvus: the gitea sign in | 00:34 |
| corvus | clarkb: ah nice | 00:34 |
| *** ekultails has quit IRC | 00:35 | |
| *** sreejithp has quit IRC | 00:37 | |
| *** uberjay has quit IRC | 00:38 | |
| *** uberjay has joined #openstack-infra | 00:38 | |
| *** jcoufal has quit IRC | 00:39 | |
| *** sdake has quit IRC | 00:41 | |
| *** wolverineav has quit IRC | 00:47 | |
| *** markvoelker has joined #openstack-infra | 00:48 | |
| *** ericyoung has quit IRC | 00:50 | |
| *** ericyoung has joined #openstack-infra | 00:51 | |
| clarkb | fwiw seems that afs02.dfw.openstack.org is currently not speaking ssh | 00:53 |
| clarkb | (it is holding up the ansible run, I was expecting the limestone mirror) | 00:53 |
| corvus | clarkb: looks like it stopped responding at 00:30 http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=6403&rra_id=5&view_type=&graph_start=1551744735&graph_end=1551747020&graph_height=120&graph_width=500&title_font_size=12 | 00:55 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/storyboard master: Use ColumnElements instead of strings in migration https://review.openstack.org/640892 | 00:57 |
| *** wolverineav has joined #openstack-infra | 00:58 | |
| fungi | clarkb: corvus: i don't suppose it's related to the cinder block device outage saturday? | 00:58 |
| clarkb | fungi: well it was working, but maybe it was unhappy and that finally tipped it over? | 00:58 |
| clarkb | I've actually got to pop out now as I said I'd take kids outside before sunset | 00:58 |
| fungi | rackspace opened a ticket saying they rebooted the backend providing afs02.dfw.openstack.org/main02 | 00:59 |
| clarkb | but can take a look after if still necessary | 00:59 |
| *** hwoarang has quit IRC | 00:59 | |
| openstackgerrit | Merged openstack-infra/system-config master: Reduce timeouts in run_all.sh https://review.openstack.org/636775 | 00:59 |
| openstackgerrit | Merged openstack-infra/system-config master: Replicate to the standalone gitea severs https://review.openstack.org/640431 | 00:59 |
| *** hwoarang has joined #openstack-infra | 01:00 | |
| corvus | (openstack) volume list | 01:00 |
| corvus | publicURL endpoint for volumev2 service in DFW region not found | 01:00 |
| corvus | why doesn't openstack work? | 01:00 |
| openstackgerrit | Merged openstack-infra/devstack-gate master: Add the setup support for stable/stein https://review.openstack.org/640641 | 01:01 |
| fungi | it works until you have to reboot something :/ | 01:01 |
| corvus | (openstack) console log show afs02.dfw.openstack.org | 01:02 |
| corvus | There is no such action: os-getConsoleOutput (HTTP 400) (Request-ID: req-a3b62e80-2627-455e-bcb8-dd9c523c50fc) | 01:02 |
| corvus | and now it's time for me to go too. | 01:02 |
| fungi | i think the console depends on the kernel not being totally crashed | 01:03 |
| *** ricolin has joined #openstack-infra | 01:03 | |
| fungi | the serial console anyway | 01:03 |
| fungi | hrm, though no the hypervisor should be buffering it | 01:04 |
| *** sdake has joined #openstack-infra | 01:07 | |
| ianw | yeah, i wonder if something's up, that server i'd "quickly" rebuild has been going for over an hour | 01:17 |
| *** markvoelker has quit IRC | 01:22 | |
| *** sthussey has quit IRC | 01:22 | |
| ianw | right, so it's the xvde that's causing all this snapshot foo foo | 01:28 |
| dmsimard | corvus: thanks, I must have screwed up somewhere in retiring 3 repos | 01:28 |
| dmsimard | I'm not finished cleaning up, I got tied up in some rabbit holes today :D | 01:30 |
| ianw | bridge.o.o doesn't appear to have any volumes attached though | 01:30 |
| dmsimard | ianw: yeah it really took more than an hour to spawn | 01:30 |
| dmsimard | everything is just sluggish | 01:31 |
| ianw | i can see it now, it fails mounting opt, that's what gets you into the prompt | 01:31 |
| dmsimard | oh, which sort of makes sense | 01:31 |
| dmsimard | because it's in the fstab and is probably not in the new VM | 01:31 |
| ianw | i'm either blind or where is the bridge.o.o volume? | 01:31 |
| ianw | yeah, and doesn't have appropriate don't mount flags | 01:31 |
| dmsimard | maybe we could try mounting a volume and give it a hint for xvde | 01:31 |
| dmsimard | ianw: iscsi ? | 01:31 |
| clarkb | the e is ephemeral. part of the flavor | 01:32 |
| dmsimard | oh | 01:32 |
| ianw | ahhhh, that's right | 01:32 |
| fungi | i was able to pull up the graphical console for afs02 | 01:35 |
| ianw | "data disk" it's called | 01:35 |
| fungi | buffer io error on dm-0 xvdc | 01:36 |
| fungi | and a bunch of hung kernel task warnings | 01:36 |
| fungi | and no working getty | 01:36 |
| *** dpawlik has joined #openstack-infra | 01:37 | |
| ianw | dependency failed for /opt & local file systems | 01:40 |
| ianw | so i wonder how to make this snapshotable | 01:40 |
| *** dpawlik has quit IRC | 01:42 | |
| fungi | probably can't snapshot servers using an ephemeral disk | 01:46 |
| *** rascasoft has joined #openstack-infra | 01:46 | |
| fungi | i sort of wonder if resize will have similar issues. may want to backup whatever we've got on that fs | 01:46 |
| ianw | fungi: i think maybe nofail would at least let it *boot*, and it would be a small swizzle to get it going | 01:46 |
| ianw | fungi: aiui, there's no resize as such, it's new image from snapshot | 01:47 |
| ianw | let me get into emergency console and add nofail, and see if it at least boots to a state we could then set the right uuid's and have a working server. if so, we can do that on bridge.o.o | 01:47 |
| fungi | ianw: the snapshot experiment was to make sure we've got a good copy of the server before we try to resize | 01:47 |
| *** dave-mccowan has joined #openstack-infra | 01:48 | |
| fungi | dmsimard confirmed he was able to use the api to do a resize of a server booted from the same image and flavor as bridge.o.o | 01:48 |
| *** gouthamr has quit IRC | 01:48 | |
| fungi | (even though the option isn't provided by the dashboard any longer) | 01:48 |
| dmsimard | yes, the expectation is that if you don't test your backups, they're broken :) | 01:48 |
| ianw | oh, ok, might have missed that bit. i'm guessing it's going to be just backend hiding making a new server anyway :) | 01:49 |
| fungi | dmsimard: i can attest to this ;) | 01:49 |
| dmsimard | ianw: so how would we restore from a snapshot should it be necessary ? | 01:49 |
| fungi | ianw: yes, i believe it is, though may reuse port and ip address | 01:49 |
| fungi | and uuid | 01:49 |
| ianw | dmsimard: the system would boot, modify the fstab for the newly attached ephemeral disk, reboot, and it should be back | 01:50 |
| ianw | i don't think anything on /opt required for boot | 01:50 |
| *** dmellado has quit IRC | 01:50 | |
| *** stevebaker has quit IRC | 01:50 | |
| *** rascasoft has quit IRC | 01:50 | |
| fungi | and hopefully we separately back up what's in /opt or just don't care if it gets blown away | 01:50 |
| dmsimard | ianw: well, I mean, there's a lot we can do here -- we can allow ourselves to take a second (fixed) snapshot :p | 01:50 |
| ianw | fungi: i'm assuming that when it ansible's itself, it writes /opt | 01:52 |
| ianw | and typing that, i just realised that the playbooks are probably on /opt/system-config | 01:52 |
| ianw | hrm, it seems like figuring this out now, while it's not a fire-drill, is probably not a bad idea | 01:52 |
| fungi | agreed | 01:52 |
| dmsimard | we have 128MB on /opt | 01:52 |
| dmsimard | this can live on / where we have 23GB available :D | 01:53 |
| ianw | it presumably wouldn't need more than system-config | 01:53 |
| dmsimard | we could probably even take out the swap since the plan is to upgrade to a flavor with more ram | 01:53 |
| fungi | so anyway, it seems that afs02.dfw is now not even able to get a graphical console connection any longer. going to try a reboot and see if it's any better | 01:53 |
| dmsimard | it all depends on the amount of effort for deploying a new bridge.o.o | 01:53 |
| *** jamesmcarthur has joined #openstack-infra | 01:55 | |
| *** hwoarang has quit IRC | 01:55 | |
| *** hwoarang has joined #openstack-infra | 01:57 | |
| *** wolverineav has quit IRC | 01:57 | |
| ianw | i think it's minimal | 01:59 |
| ianw | but a resize and fiddle two lines in fstab is even more minimal :) | 01:59 |
| fungi | assuming the resize doesn't revert and boot the old instance again if something goes sideways with the new instance it's trying to migrate to | 02:00 |
| fungi | the reboot of afs02.dfw seems to have made it reachable via ssh again | 02:01 |
| *** whoami-rajat has joined #openstack-infra | 02:02 | |
| fungi | #status log afs02.dfw rebooted after hanging, likely related to outage for main02 block device on 2019-03-02 | 02:02 |
| openstackstatus | fungi: finished logging | 02:02 |
| ianw | well nofail fixed up the boot | 02:04 |
| dmsimard | ianw: taking the snapshot and booting the VM takes a while though | 02:04 |
| *** jamesmcarthur has quit IRC | 02:04 | |
| ianw | guess what it doesn't have ... networking | 02:04 |
| dmsimard | I mean, we need to keep this in mind in the future in case of actual disaster recovery | 02:04 |
| dmsimard | ianw: I think I know that part | 02:04 |
| *** sdake has quit IRC | 02:05 | |
| ianw | yeah, we almost need to run virt-sysprep on it | 02:05 |
| dmsimard | ianw: I'm going to bet that the network config is persisted in /etc/netplan | 02:05 |
| *** rascasoft has joined #openstack-infra | 02:05 | |
| fungi | looking at afs02.dfw, the filesystem/logvol seems to be intact following reboot | 02:06 |
| *** jamesmcarthur has joined #openstack-infra | 02:06 | |
| fungi | and with that, i'm knocking off for the evening as well | 02:06 |
| *** jamesmcarthur has quit IRC | 02:10 | |
| *** anteaya has quit IRC | 02:11 | |
| *** rascasoft has quit IRC | 02:12 | |
| *** hongbin has joined #openstack-infra | 02:13 | |
| *** markvoelker has joined #openstack-infra | 02:19 | |
| ianw | dmsimard: ok, i got 104.239.240.196 up ... let's do a check list in https://etherpad.openstack.org/p/bridge-snapshot-bringup | 02:26 |
| dmsimard | I'll catch up later, need to take care of something | 02:27 |
| *** sdake has joined #openstack-infra | 02:28 | |
| openstackgerrit | Merged openstack-infra/storyboard master: Use ColumnElements instead of strings in migration https://review.openstack.org/640892 | 02:29 |
| *** hwoarang has quit IRC | 02:33 | |
| *** wolverineav has joined #openstack-infra | 02:33 | |
| *** hwoarang has joined #openstack-infra | 02:35 | |
| *** jamesmcarthur has joined #openstack-infra | 02:37 | |
| *** psachin has joined #openstack-infra | 02:39 | |
| ianw | dmsimard: ok, so https://etherpad.openstack.org/p/bridge-snapshot-bringup is a plan, if nothing else. worked with 104.239.240.196 , which i've now shut down to avoid confusion | 02:50 |
| *** gouthamr has joined #openstack-infra | 02:50 | |
| *** markvoelker has quit IRC | 02:51 | |
| *** jamesmcarthur has quit IRC | 02:51 | |
| *** jamesmcarthur has joined #openstack-infra | 02:54 | |
| *** dmellado has joined #openstack-infra | 02:55 | |
| *** wolverineav has quit IRC | 03:00 | |
| *** eventingmonkey has quit IRC | 03:01 | |
| *** wolverineav has joined #openstack-infra | 03:01 | |
| *** wolverineav has quit IRC | 03:06 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:10 | |
| *** jamesmcarthur has quit IRC | 03:10 | |
| *** jamesmcarthur has joined #openstack-infra | 03:10 | |
| *** apetrich has quit IRC | 03:15 | |
| openstackgerrit | Riju Khatri proposed openstack-infra/storyboard-webclient master: Add activity indicator to new worklist modal and detail https://review.openstack.org/640845 | 03:16 |
| *** roman_g has quit IRC | 03:17 | |
| *** yamamoto has joined #openstack-infra | 03:17 | |
| *** stevebaker has joined #openstack-infra | 03:25 | |
| *** wolverineav has joined #openstack-infra | 03:31 | |
| *** rkukura has quit IRC | 03:34 | |
| *** yamamoto has quit IRC | 03:35 | |
| *** yamamoto has joined #openstack-infra | 03:35 | |
| *** dpawlik has joined #openstack-infra | 03:38 | |
| *** dave-mccowan has quit IRC | 03:42 | |
| *** dpawlik has quit IRC | 03:43 | |
| *** markvoelker has joined #openstack-infra | 03:48 | |
| *** sdake has quit IRC | 03:51 | |
| *** gyee has quit IRC | 03:54 | |
| *** jamesmcarthur has quit IRC | 03:57 | |
| *** rascasoft has joined #openstack-infra | 04:00 | |
| *** rascasoft has quit IRC | 04:04 | |
| *** wolverineav has quit IRC | 04:07 | |
| *** ramishra has joined #openstack-infra | 04:09 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add triggers information to pipeline list https://review.openstack.org/637670 | 04:15 |
| *** udesale has joined #openstack-infra | 04:18 | |
| *** markvoelker has quit IRC | 04:22 | |
| *** ykarel|away has joined #openstack-infra | 04:31 | |
| *** ykarel|away is now known as ykarel | 04:31 | |
| *** raukadah is now known as chandankumar | 04:34 | |
| *** jamesmcarthur has joined #openstack-infra | 04:35 | |
| *** jamesmcarthur has quit IRC | 04:44 | |
| *** armax has quit IRC | 04:48 | |
| *** jamesmcarthur has joined #openstack-infra | 04:48 | |
| *** owalsh_ has joined #openstack-infra | 04:50 | |
| *** owalsh has quit IRC | 04:51 | |
| *** jamesmcarthur has quit IRC | 04:52 | |
| *** wolverineav has joined #openstack-infra | 05:00 | |
| *** jamesmcarthur has joined #openstack-infra | 05:00 | |
| *** wolverineav has quit IRC | 05:06 | |
| *** janki has joined #openstack-infra | 05:08 | |
| *** hongbin has quit IRC | 05:09 | |
| *** markvoelker has joined #openstack-infra | 05:19 | |
| *** udesale has quit IRC | 05:22 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add DIB_APT_MINIMAL_CREATE_INTERFACES toggle https://review.openstack.org/639865 | 05:28 |
| *** jamesmcarthur has quit IRC | 05:35 | |
| *** rkukura has joined #openstack-infra | 05:36 | |
| *** dpawlik has joined #openstack-infra | 05:39 | |
| *** dpawlik has quit IRC | 05:44 | |
| *** jhesketh has quit IRC | 05:47 | |
| *** jhesketh has joined #openstack-infra | 05:48 | |
| *** sdake has joined #openstack-infra | 05:48 | |
| AJaeger | fungi, clarkb, docs-old can be removed that is old content | 05:50 |
| *** sdake has quit IRC | 05:50 | |
| *** markvoelker has quit IRC | 05:51 | |
| *** pcaruana has joined #openstack-infra | 05:52 | |
| *** sdake has joined #openstack-infra | 05:56 | |
| *** jamesmcarthur has joined #openstack-infra | 06:03 | |
| *** pcaruana has quit IRC | 06:07 | |
| *** jamesmcarthur has quit IRC | 06:09 | |
| *** rh-jelabarre has quit IRC | 06:13 | |
| *** jtomasek has joined #openstack-infra | 06:16 | |
| *** dpawlik has joined #openstack-infra | 06:19 | |
| *** ccamacho has quit IRC | 06:21 | |
| *** dpawlik has quit IRC | 06:24 | |
| *** dims has quit IRC | 06:24 | |
| *** e0ne has joined #openstack-infra | 06:24 | |
| *** dims has joined #openstack-infra | 06:26 | |
| *** e0ne has quit IRC | 06:33 | |
| *** dims has quit IRC | 06:36 | |
| *** dims has joined #openstack-infra | 06:37 | |
| *** jamesmcarthur has joined #openstack-infra | 06:40 | |
| *** jamesmcarthur has quit IRC | 06:44 | |
| *** e0ne has joined #openstack-infra | 06:46 | |
| *** markvoelker has joined #openstack-infra | 06:48 | |
| *** wolverineav has joined #openstack-infra | 06:48 | |
| *** wolverineav has quit IRC | 06:53 | |
| *** jamesmcarthur has joined #openstack-infra | 06:56 | |
| *** jamesmcarthur has quit IRC | 07:01 | |
| *** dayou has quit IRC | 07:01 | |
| *** dayou has joined #openstack-infra | 07:02 | |
| *** AJaeger has quit IRC | 07:05 | |
| *** e0ne has quit IRC | 07:07 | |
| *** quiquell|off is now known as quiquell | 07:08 | |
| *** AJaeger has joined #openstack-infra | 07:09 | |
| *** slaweq has joined #openstack-infra | 07:11 | |
| *** wolverineav has joined #openstack-infra | 07:14 | |
| *** wolverineav has quit IRC | 07:14 | |
| *** pgaxatte has joined #openstack-infra | 07:16 | |
| *** markvoelker has quit IRC | 07:22 | |
| *** aojea has joined #openstack-infra | 07:23 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Add DIB_APT_MINIMAL_CREATE_INTERFACES toggle https://review.openstack.org/639865 | 07:24 |
| *** dpawlik has joined #openstack-infra | 07:25 | |
| *** stakeda has joined #openstack-infra | 07:30 | |
| openstackgerrit | Ivoline Ngong proposed openstack-infra/storyboard master: Updated_at content modified https://review.openstack.org/638840 | 07:30 |
| *** flaper87 has quit IRC | 07:32 | |
| *** flaper87 has joined #openstack-infra | 07:33 | |
| *** yamamoto has quit IRC | 07:34 | |
| *** kopecmartin|off is now known as kopecmartin | 07:34 | |
| *** quiquell is now known as quiquell|brb | 07:35 | |
| *** yamamoto has joined #openstack-infra | 07:36 | |
| *** dayou has quit IRC | 07:37 | |
| mordred | infra-root: I'm up with an upset stomach - so I'm up, not actually here - and may not be as here in the morning as I might like | 07:38 |
| mordred | infra-root: that said - the gitea dockers are having issues - upon first inspection it's something in the mariadb container - seems like a bonged restart of some sort? | 07:38 |
| *** apetrich has joined #openstack-infra | 07:43 | |
| mordred | infra-root: on gitea01 I moved /var/gitea/db/tc.log out of the way and it seems to have been able to recover | 07:50 |
| mordred | I moved it out of the way on all of the hosts and things seem to be better | 07:50 |
| mordred | the old tc.log files are all in /var/gitea | 07:51 |
| mordred | in case anyone wants to post-mortem | 07:51 |
| *** udesale has joined #openstack-infra | 07:51 | |
| mordred | looking at https://github.com/docker-library/mariadb/issues/201 we may want to tie our mariadb image to a specific version rather than latest | 07:53 |
| mordred | and with that - I shall go back to communing with my stomach | 07:54 |
| *** snapiri has joined #openstack-infra | 07:57 | |
| *** dayou has joined #openstack-infra | 07:58 | |
| *** udesale has quit IRC | 08:01 | |
| AJaeger | mordred: get well soon! | 08:05 |
| openstackgerrit | Ivoline Ngong proposed openstack-infra/storyboard master: Updated_at content modified https://review.openstack.org/638840 | 08:06 |
| *** quiquell|brb is now known as quiquell | 08:07 | |
| *** kjackal has joined #openstack-infra | 08:09 | |
| *** rascasoft has joined #openstack-infra | 08:09 | |
| *** rpittau|afk is now known as rpittau | 08:16 | |
| *** rh-jelabarre has joined #openstack-infra | 08:17 | |
| *** e0ne has joined #openstack-infra | 08:17 | |
| *** markvoelker has joined #openstack-infra | 08:18 | |
| *** sajauddin has joined #openstack-infra | 08:18 | |
| *** pcaruana has joined #openstack-infra | 08:18 | |
| *** jtomasek has quit IRC | 08:20 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Pin mariadb to 10.4 https://review.openstack.org/640949 | 08:20 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add a stop timeout to gitea docker-compose up https://review.openstack.org/640950 | 08:20 |
| mordred | AJaeger: thanks! I think the worst of it right now is simply the fact that I'm awake | 08:20 |
| *** tkajinam has quit IRC | 08:22 | |
| *** owalsh_ is now known as owalsh | 08:24 | |
| *** pcaruana has quit IRC | 08:25 | |
| *** ginopc has joined #openstack-infra | 08:25 | |
| *** yolanda has joined #openstack-infra | 08:25 | |
| *** helenfm has joined #openstack-infra | 08:29 | |
| *** ccamacho has joined #openstack-infra | 08:29 | |
| *** sdake has quit IRC | 08:36 | |
| *** pcaruana has joined #openstack-infra | 08:37 | |
| *** ykarel is now known as ykarel|lunch | 08:41 | |
| *** sajauddin has left #openstack-infra | 08:42 | |
| *** priteau has joined #openstack-infra | 08:42 | |
| *** sajauddin has joined #openstack-infra | 08:43 | |
| *** roman_g has joined #openstack-infra | 08:43 | |
| *** pcaruana has quit IRC | 08:44 | |
| *** sajauddin_ has joined #openstack-infra | 08:47 | |
| *** sajauddin_ has left #openstack-infra | 08:48 | |
| *** markvoelker has quit IRC | 08:51 | |
| *** jpena|off is now known as jpena | 08:52 | |
| *** jpich has joined #openstack-infra | 08:56 | |
| *** e0ne has quit IRC | 08:58 | |
| *** iurygregory has quit IRC | 08:58 | |
| *** e0ne has joined #openstack-infra | 09:00 | |
| *** pcaruana has joined #openstack-infra | 09:01 | |
| *** iurygregory has joined #openstack-infra | 09:01 | |
| openstackgerrit | Rico Lin proposed openstack-infra/project-config master: Add an openstack/auto-scaling-sig repository https://review.openstack.org/637125 | 09:08 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] Web: plug the authorization engine https://review.openstack.org/640884 | 09:10 |
| *** pgaxatte has quit IRC | 09:10 | |
| *** pgaxatte has joined #openstack-infra | 09:12 | |
| *** panda|ruck|off is now known as panda|ruck|flu | 09:13 | |
| *** roman_g has quit IRC | 09:20 | |
| *** geguileo has joined #openstack-infra | 09:20 | |
| AJaeger | mordred: yeah, that is always bad ;( | 09:20 |
| *** dtantsur|afk is now known as dtantsur | 09:22 | |
| openstackgerrit | Ivoline Ngong proposed openstack-infra/storyboard master: Updated_at content modified https://review.openstack.org/638840 | 09:22 |
| *** bhavikdbavishi has quit IRC | 09:27 | |
| *** bhavikdbavishi has joined #openstack-infra | 09:27 | |
| ricolin | fungi, test job in https://review.openstack.org/637125 is green now | 09:30 |
| *** IvensZambrano has joined #openstack-infra | 09:31 | |
| *** stakeda has quit IRC | 09:31 | |
| *** derekh has joined #openstack-infra | 09:36 | |
| *** ykarel|lunch is now known as ykarel | 09:39 | |
| *** roman_g has joined #openstack-infra | 09:41 | |
| *** markvoelker has joined #openstack-infra | 09:48 | |
| openstackgerrit | Camila Moura proposed openstack-infra/storyboard master: Creates the tag with project name and priority https://review.openstack.org/640793 | 09:50 |
| openstackgerrit | Ivoline Ngong proposed openstack-infra/storyboard master: Updated_at content modified https://review.openstack.org/638840 | 09:52 |
| openstackgerrit | Camila Moura proposed openstack-infra/storyboard master: Creates the tag with project name and priority https://review.openstack.org/640793 | 09:53 |
| *** sdake has joined #openstack-infra | 09:55 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] Web: plug the authorization engine https://review.openstack.org/640884 | 09:55 |
| *** bhavikdbavishi has quit IRC | 10:00 | |
| *** sajauddin has quit IRC | 10:02 | |
| *** cmoura has joined #openstack-infra | 10:03 | |
| *** yamamoto has quit IRC | 10:07 | |
| *** kjackal has quit IRC | 10:08 | |
| ttx | fungi, clarkb: regarding zuul.opendev.org doing 404s on direct links -- tristanC points to missing mod_rewrite rules. The rules in puppet-zuul (https://git.openstack.org/cgit/openstack-infra/puppet-zuul/tree/templates/zuulv3.vhost.erb#n53) are indeed slightly different from what Zuul docs mandate (https://zuul-ci.org/docs/zuul/admin/installation.html#static-offload) | 10:09 |
| ttx | However I feel like there is some history I'm missing there, so I'll hold before proposing a fix | 10:10 |
| *** kjackal has joined #openstack-infra | 10:10 | |
| *** gfidente has joined #openstack-infra | 10:12 | |
| *** luizbag has joined #openstack-infra | 10:12 | |
| *** markvoelker has quit IRC | 10:22 | |
| openstackgerrit | Ivoline Ngong proposed openstack-infra/storyboard master: Updated_at content modified https://review.openstack.org/638840 | 10:27 |
| *** yamamoto has joined #openstack-infra | 10:42 | |
| *** sdake has quit IRC | 10:51 | |
| *** sdake has joined #openstack-infra | 10:55 | |
| *** bhavikdbavishi has joined #openstack-infra | 10:56 | |
| *** electrofelix has joined #openstack-infra | 10:57 | |
| *** markvoelker has joined #openstack-infra | 11:19 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] Web: plug the authorization engine https://review.openstack.org/640884 | 11:21 |
| *** janki has quit IRC | 11:22 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/elastic-recheck master: Make the functional queries test work with python3 https://review.openstack.org/640984 | 11:26 |
| frickler | infra-root: mriedem: ^^ this seems to have been broken for a while, not sure how it was passing earlier, though, since the move to py3 was well before we started seeing all failures http://zuul.openstack.org/builds?job_name=elastic-recheck-tox-queries | 11:28 |
| *** e0ne has quit IRC | 11:32 | |
| *** e0ne has joined #openstack-infra | 11:42 | |
| *** rascasoft has quit IRC | 11:43 | |
| *** markvoelker has quit IRC | 11:51 | |
| *** bhavikdbavishi has quit IRC | 11:54 | |
| *** udesale has joined #openstack-infra | 12:00 | |
| *** jtomasek has joined #openstack-infra | 12:05 | |
| *** ociuhandu has quit IRC | 12:14 | |
| *** ccamacho has quit IRC | 12:17 | |
| *** dave-mccowan has joined #openstack-infra | 12:20 | |
| *** xek has joined #openstack-infra | 12:27 | |
| *** apetrich has quit IRC | 12:30 | |
| *** jpena is now known as jpena|lunch | 12:31 | |
| *** apetrich has joined #openstack-infra | 12:31 | |
| *** apetrich has quit IRC | 12:38 | |
| *** sdake has quit IRC | 12:39 | |
| *** mriedem has joined #openstack-infra | 12:41 | |
| *** yamamoto has quit IRC | 12:44 | |
| *** yamamoto has joined #openstack-infra | 12:46 | |
| *** tosky has joined #openstack-infra | 12:48 | |
| *** markvoelker has joined #openstack-infra | 12:49 | |
| mriedem | does something need to be restarted to kick http://status.openstack.org/elastic-recheck/data/integrated_gate.html ? | 12:50 |
| mriedem | we fixed the backing query that populates that page yesterday but the results are still from feb 8 | 12:51 |
| *** yamamoto has quit IRC | 12:52 | |
| *** apetrich has joined #openstack-infra | 12:56 | |
| *** yamamoto has joined #openstack-infra | 12:58 | |
| *** panda|ruck|flu is now known as panda|ruck|lunch | 13:03 | |
| *** gfidente has quit IRC | 13:06 | |
| *** weshay is now known as weshay|rover | 13:09 | |
| frickler | mriedem: hmm, it does say "Generated at: 2019-03-05T12:50" for me. maybe you need to force-reload? | 13:10 |
| mriedem | ah yes that always gets me | 13:11 |
| frickler | mriedem: did you see my comment about https://review.openstack.org/640984 before? I stumbled about that when looking at your fix earlier | 13:12 |
| mriedem | nope | 13:14 |
| *** yamamoto has quit IRC | 13:14 | |
| mriedem | ah i see http://logs.openstack.org/69/640869/1/check/elastic-recheck-tox-queries/7b6eae5/testr_results.html.gz | 13:14 |
| mriedem | hmm, i dont know why elastic-recheck-tox-queries is non-voting | 13:14 |
| frickler | from my understanding it should have been broken since the basepython was changed to py3, but it was passing until end of january. sadly the logs from then have expired http://zuul.openstack.org/builds?job_name=elastic-recheck-tox-queries | 13:16 |
| *** quiquell is now known as quiquell|lunch | 13:16 | |
| *** janki has joined #openstack-infra | 13:17 | |
| *** jamesmcarthur has joined #openstack-infra | 13:18 | |
| *** sdake has joined #openstack-infra | 13:21 | |
| *** markvoelker has quit IRC | 13:21 | |
| *** jesusaur has quit IRC | 13:22 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Make the functional queries test work with python3 https://review.openstack.org/640984 | 13:23 |
| *** jesusaur has joined #openstack-infra | 13:25 | |
| frickler | corvus: clarkb: according to this doc we should be able to reload the replication config instead of doing a full gerrit restart, do we want to try that? https://gerrit.googlesource.com/plugins/replication/+doc/master/src/main/resources/Documentation/config.md#enabling-replication | 13:30 |
| frickler | didn't check whether that would need a newer gerrit version, though | 13:30 |
| *** dtantsur is now known as dtantsur|brb | 13:31 | |
| *** jamesmcarthur has quit IRC | 13:31 | |
| *** rlandy has joined #openstack-infra | 13:36 | |
| *** oanson has joined #openstack-infra | 13:36 | |
| *** quiquell|lunch is now known as quiquell | 13:39 | |
| *** jpena|lunch is now known as jpena | 13:40 | |
| *** priteau has quit IRC | 13:41 | |
| *** dpawlik has quit IRC | 13:43 | |
| *** jamesmcarthur has joined #openstack-infra | 13:46 | |
| *** rascasoft has joined #openstack-infra | 13:50 | |
| *** panda|ruck|lunch is now known as panda|ruck|flu | 13:50 | |
| *** priteau has joined #openstack-infra | 13:52 | |
| *** yamamoto has joined #openstack-infra | 13:53 | |
| *** jcoufal has joined #openstack-infra | 13:53 | |
| Shrews | infra-root: codesearch is giving a "too many open files" error. is the solution to restart houndd? | 14:00 |
| *** yamamoto_ has joined #openstack-infra | 14:01 | |
| * Shrews tries | 14:02 | |
| *** gfidente has joined #openstack-infra | 14:03 | |
| *** yamamoto has quit IRC | 14:03 | |
| *** sthussey has joined #openstack-infra | 14:05 | |
| *** jamesmcarthur has quit IRC | 14:06 | |
| *** kgiusti has joined #openstack-infra | 14:06 | |
| Shrews | seems to be working again | 14:06 |
| Shrews | #status log houndd restarted on codesearch to correct "too many open files" error | 14:06 |
| openstackstatus | Shrews: finished logging | 14:06 |
| iurygregory | Hi everyone, in zuulv2 we had the BASE env variable set but in Zuulv3 this changed? | 14:11 |
| *** sdake has quit IRC | 14:11 | |
| *** janki has quit IRC | 14:12 | |
| *** janki has joined #openstack-infra | 14:12 | |
| *** dklyle has joined #openstack-infra | 14:13 | |
| *** dpawlik has joined #openstack-infra | 14:13 | |
| *** ykarel is now known as ykarel|away | 14:14 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] Web: plug the authorization engine https://review.openstack.org/640884 | 14:17 |
| *** ykarel|away has quit IRC | 14:18 | |
| *** markvoelker has joined #openstack-infra | 14:18 | |
| *** ykarel|away has joined #openstack-infra | 14:19 | |
| *** sdake has joined #openstack-infra | 14:20 | |
| frickler | Shrews: I've seen that occasionally, but always worked again after a reload/new search | 14:22 |
| *** sdake has quit IRC | 14:23 | |
| Shrews | frickler: it seemed to be affecting git operations by hound, according to the log. at any rate, seems ok for now | 14:25 |
| *** asettle has joined #openstack-infra | 14:32 | |
| jangutter | has anyone got a magic spell for disk-image-builder that will assemble an EFI bootable whole-disk-image for centos7 ? | 14:32 |
| *** sdake has joined #openstack-infra | 14:33 | |
| pabelanger | jangutter: I think yolanda or ianw might know more | 14:34 |
| jangutter | pabelanger: thanks, for reference, I tried - centos7 block-device-gpt growroot dhcp-all-interfaces openssh-server cloud-init grub2 stable-interface-names bootloader | 14:35 |
| jangutter | pabelanger: a small blue imp appeared in a sulfurous haze, but nothing much else. | 14:36 |
| pabelanger | jangutter: http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/nb03.openstack.org.yaml#n63 is the nodepool config we are using for arm64 images, but only debuntu right now | 14:37 |
| pabelanger | not sure why we didn't do centos | 14:37 |
| pabelanger | or fedora | 14:37 |
| jangutter | pabelanger: python3 probably :-p | 14:37 |
| jangutter | pabelanger: are those booting baremetal nodes, or virtual? | 14:38 |
| pabelanger | vms | 14:38 |
| pabelanger | but require EFI, our only cloud that does right now | 14:39 |
| jangutter | pabelanger: ah, added wrinkle, I'm trying baremetal. | 14:39 |
| jangutter | pabelanger: ...again.... | 14:40 |
| pabelanger | jangutter: maybe yolanda knows more in this case | 14:40 |
| iurygregory | Does anyone have any idea what var I can use instead of BASE when switching to Zuulv3? (http://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/playbooks/devstack_gate_vars.yaml) | 14:40 |
| *** dtantsur|brb is now known as dtantsur | 14:40 | |
| iurygregory | https://review.openstack.org/#/c/633010/18/ironicclient/tests/functional/hooks/post_test_hook.sh it seems to be empty when running with zuulv3 | 14:41 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] Web: plug the authorization engine https://review.openstack.org/640884 | 14:46 |
| pabelanger | iurygregory: you need to use environment: '{{ zuul | zuul_legacy_vars }}' to access legacy variables | 14:48 |
| pabelanger | I'm not sure what related BASE however | 14:48 |
| *** sdake has quit IRC | 14:48 | |
| yolanda | hi so yes, i have composed bootable EFI images | 14:48 |
| yolanda | in the context of TripleO but it will be useful as a sample anyway, let me check links | 14:49 |
| iurygregory | pabelanger, even if i'm moving a job to zuulv3 I can use zuul_legacy_vars? | 14:49 |
| pabelanger | iurygregory: you could, but likely shouldn't. Depending where base is defined, you might need to template your hook via jinja2 | 14:50 |
| yolanda | jangutter, pabelanger , see https://github.com/openstack/tripleo-common/blob/master/image-yaml/overcloud-hardened-images-uefi.yaml and https://github.com/openstack/tripleo-common/blob/master/image-yaml/overcloud-hardened-images-uefi-centos7.yaml | 14:50 |
| pabelanger | eg: {{ base }} | 14:50 |
| yolanda | the trick is to add this element http://git.openstack.org/cgit/openstack/tripleo-image-elements/tree/elements/overcloud-secure-uefi | 14:50 |
| iurygregory | pabelanger, its possible to create an env var in the job definition? | 14:51 |
| *** sdake has joined #openstack-infra | 14:51 | |
| pabelanger | iurygregory: yes: https://zuul-ci.org/docs/zuul/user/jobs.html#variables | 14:51 |
| *** markvoelker has quit IRC | 14:52 | |
| jangutter | yolanda: is there a minimum DIB version required for that element? | 14:52 |
| iurygregory | pabelanger, going to try to set BASE in the job definition to see if it works | 14:53 |
| jangutter | yolanda: (I'm running 2.18.0) | 14:53 |
| pabelanger | iurygregory: you'll still need to pass the variable into the environment field for the shell task. I want to say there is a better way to do this, but I'm currently not sure. I'd also ask in #openstack-qa, they've done a good job rewriting jobs | 14:54 |
| jangutter | yolanda: looking at my spell, the delta is block-device-default.yaml I think. | 14:54 |
| iurygregory | pabelanger, tks for the help i will try to ask them too =) | 14:54 |
| *** pcaruana has quit IRC | 14:57 | |
| yolanda | jangutter, it's been there for some time. I don't remember the exact tag when it came, but anything from the last 6 months will be ok | 14:58 |
| yolanda | in order to make uefi work, you need to define an special partition layout | 14:58 |
| *** munimeha1 has joined #openstack-infra | 14:59 | |
| *** ekultails has joined #openstack-infra | 15:00 | |
| *** e0ne has quit IRC | 15:02 | |
| *** ccamacho has joined #openstack-infra | 15:03 | |
| *** sdake has quit IRC | 15:04 | |
| jangutter | yolanda: ahhh, so block-device-efi is close, but not close enough. | 15:05 |
| logan- | jangutter: I see above you were trying block-device-gpt, but wouldn't you rather use block-device-uefi instead? | 15:06 |
| *** sdake has joined #openstack-infra | 15:06 | |
| jangutter | logan: huh... drat, why is it gpt and not efi? | 15:06 |
| logan- | block-device-efi I mean | 15:06 |
| jangutter | logan-: thanks, that's my n-th PEBCAK moment of the day. | 15:07 |
| logan- | block-device-gpt creates a GPT labelled root disk with a BSP instead of an ESP | 15:07 |
| logan- | np :) hope it works for you | 15:07 |
| jangutter | logan: | 15:08 |
| jangutter | logan-: That makes sense, you need Extra Sensory Perception rather than just Baffling Stupid Problems. | 15:08 |
| *** eharney has joined #openstack-infra | 15:09 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add an openstack/auto-scaling-sig repository https://review.openstack.org/637125 | 15:09 |
| fungi | iurygregory: you might have more luck asking in #openstack-qa if you're trying to use the zuul v3 job definition from the devstack repo, but looks like it uses devstack_base_dir as the ansible variable name for that: http://git.openstack.org/cgit/openstack-dev/devstack/tree/roles/run-devstack/defaults/main.yaml | 15:09 |
| fungi | (i strongly recommend against altering "legacy" devstack-gate jobs, they're not really maintainable) | 15:10 |
| iurygregory | fungi, yeah i saw that, trying to figure out if its possible to call that variable in the shell | 15:10 |
| *** anteaya has joined #openstack-infra | 15:10 | |
| fungi | iurygregory: you can use jinja2 template variable expansion in shell or command tasks in your ansible | 15:11 |
| fungi | or i think you can pass shell and/or command tasks an environment array | 15:12 |
| tosky | fungi: I suggested the usage of tox_environment | 15:12 |
| iurygregory | fungi, ack 1st I'm going to try with tox_environment to see if goes | 15:12 |
| fungi | yeah, if you're trying to call it from within the scope of something tox is running, then that ought to be the place to plumb it in | 15:12 |
| *** bhavikdbavishi has joined #openstack-infra | 15:12 | |
| *** anteaya has left #openstack-infra | 15:13 | |
| *** anteaya has joined #openstack-infra | 15:13 | |
| *** dpawlik has quit IRC | 15:13 | |
| *** e0ne has joined #openstack-infra | 15:13 | |
| *** anteaya has quit IRC | 15:17 | |
| *** openstackgerrit has quit IRC | 15:28 | |
| jangutter | pabelanger, yolanda, logal-: switching block-device-gpt with block-device-efi made it go further (it's now failing to mount /sysroot). I'm pretty sure I can debug it further, thanks! | 15:32 |
| *** jamesmcarthur has joined #openstack-infra | 15:37 | |
| yolanda | jangutter, good! | 15:41 |
| yolanda | let me know if you need some help | 15:41 |
| jangutter | yolanda: thanks very much! | 15:42 |
| *** pcaruana has joined #openstack-infra | 15:42 | |
| priteau | jangutter: I've built CentOS7 EFI whole disk images for Ironic before. I use these options: | 15:43 |
| priteau | -p biosdevname,gdisk block-device-efi disable-selinux enable-serial-console grub2 vm | 15:43 |
| *** pgaxatte has quit IRC | 15:43 | |
| jangutter | yolanda: looking at blkid on the node, it looks like there are two partitions with LABEL="cloudimg-rootfs".... I'm not exactly sure where that gets rendered. | 15:44 |
| jangutter | priteau: yeah, block-device-gpt is _not_ block-device-efi | 15:45 |
| priteau | Make sure you use the "vm" element as well, to create a whole disk image | 15:45 |
| *** armax has joined #openstack-infra | 15:46 | |
| *** chandankumar is now known as raukadah | 15:47 | |
| *** markvoelker has joined #openstack-infra | 15:48 | |
| *** sdake has quit IRC | 15:49 | |
| fungi | jangutter: also, while i find this discussion actually riveting and sort of on-topic for here, we do also have a dedicated #openstack-dib channel which might reach even more of the people you want for such future dib-related troubleshooting | 15:50 |
| *** udesale has quit IRC | 15:50 | |
| jangutter | fungi, PEBCAK n+1 moment of the day. | 15:50 |
| jangutter | fungi, thanks, I did not even know that existed, | 15:51 |
| fungi | no worries, not complaining or anything, just thought you might not know it's there | 15:51 |
| jangutter | fungi: in fact next time I shall try harder to find the IRC channel for a particular tool. | 15:52 |
| *** luizbag has quit IRC | 15:53 | |
| clarkb | frickler: we can test that reload replication command against review-dev and try it on production if it works on dev | 15:55 |
| fungi | jangutter: we have https://wiki.openstack.org/wiki/IRC as an index, though no guarantee it's complete | 15:55 |
| clarkb | ttx: I think we've decided to have zuul serve the static assets for simplicity of deployment, then if we need to we can have apache cache them on the way through | 15:55 |
| clarkb | ttx: what are the 404s on direct links? | 15:56 |
| fungi | though also the rewrite rules in there look basically the same to me as well | 15:56 |
| ttx | clarkb: https://storyboard.openstack.org/#!/story/2004987 | 15:58 |
| *** openstackgerrit has joined #openstack-infra | 15:59 | |
| *** xek has quit IRC | 15:59 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] Web: plug the authorization engine https://review.openstack.org/640884 | 15:59 |
| ttx | see example reproduction there | 15:59 |
| fungi | our t/(.*) rule is first instead of last and slightly more greedy | 15:59 |
| *** xek has joined #openstack-infra | 15:59 | |
| openstackgerrit | Rico Lin proposed openstack-infra/irc-meetings master: Change Heat meeting time https://review.openstack.org/641062 | 15:59 |
| fungi | ttx: note that when looking at that erb template, it's the <% else -%> block which is relevant to the zuul.opendev.org vhost | 16:00 |
| *** sdake has joined #openstack-infra | 16:00 | |
| ttx | yep, I understood that much | 16:00 |
| ttx | was just wondering (1) why there were small differences in the Rewrite set and (2) if those small differences could actually trigger that issue | 16:01 |
| clarkb | let me grab the actual file content as that might be easier to read | 16:02 |
| fungi | already doing it | 16:02 |
| mordred | fungi, clarkb, ttx: that direct link works for me | 16:02 |
| fungi | ttx: clarkb: this is how it gets expanded on the server: http://paste.openstack.org/show/747292/ | 16:03 |
| clarkb | fungi beat me to it | 16:03 |
| fungi | mordred: it has something to do with having to browse to a different url first and then it works | 16:03 |
| clarkb | mordred: ya it works for me too so I guess I've hit the different url first? | 16:04 |
| mordred | but - I can confirm that refreshing the status page for the openstack tenant fails | 16:04 |
| fungi | right, that's the easier way to reproduce | 16:04 |
| *** ricolin has quit IRC | 16:04 | |
| *** raukadah is now known as chandankumar | 16:04 | |
| *** bhavikdbavishi has quit IRC | 16:05 | |
| mordred | if we're deferring to the underlying zuul-web process, do we need any of those other than the ws:// one? | 16:06 |
| clarkb | REQUEST_URI == REQUEST_FILENAME says apache docs | 16:07 |
| clarkb | mordred: no, I don't think so, but we seem to not be deferring currently or atetmpting to not defer | 16:07 |
| mordred | clarkb: nod. maybe it's worth shifting our approach to deferring? | 16:08 |
| mordred | but also - don't listen to me - my brain is not working great this morning | 16:08 |
| ttx | basically that bug prevents from linking to one specific tenant status page | 16:09 |
| ttx | which kind of sucks link-wise | 16:09 |
| clarkb | we lack RewriteRule t/(.*)/(.*).html(.*) /t/$1/$2$3 [R=301,L,NE] ? | 16:10 |
| mordred | ttx: yeah. totally. it's definitely an important bug to fix | 16:10 |
| clarkb | which is the backward compatible rewrite, maybe its less about just backward compat and necessary? | 16:10 |
| *** bhavikdbavishi has joined #openstack-infra | 16:10 | |
| clarkb | though I guess there is no .html to match here | 16:11 |
| jangutter | pabelanger: you'll be glad to know it's not necessary to hack nodepool to do the "triple-image" thing for baremetal UEFI boot. | 16:14 |
| openstackgerrit | Doug Wiegley proposed openstack-infra/elastic-recheck master: Add query for 1818696, weird qos port delete issue https://review.openstack.org/641067 | 16:14 |
| jangutter | pabelanger: lets hope that part of Cloud history never troubles us again. | 16:14 |
| fungi | i think that was mostly an aws-ism which got cargo-culted into unnecessary places | 16:15 |
| clarkb | mordred: fungi /api/tenant/openstack/status HTTP/1.1" 200 171639 "https://zuul.opendev.org/t/openstack/status" is a working request path | 16:15 |
| clarkb | /t/openstack/status HTTP/1.1" 404 726 "https://storyboard.openstack.org/" is the failing | 16:15 |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Revert "Revert "Save registry logs"" https://review.openstack.org/640561 | 16:15 |
| mordred | clarkb: /t/openstack/status should be being rewritten to serve index.html | 16:16 |
| mordred | but that's apparently happening | 16:16 |
| clarkb | RewriteRule ^/t/(.*) /$1 [L] is our first rule | 16:17 |
| clarkb | so that becomes /openstack/status | 16:17 |
| clarkb | and the [L] means stop processing? | 16:17 |
| clarkb | also that rule isn't in our docs | 16:17 |
| mordred | I think that rule might be a thing that should go away | 16:18 |
| clarkb | ya that may be our fix | 16:18 |
| clarkb | https://git.openstack.org/cgit/openstack-infra/puppet-zuul/tree/templates/zuulv3.vhost.erb#n48 is where that comes from | 16:18 |
| openstackgerrit | Riju Khatri proposed openstack-infra/storyboard-webclient master: Fix project group view checkboxes https://review.openstack.org/640590 | 16:21 |
| *** markvoelker has quit IRC | 16:22 | |
| openstackgerrit | Riju Khatri proposed openstack-infra/storyboard-webclient master: Show in-progress indicator when opening a board https://review.openstack.org/640593 | 16:23 |
| openstackgerrit | Doug Wiegley proposed openstack-infra/elastic-recheck master: Add query for 1818697, fullstack qos bug https://review.openstack.org/641071 | 16:26 |
| mordred | infra-root: re last night's mariadb restart/tc.log thing, Shrews found https://jira.mariadb.org/browse/MDEV-16442 which I have commented on. best we can tell, tc.log is useful for us, so updating $something to just always delete it during restarts is probably the safest action we can take, amusingly enough | 16:27 |
| Shrews | s/useful/useless/ | 16:27 |
| corvus | Shrews: thank you i was struggling with that :) | 16:27 |
| Shrews | :) | 16:28 |
| clarkb | mordred: is or isn't useful? | 16:28 |
| mordred | isn't | 16:28 |
| mordred | listen to Shrews | 16:28 |
| clarkb | ah ya sorry early still | 16:28 |
| corvus | Shrews, mordred: maybe we should change the docker command? | 16:29 |
| *** anteaya has joined #openstack-infra | 16:29 | |
| *** pcrews has joined #openstack-infra | 16:29 | |
| clarkb | mordred: going back to zuul tenant urls. is t/openstack/status a valid zuul url if we pass it through to index.html (and remove the t/* rewrite? | 16:29 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Delete tc.log file when we start mariadb https://review.openstack.org/641073 | 16:31 |
| mordred | corvus: ^^ do you think something liek that would work? | 16:31 |
| mordred | corvus: or do you think instead we should change the command on the mariadb entry to be "rm -f /var/lib/mysql/tc.log && whatever-the-command-was-before" | 16:32 |
| Shrews | mordred: any chance mariadb upgraded itself recently? | 16:32 |
| Shrews | i ask b/c of https://github.com/docker-library/mariadb/issues/201 | 16:32 |
| clarkb | Shrews: our docker compose was updated to do a pull before up | 16:33 |
| corvus | Shrews, clarkb: likely so, this will prevent further upgrades: https://review.openstack.org/640949 | 16:33 |
| corvus | mordred: i'm looking into your change | 16:34 |
| mordred | corvus: I'm not sure if it would get re-run if only the mariadb image updated | 16:34 |
| mordred | corvus: oh - maybe it should use the mariadb image instead of ubuntu | 16:34 |
| openstackgerrit | Merged openstack-infra/system-config master: Pin mariadb to 10.4 https://review.openstack.org/640949 | 16:34 |
| mordred | so that it would be clear to docker-compose that it needs to run after pull? | 16:35 |
| corvus | mordred: interesting supposition. that *seems* like it should work. probably worth a test. | 16:35 |
| mordred | lemme push an update | 16:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Delete tc.log file when we start mariadb https://review.openstack.org/641073 | 16:36 |
| corvus | mordred: the other thing i was looking into is whether depends_on is effective. it waits until the service is "started", which, given that it is a single command, should mean that the rm command is run before mariadb is started. | 16:36 |
| *** quiquell is now known as quiquell|off | 16:37 | |
| corvus | (depends_on is largely ineffective for long running services because docker has no idea when they are ready, only when they are started. but "started" is probably good enough for a single command.) | 16:37 |
| corvus | though, since all docker commands are run in the foreground, i suppose it's still technically possible to be a little racy | 16:38 |
| fungi | is this related to the zuul-preview.o.o outate? | 16:38 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] Web: plug the authorization engine https://review.openstack.org/640884 | 16:38 |
| mordred | corvus: yeah - I was actually copy-pasting that from the zuul quick-start | 16:38 |
| corvus | mordred: all in all, it may be better to just change the startup command? | 16:38 |
| corvus | at least there are fewer moving parts that way | 16:38 |
| corvus | fungi: the what? | 16:39 |
| fungi | the container deployment errors/mariadb issue mordred mentioned overnight | 16:39 |
| * fungi rereads | 16:39 | |
| mordred | corvus: indeed. the mariadb image has ENTRYPOINT docker-entrypoint.sh and mysqld as CMD | 16:39 |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-zuul master: Don't rewrite t/ prefix on global zuul vhost https://review.openstack.org/641077 | 16:39 |
| corvus | fungi: mariadb is used for gitea. what's a zuul-preview outate? is it an update or outage? | 16:39 |
| clarkb | mordred: fungi ttx ^ I think that may do it | 16:39 |
| fungi | outage, yes sorry | 16:40 |
| corvus | fungi: i didn't know there was one. where can i learn more? | 16:40 |
| fungi | yeah, rereading scrollback i guess he only said gitea docker issues | 16:40 |
| fungi | and suddenly the zuul-preview url i've been trying to load for minutes spontaneously popped up in my browser | 16:40 |
| *** janki has quit IRC | 16:41 | |
| fungi | so maybe it was a networking issue on my end | 16:41 |
| *** ramishra has quit IRC | 16:41 | |
| corvus | fungi: well, there is an issue with zuul-preview error handling | 16:41 |
| *** kopecmartin is now known as kopecmartin|off | 16:41 | |
| *** janki has joined #openstack-infra | 16:41 | |
| *** auristor has quit IRC | 16:42 | |
| fungi | ahh, okay. for some reason trying to request http://site.7fe187f62d4a4f169c094fb1ebe4e41c.openstack.zuul-preview.opendev.org/ takes minutes to get a response | 16:43 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-preview master: Return NULL on all errors https://review.openstack.org/641078 | 16:44 |
| corvus | fungi, mordred: ^ i think that will help | 16:44 |
| corvus | it's currently stuck in rewrite loops for /Error/Error/Error etc | 16:45 |
| *** quiquell|off is now known as quiquell | 16:45 | |
| *** helenfm has quit IRC | 16:45 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Override mariadb command to delete tc.log https://review.openstack.org/641079 | 16:45 |
| *** jpena is now known as jpena|brb | 16:46 | |
| mordred | corvus: how's that look? | 16:46 |
| corvus | mordred: sounds plausible :) | 16:47 |
| mordred | clarkb: lgtm | 16:47 |
| mordred | corvus: yay! | 16:47 |
| mordred | corvus: I'm pretty much only looking for plausible today | 16:47 |
| fungi | oof, good catch! | 16:47 |
| clarkb | frickler: we have to set https://review.openstack.org/Documentation/config-gerrit.html#plugins.allowRemoteAdmin to true to allow that reload command to work | 16:47 |
| clarkb | frickler: it appears to be set to false so we'll need a restart to either get that in or to update the replication config :/ | 16:48 |
| *** roman_g has quit IRC | 16:51 | |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Correct provides/requires on buildset-registry job https://review.openstack.org/641083 | 16:53 |
| *** roman_g has joined #openstack-infra | 16:53 | |
| fungi | clarkb: frickler: though it's worth discussing whether we want to turn that on in the needed restart | 16:54 |
| *** psachin has quit IRC | 16:54 | |
| clarkb | fungi: ya the default being false implies that maybe there are reasons to not flip it to true :) | 16:55 |
| *** ccamacho has quit IRC | 16:55 | |
| clarkb | fungi: looks like you can install and uninstall as well as enable/disable plugins remotely in addition to simply reloading an existing plugin | 16:55 |
| fungi | yeah, reading the description there it seems maybe more dangerous, but then again our gerrit admins can already do those things just less seamlessly | 16:56 |
| *** auristor has joined #openstack-infra | 16:56 | |
| fungi | so maybe having it disabled is more a guard against a compromised gerrit admin account or some buggy codepath giving users admin privs | 16:56 |
| fungi | (in our case) | 16:56 |
| *** jamesmcarthur has quit IRC | 16:57 | |
| corvus | clarkb, fungi, mordred: can you review some base-jobs changes required for me to continue testing buildset registries: https://review.openstack.org/640561 and https://review.openstack.org/641083 | 16:58 |
| clarkb | fungi: I'd be happy to leave it disabled for now and stick with the restart we know | 16:58 |
| clarkb | corvus: yes I can take a look | 16:58 |
| corvus | mordred, clarkb, fungi, frickler: i think we're ready to restart gerrit, shall i do that now? | 16:59 |
| fungi | corvus: so the fix for the revert-revert was to remove the promote playbook? | 16:59 |
| *** quiquell is now known as quiquell|off | 17:00 | |
| corvus | fungi: no, sorry, there's a rebase in between those 2 patchsets too | 17:00 |
| *** wolverineav has joined #openstack-infra | 17:00 | |
| fungi | corvus: gerrit restart sounds fine to me though we should give #openstack-release a heads up about replication delays to git.o.o (maybe longer ones than in past restarts even) | 17:00 |
| corvus | fungi: it's splitting the post.yaml playbook into two plays -- it was only running on localhost, but needs to run on the registry host | 17:00 |
| clarkb | corvus: yes, the config has been applied (I just checked) and I'm satisfied there isn't a short cut in the form of a replication config reload as above. Restart would be the next step and I'm happy for someone else to do it while I review those changes and catch up on email/irc | 17:01 |
| clarkb | hosts: all being the important bit in that playbook I guess | 17:02 |
| fungi | right, see it | 17:03 |
| dtantsur | hi folks! is it possible to delete some files from https://tarballs.openstack.org/ironic-python-agent/tinyipa/files/ manually? the files without a branch are not correct | 17:03 |
| fungi | dtantsur: how did they get there? was it due to a problem which has since been corrected? | 17:04 |
| corvus | fungi: -release notified and acked | 17:04 |
| dtantsur | fungi: yep, we had an issue in our post job. it has just been fixed | 17:04 |
| clarkb | corvus: does https://review.openstack.org/#/c/641083/1/zuul.yaml create a loop that can't be satisfied? | 17:04 |
| *** wolverineav has quit IRC | 17:04 | |
| dtantsur | fungi: fix https://review.openstack.org/641007 | 17:05 |
| corvus | clarkb: remember that provides/requires is for sharing data between buildsets. come to think of it "uses" might have been better than "requires". | 17:06 |
| corvus | clarkb: so if there are no dependent changes which provide docker-images, that's okay. | 17:06 |
| clarkb | oh right. I keep thinking within a buildset | 17:06 |
| *** jamesmcarthur has joined #openstack-infra | 17:06 | |
| clarkb | (in my defense the dep trees have always been within a buildset when it comes to the job level :) ) | 17:06 |
| corvus | clarkb: it just says "if there are dependent changes which provider docker-images, then please make sure i have them because i require [can use] them" | 17:06 |
| clarkb | corvus: yup | 17:07 |
| corvus | clarkb: yep, it's not "automatically generate a job graph based on these expressions". we may want to figure out how to convey that, because that's certainly a reasonable assumption to make. | 17:08 |
| corvus | status notice Gerrit is being restarted for a configuration change, it will be briefly offline. | 17:08 |
| corvus | how's that ^? | 17:08 |
| openstackgerrit | Merged openstack-infra/zuul-preview master: Return NULL on all errors https://review.openstack.org/641078 | 17:08 |
| clarkb | corvus: lgtm | 17:08 |
| corvus | #status notice Gerrit is being restarted for a configuration change, it will be briefly offline. | 17:09 |
| openstackstatus | corvus: sending notice | 17:09 |
| *** xek_ has joined #openstack-infra | 17:10 | |
| -openstackstatus- NOTICE: Gerrit is being restarted for a configuration change, it will be briefly offline. | 17:10 | |
| corvus | restarting now | 17:11 |
| *** bhavikdbavishi has quit IRC | 17:12 | |
| *** jpena|brb is now known as jpena | 17:12 | |
| *** xek has quit IRC | 17:12 | |
| openstackstatus | corvus: finished sending notice | 17:12 |
| *** bhavikdbavishi has joined #openstack-infra | 17:12 | |
| fungi | dtantsur: does that look better? | 17:12 |
| *** chandankumar is now known as raukadah | 17:12 | |
| *** armax has quit IRC | 17:13 | |
| corvus | [2019-03-05 17:13:30,742] [main] INFO org.eclipse.jetty.util.log : Logging initialized @102594ms | 17:13 |
| corvus | that's a lot of milliseconds :) | 17:13 |
| *** jamesmcarthur has quit IRC | 17:14 | |
| corvus | ok, gerrit seems to be back up | 17:14 |
| corvus | 37794 tasks | 17:14 |
| openstackgerrit | Merged opendev/base-jobs master: Revert "Revert "Save registry logs"" https://review.openstack.org/640561 | 17:14 |
| fungi | thanks for the restart corvus! | 17:14 |
| *** jistr|sick is now known as jistr | 17:14 | |
| dtantsur | fungi: yes, thank you! | 17:15 |
| fungi | #static log manually removed branchless artifacts from /srv/static/tarballs/ironic-python-agent/tinyipa/files/ on static.o.o | 17:15 |
| fungi | dtantsur: let us know if you need any other help | 17:15 |
| corvus | naturally, we forgot to accept the hostkeys | 17:15 |
| clarkb | corvus: ah of course | 17:15 |
| corvus | fixing that now | 17:15 |
| corvus | er... | 17:17 |
| corvus | does that require another restart | 17:17 |
| corvus | i think it might -- i think i noticed that in my testing yesterday | 17:17 |
| corvus | fungi, clarkb: shall i restart again? | 17:18 |
| clarkb | if that is necessary we may as well take the hit now | 17:18 |
| fungi | yeah, better now than later i guess | 17:18 |
| clarkb | (its cheaper after the first one than waiting for tomorrow) | 17:18 |
| fungi | sorry i didn't think to check that either :/ | 17:18 |
| *** e0ne has quit IRC | 17:18 | |
| corvus | i *did* this in my local testing yesterday and forgot about it today. :( | 17:19 |
| *** markvoelker has joined #openstack-infra | 17:19 | |
| corvus | back up again | 17:19 |
| fungi | we've got a couple commits queued in release-post, not sure if they'll care about a gerrit blip though | 17:20 |
| fungi | oh, and they just vanished anyway | 17:20 |
| corvus | why is it still rejecting the host key? | 17:20 |
| fungi | is the error in the gerrit error.log? | 17:21 |
| corvus | replication | 17:21 |
| corvus | org.eclipse.jgit.errors.TransportException: ssh://git@gitea08.opendev.org:222/openstack-attic/image-api.git: reject HostKey: gitea08.opendev.org | 17:21 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-preview master: Add perf testing framework https://review.openstack.org/636792 | 17:21 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-preview master: Reimplement in Rust https://review.openstack.org/636793 | 17:21 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-preview master: Remove C++ version https://review.openstack.org/636794 | 17:21 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-preview master: Use rust:slim base image https://review.openstack.org/636795 | 17:21 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-preview master: Use slice matching for hostname unpacking https://review.openstack.org/636796 | 17:21 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-preview master: Port in changes from the c++ version https://review.openstack.org/641089 | 17:21 |
| *** ykarel|away has quit IRC | 17:21 | |
| corvus | wrong key type? | 17:22 |
| clarkb | corvus: or port? | 17:22 |
| corvus | looks like it did ecdsa | 17:22 |
| clarkb | did you accept for port 22 not 222? | 17:22 |
| corvus | no i ssh'd to 222 | 17:22 |
| clarkb | k | 17:22 |
| fungi | http://paste.openstack.org/show/747298/ | 17:22 |
| *** yamamoto_ has quit IRC | 17:22 | |
| fungi | i dunno what the "key_load_public: invalid format" is about | 17:23 |
| clarkb | fungi: it means ssh doesn't understand your public key. We may just have the private key data there | 17:23 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Upgrade all dev servers to puppet 4 https://review.openstack.org/630391 | 17:24 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Upgrade some servers to puppet 4 https://review.openstack.org/634726 | 17:24 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config master: Upgrade git01.openstack.org to puppet 4 https://review.openstack.org/634727 | 17:24 |
| *** yamamoto has joined #openstack-infra | 17:24 | |
| fungi | we have a ~gerrit2/.ssh/id_rsa.pub | 17:24 |
| *** yamamoto has quit IRC | 17:24 | |
| clarkb | there goes my idea then | 17:25 |
| fungi | but it doesn't look like an ssh public key to me | 17:25 |
| *** yamamoto has joined #openstack-infra | 17:25 | |
| * clarkb pops out real quick for breakfast | 17:25 | |
| fungi | looks like it's missing the ssh-rsa prefix | 17:25 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Fixed cloud assets config https://review.openstack.org/639792 | 17:26 |
| corvus | so does anyone have handy the command to tell ssh to use the rsa host key? | 17:26 |
| clarkb | corvus: ssh-keyscan -t rsa iirc | 17:26 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Revert "Revert "Update API code to work with Presentation Moderators collection (+N)"" https://review.openstack.org/641091 | 17:26 |
| clarkb | fungi: puppet authorized keys you set the key type in the resource not the actual data. I wonder if we reuse that data value without being part of that authorized keys resources and thus lose the prefix | 17:27 |
| fungi | i think we need to do something like -o HostKeyAlgorithms=ssh-rsa-cert-v01@openssh.com | 17:27 |
| *** jamesmcarthur has joined #openstack-infra | 17:27 | |
| *** rkukura has quit IRC | 17:27 | |
| *** panda|ruck|flu is now known as panda|ruck|off | 17:27 | |
| fungi | ssh -p 222 -o HostKeyAlgorithms=rsa-sha2-512,rsa-sha2-256,ssh-rsa git@gitea08.opendev.org | 17:28 |
| fungi | corvus: ^ | 17:28 |
| corvus | fungi, clarkb: i think i got it with the keyscan method | 17:28 |
| fungi | k | 17:28 |
| corvus | ssh-keyscan -t rsa -p 222 gitea08.opendev.org >> known_hosts | 17:28 |
| corvus | and it looks like a restart is not required | 17:28 |
| fungi | good to know | 17:28 |
| corvus | replication to gitea is happening now | 17:28 |
| *** yamamoto has quit IRC | 17:29 | |
| corvus | so now we could either restart again to restart replication from the top, or leave this running and kick off another full replication to the gitea servers when this is done | 17:29 |
| fungi | so we accepted the ecdsa hostkey instead i guess? | 17:29 |
| fungi | and gerrit has broken ecdsa support | 17:29 |
| corvus | yep | 17:29 |
| corvus | i think i lean toward the second option, even though i think it will invalidate my bet with mordred on the replication time | 17:30 |
| mordred | corvus: I vote for the choice that maximizes my winning chances | 17:31 |
| corvus | looks like content is showing up in https://opendev.org/explore/repos | 17:31 |
| clarkb | gerrit retries indefinitely to replicate | 17:31 |
| clarkb | or at least it did when github rejected nova specs | 17:31 |
| corvus | clarkb: i do see retries at the end of the queue, so we may be covered. | 17:32 |
| corvus | ftr, we're currently at 35380 tasks | 17:32 |
| *** bhavikdbavishi1 has joined #openstack-infra | 17:32 | |
| *** jpich has quit IRC | 17:33 | |
| *** bhavikdbavishi has quit IRC | 17:33 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 17:33 | |
| mordred | https://opendev.org/openstack-infra/puppet-zuul exists | 17:33 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Revert "Revert "Update API code to work with Presentation Moderators collection (+N)"" https://review.openstack.org/641091 | 17:33 |
| *** aojea has quit IRC | 17:34 | |
| *** sdake has quit IRC | 17:35 | |
| pabelanger | mordred: woah! | 17:37 |
| openstackgerrit | Merged openstack-infra/puppet-zuul master: Don't rewrite t/ prefix on global zuul vhost https://review.openstack.org/641077 | 17:39 |
| *** xek_ has quit IRC | 17:42 | |
| *** xek_ has joined #openstack-infra | 17:42 | |
| corvus | fungi: can you +3 https://review.openstack.org/641083 ? | 17:43 |
| corvus | when that lands, i can get back to testing with untrusted projects :) | 17:44 |
| fungi | ahh, yep was just catching up on what clarkb asked about provides/requires | 17:44 |
| corvus | clarkb, mordred: i was just looking at this failure, which happened twice: http://logs.openstack.org/00/640900/1/check/system-config-run-gitea/d38ca02/hosts/bridge.openstack.org/ara-report/result/a306f3b9-1bf6-4255-b62f-2f8ee76972e3/ | 17:45 |
| mordred | corvus: aroo? | 17:47 |
| corvus | and i had a terrible thought -- what if docker uses the mirrors in parallel? so that it tried to pull mariadb from the first mirrer and gitea-web from the second (which failed so it rolled over to upstream dockerhub) | 17:47 |
| corvus | (the earlier change to get the registry logs should help confirm that, but we haven't run with that yet) | 17:48 |
| clarkb | that would be sad | 17:48 |
| corvus | though, in this case, the first mirror should have nothing... so it'd have to be "mariadb: use the first, fail, use the second" followed by "gitea: use the first, fail, skip the second because it's un use, use upstream" | 17:49 |
| *** agopi has joined #openstack-infra | 17:50 | |
| *** rkukura has joined #openstack-infra | 17:51 | |
| *** markvoelker has quit IRC | 17:51 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient master: Show in-progress indicator when opening a board https://review.openstack.org/640593 | 17:52 |
| fungi | a really hacky way to prevent that might be to override hostname lookups | 17:52 |
| *** kjackal has quit IRC | 17:52 | |
| *** rpittau is now known as rpittau|afk | 17:52 | |
| corvus | fungi: i don't think we can do that | 17:52 |
| corvus | the only behavior we can control is the registry itself, we can't impose requirements on the system under test | 17:53 |
| fungi | oh, good point | 17:53 |
| fungi | at least not without also controlling the recursive resolver it uses | 17:53 |
| *** gyee has joined #openstack-infra | 17:54 | |
| corvus | the docker source code is impossible to follow, so i have no idea where this is actually used | 17:55 |
| clarkb | I asked gitea for cinder commit listing via UI. It isn't responding very quickly, but does eventually erspond. Somethnig we may want to look at (possible we just need to use the service so that caches are built up) | 17:56 |
| clarkb | also cinder seems to default ot stable/ocata branch and not master | 17:56 |
| fungi | or may just be busy handling the replication flood still | 17:56 |
| smcginnis | I'm really liking this interface over the cgit one. | 17:57 |
| openstackgerrit | Merged opendev/base-jobs master: Correct provides/requires on buildset-registry job https://review.openstack.org/641083 | 17:57 |
| clarkb | fungi: ya | 17:57 |
| smcginnis | clarkb: There's nothing under openstack/cinder master, so maybe it defaulted to ocata since that's the latest it has? | 17:57 |
| clarkb | smcginnis: oh maybe? possible that this is lag in replication and it doing its best | 17:58 |
| fungi | yeah, i wouldn't read much into any observed behaviors until replication catches up | 17:59 |
| corvus | especially since it's replicating the "c" repos now :) | 17:59 |
| clarkb | infra-root I'm going to add those RO afs volumes to afs01.dfw and afs02.dfw now | 17:59 |
| clarkb | I'll start with project.* then before I do the opensuse addsite I'll disable the mirroring cron for that site via holding the lock | 18:02 |
| *** ginopc has quit IRC | 18:02 | |
| clarkb | I don't want that one to try and replicate via the normal cron process | 18:02 |
| clarkb | The project.* volumes shouldn't be replicated by anything and if they are they are small | 18:02 |
| *** agopi has quit IRC | 18:04 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Update API code to work with Presentation Moderators collection (+N) https://review.openstack.org/641091 | 18:06 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] Zuul Web: add /api/user/actions endpoint https://review.openstack.org/641099 | 18:07 |
| dtantsur | folks, is there something we can urgently do when a third party CI goes a bit crazy like in https://review.openstack.org/#/c/639050/ ? | 18:11 |
| dtantsur | I doubt the contributors are online at this hour.. | 18:12 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: vos release project website volumes https://review.openstack.org/641101 | 18:12 |
| clarkb | infra-root ^ that will ensure that the RO volumes for zuul-ci and docs.starlingx.io are releases regularly. Note that we appear to already use the RO path in our apache docroot config so I don't think I need to update anything there | 18:13 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] Zuul Web: add /api/user/actions endpoint https://review.openstack.org/641099 | 18:13 |
| clarkb | dtantsur: we can disable their account | 18:13 |
| dtantsur | clarkb: lemme try re-upload the patch before we resort to such measures. | 18:14 |
| clarkb | dtantsur: ok let us know if ironic would like us to do that and we'll get it done | 18:14 |
| openstackgerrit | Riju Khatri proposed openstack-infra/storyboard-webclient master: Fix project group view checkboxes https://review.openstack.org/640590 | 18:14 |
| dtantsur | clarkb: OUCH, it became even worse: https://review.openstack.org/#/c/639050/. we have to ask you for help.. | 18:16 |
| clarkb | dtantsur: ok it will be a few minutes to figure out what that user is in the db then we can disable it | 18:17 |
| dtantsur | thanks! | 18:17 |
| clarkb | unless I can do it by full name | 18:18 |
| clarkb | fungi: ^ do you know | 18:18 |
| fungi | i'll look | 18:18 |
| *** e0ne has joined #openstack-infra | 18:19 | |
| fungi | 29800 | 18:19 |
| fungi | i've just set it inactive | 18:20 |
| clarkb | 29800 | 18:20 |
| clarkb | cool you found what I found | 18:20 |
| fungi | yeah, you can't set-account using the full (display) name | 18:21 |
| fungi | tried on a whim | 18:21 |
| dtantsur | looks quiet now, thanks a lot! | 18:21 |
| fungi | you can use e-mail address unless it's erroneously non-unique | 18:22 |
| fungi | that account also had an open connection to the ssh api (3ade947f) which i've closed | 18:23 |
| clarkb | fungi: do you want to send email to the accoutn email addr or should I? | 18:25 |
| fungi | i can do it in a sec | 18:26 |
| fungi | usually we just reach out to the third-part-announce ml | 18:26 |
| fungi | er, third-party-announce | 18:27 |
| *** jamesdenton has quit IRC | 18:27 | |
| clarkb | fungi: ya, though I wonder if at this point anyone pays attention to that | 18:28 |
| fungi | looks like they don't mention any irc contacts on https://wiki.openstack.org/wiki/ThirdPartySystems/Huawei_Ironic_CI | 18:28 |
| clarkb | I've taken the suse lock on mirror-update. Adding site to that volume then releasing it | 18:28 |
| *** jamesdenton has joined #openstack-infra | 18:29 | |
| *** jpena is now known as jpena|off | 18:33 | |
| clarkb | fwiw https://opendev.org/openstack/designate/branches/ shows default branch as stable/ocata | 18:34 |
| *** xek_ has quit IRC | 18:34 | |
| clarkb | so that might be a cleanup we have to do? | 18:34 |
| *** xek has joined #openstack-infra | 18:35 | |
| *** mriedem has quit IRC | 18:37 | |
| fungi | i wonder how it decides that | 18:38 |
| *** wolverineav has joined #openstack-infra | 18:38 | |
| *** mriedem has joined #openstack-infra | 18:39 | |
| clarkb | git push doesn't replicate HEAD | 18:40 |
| clarkb | possible it infers it through first pushed branch and that is the order gerrit pushes in? | 18:40 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] Zuul Web: add /api/user/actions endpoint https://review.openstack.org/641099 | 18:44 |
| clarkb | https://github.com/go-gitea/gitea/pull/3715 I think that may be what is happening | 18:45 |
| *** IvensZambrano has quit IRC | 18:46 | |
| clarkb | https://github.com/go-gitea/gitea/issues/3913 an as of yet unfixed bug saying the behavior added in ^ is problematic for our git push --mirror use case | 18:46 |
| *** markvoelker has joined #openstack-infra | 18:48 | |
| clarkb | heh the swagger api docs don't metion repo settings url. I guess this i the thing that was mentioned before about how they don't have all the endpoints exposed there | 18:50 |
| openstackgerrit | Riju Khatri proposed openstack-infra/storyboard-webclient master: Show in-progress indicator when opening a board https://review.openstack.org/640593 | 18:50 |
| *** gfidente is now known as gfidente|afk | 18:51 | |
| corvus | we could do a one-time fix for all the repos easily enough | 18:52 |
| corvus | but if we can't update the settings of all the projects after creation, then every new project (that has more than one branch) is likely to end up with the wrong branch | 18:52 |
| *** jamesmcarthur has quit IRC | 18:52 | |
| clarkb | corvus: I wonder if we can git push the null commit to the project as part of project creation to make gitea think that master is already there | 18:55 |
| *** yamamoto has joined #openstack-infra | 18:55 | |
| clarkb | in theory you should always be able to fast forward from the null commit right? | 18:55 |
| clarkb | actually it doesn't matter bceause gerrit will force push and overwrite | 18:55 |
| clarkb | so that might be the hack null commit or not | 18:55 |
| clarkb | assuming that setting it explicitly as a setting does get overwritten on first push | 18:57 |
| clarkb | if not then setting it as a setting is likely better | 18:57 |
| clarkb | Reminder: Our team meeting starts in ~3 minutes over in #openstack-meeting | 18:57 |
| corvus | mordred: can you look into that ^ ? | 18:58 |
| corvus | mordred: setting the default branch explicitly on creation, to see if that fixes the default branch problem | 18:58 |
| mordred | corvus: I can look in to that | 19:01 |
| mordred | corvus: is there a "good" way for me to log in to one of our giteas - just grab the root password from one of the configs? | 19:02 |
| mordred | corvus: (I tried using try.gitea the other day to poke at default branch, but what I found there did not work on our gitea - so I'd like to poke at ours) | 19:02 |
| *** electrofelix has quit IRC | 19:02 | |
| corvus | mordred: i think that's it | 19:02 |
| mordred | kk | 19:02 |
| *** yamamoto has quit IRC | 19:07 | |
| *** e0ne has quit IRC | 19:17 | |
| *** jamesmcarthur has joined #openstack-infra | 19:21 | |
| *** markvoelker has quit IRC | 19:21 | |
| dtantsur | clarkb, fungi, have you reached out to huawei folks re that problem or should I? | 19:22 |
| clarkb | dtantsur: fungi said he would email our third-party-announce list (I've not checked if that was done yet as I'm running the infra meeting now) | 19:22 |
| SotK | mordred, diablo_rojo: I'm around to chat about formsubmit swift stuff for a few hours after the meeting btw | 19:22 |
| dtantsur | I guess it won't hurt if I send them another one | 19:24 |
| *** jamesmcarthur_ has joined #openstack-infra | 19:24 | |
| *** jamesmcarthur has quit IRC | 19:25 | |
| *** armax has joined #openstack-infra | 19:26 | |
| fungi | dtantsur: clarkb: i haven't yet, no | 19:26 |
| *** dtantsur is now known as dtantsur|afk | 19:31 | |
| *** sdake has joined #openstack-infra | 19:33 | |
| *** e0ne has joined #openstack-infra | 19:37 | |
| *** kjackal has joined #openstack-infra | 19:37 | |
| *** wolverineav has quit IRC | 19:41 | |
| *** wolverineav has joined #openstack-infra | 19:42 | |
| *** wolverineav has quit IRC | 19:43 | |
| *** wolverineav has joined #openstack-infra | 19:43 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Revert "Revert "Set default branch for repo"" https://review.openstack.org/641108 | 19:44 |
| *** jento has joined #openstack-infra | 19:45 | |
| *** wolverineav has quit IRC | 19:45 | |
| *** wolverineav has joined #openstack-infra | 19:46 | |
| *** wolverineav has quit IRC | 19:46 | |
| *** wolverineav has joined #openstack-infra | 19:46 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Use auto_init to create a repo with a master branch https://review.openstack.org/641110 | 19:46 |
| mordred | clarkb, corvus: ^^ there's two options | 19:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove releases link from gitea https://review.openstack.org/641111 | 19:51 |
| *** dave-mccowan has quit IRC | 19:55 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Revert "Revert "Set default branch for repo"" https://review.openstack.org/641108 | 19:55 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Use auto_init to create a repo with a master branch https://review.openstack.org/641110 | 19:55 |
| clarkb | mordred: thanks I'll review that now then I need to find lunch | 19:59 |
| *** dave-mccowan has joined #openstack-infra | 20:01 | |
| clarkb | mordred: with auto_init do we still need to set the branch to master or is master implied in that case? | 20:02 |
| frickler | clarkb: mordred: maybe we're not there yet, but how about setting up a gitea-dev instance where we could test such things? also for the issues/releases tabs | 20:02 |
| corvus | frickler: we could also write a test job that exercises project creation | 20:03 |
| *** xek has quit IRC | 20:03 | |
| *** xek has joined #openstack-infra | 20:03 | |
| clarkb | a setup to run a gitea docker compose locally might be better than long running dev instance? | 20:03 |
| corvus | i say "also", but i think i may actually mean "instead" | 20:03 |
| clarkb | In theory the use of docker images makes that easier than if we had to bootstrap from scratch | 20:03 |
| clarkb | corvus: ++ | 20:04 |
| corvus | yeah. i think 'run gitea locally in docker' to develop things, and 'run zuul jobs to test changes' is a good recipe | 20:04 |
| corvus | we just don't have test coverage of project creation yet, only bootstrapping the service. | 20:05 |
| mordred | corvus: yeah - seems like running the project creation role would give us some good coverage there | 20:05 |
| frickler | hmm, yeah, testing locally should work | 20:05 |
| mordred | because the original 'set branch to master' threw a 404 and ansible was sad - so gate test would have totally shown an error | 20:06 |
| *** dpawlik has joined #openstack-infra | 20:09 | |
| clarkb | infra-root can I get a quick review on https://review.openstack.org/#/c/641101/ as part of my afs prep work? | 20:11 |
| *** eharney has quit IRC | 20:12 | |
| *** dpawlik has quit IRC | 20:16 | |
| *** sdake has quit IRC | 20:17 | |
| *** sdake has joined #openstack-infra | 20:19 | |
| *** markvoelker has joined #openstack-infra | 20:19 | |
| clarkb | https://review.openstack.org/#/c/641077/1 merged to fix the zuul ui redirect thing but hasn't applied on zuul01 yet. Looking at puppet modules on zuul01 the module hasn't updated to that version but pupet has run several times since merging | 20:20 |
| clarkb | I'll have to track that down after lucnh if we don't already know why that is happening. Did we stop running the update repos playbook? | 20:20 |
| *** whoami-rajat has quit IRC | 20:22 | |
| *** bhavikdbavishi has quit IRC | 20:23 | |
| *** wolverineav has quit IRC | 20:31 | |
| *** wolverineav has joined #openstack-infra | 20:32 | |
| *** wolverineav has quit IRC | 20:37 | |
| *** markvoelker has quit IRC | 20:52 | |
| corvus | clarkb, mordred: oh, whew -- i *think* the docker registry mirror error from earlier may be due to leftover v1 manifests -- i'm digging into that theory now | 20:54 |
| mordred | corvus: kk | 20:56 |
| corvus | clarkb, mordred: yes, the last gitea image changes merged ~50 minutes before the v1/v2 manifest fix merged, so they're still v1 images. the next gitea image updates should fix that. so i'll go review the change to remove the relases tab :) | 20:57 |
| mordred | corvus: woot! | 20:57 |
| *** jamesmcarthur has joined #openstack-infra | 20:57 | |
| *** jamesmcarthur_ has quit IRC | 21:00 | |
| *** wolverineav has joined #openstack-infra | 21:02 | |
| *** eharney has joined #openstack-infra | 21:03 | |
| priteau | The Blazar gate was failing around 7 PM UTC because it is pulling keystoneauth1 3.13.0 which has a bug, even though 3.13.1 was released earlier today. Could this be caused by an out of date PyPI mirror? | 21:05 |
| priteau | Nevermind, it may be due to upper-constraints. | 21:06 |
| ttx | clarkb: https://review.openstack.org/#/c/641077/ did fix https://storyboard.openstack.org/#!/story/2004987 | 21:07 |
| clarkb | ttx: cool I was just about to look at why that wasn't applying but it appears that it just applied | 21:08 |
| priteau | ^^^ keystoneauth1 bump here: https://review.openstack.org/#/c/641036/1 | 21:08 |
| clarkb | priteau: we usually defer to requirements core on those but if smcginnis dhellmann prometheanfire or tonyb don't get to it soon I suppose we can approve it | 21:09 |
| prometheanfire | hi? | 21:09 |
| smcginnis | Need to wait for the zuul jobs to complete yet. | 21:10 |
| priteau | I've just resubmitted a blazar patch with a Depends-On on it to check that it fixes the problem | 21:10 |
| prometheanfire | clarkb: priteau asking because of the release timeline? | 21:10 |
| smcginnis | priteau: I don't believe depends-on will work in this situation. | 21:10 |
| clarkb | smcginnis: prometheanfire: these things are why I will defer to the requirements team generally :) | 21:10 |
| *** pcaruana has quit IRC | 21:10 | |
| clarkb | (I don't know what all the criteria are) | 21:10 |
| * prometheanfire thinks non-client libs are frozen | 21:11 | |
| priteau | prometheanfire: Asking because 3.13.0 has broken our gate jobs | 21:11 |
| *** wolverineav has quit IRC | 21:11 | |
| * prometheanfire has been distracted | 21:11 | |
| prometheanfire | priteau: it's still early enough that it's probably fine | 21:11 |
| tonyb | priteau: Looks pretty straight forward just waiting on zuul | 21:11 |
| clarkb | prometheanfire: understood, no worries. I'm sure smcginnis or dhellmann will be able to handle it | 21:11 |
| clarkb | oh and not tony :) | 21:11 |
| clarkb | *now | 21:11 |
| fungi | depends-on does usually handle proposed constraints changes for openstack's tox jobs | 21:11 |
| tonyb | fungi: It certainly should :) | 21:12 |
| smcginnis | Wasn't sure if the constraints line in tox.ini would interfere with that. | 21:12 |
| fungi | the role we use grabs the constraints file from zuul's checkout of the requirements repo, but yeah if the project does something funky in tox.ini to break that then ymmv | 21:12 |
| clarkb | infra-root mirror.opensuse replication to afs02.dfw completed and I've removed my hold of that lock on mirror-update.o.o | 21:13 |
| *** e0ne has quit IRC | 21:13 | |
| smcginnis | Hmm, I don't see 641036 in zuul.o.o | 21:13 |
| tonyb | smcginnis: what fungi said ;P | 21:13 |
| clarkb | mordred: is the and in your release template postfix? | 21:14 |
| priteau | Thanks everyone, hopefully zuul won't take too long | 21:14 |
| clarkb | smcginnis: possile that it tried to report back when we restarted gerrit today | 21:15 |
| clarkb | in that case may need a recheck | 21:15 |
| gouthamr | hello o/ I have a patch stuck in "Ready to Submit", https://review.openstack.org/#/c/638075/ | 21:19 |
| gouthamr | haven't seen that issue before, something wrong with the vote from zuul? ^ | 21:20 |
| clarkb | gouthamr: the issue is the parent commit for that change is not the latest patchset on the parent change. Gerrit only allows the latest patchset to merge so a change depending on not the latest patchset cannot merge. The fix is to rebase | 21:22 |
| clarkb | gouthamr: Gerrit tries to indicate this to you via the orange dot on the Parent(s) entry on the change page | 21:22 |
| dmsimard | clarkb, gouthamr: the parent change actually hasn't merged yet | 21:22 |
| dmsimard | oh, yeah, it has | 21:23 |
| dmsimard | the outdated patchset hasn't merged, the updated one has :p | 21:23 |
| clarkb | dmsimard: ya | 21:23 |
| gouthamr | clarkb: ah! since zuul always tests by rebasing, assumed this merge would work | 21:24 |
| gouthamr | clarkb dmsimard: thanks for the clarification, rebased now | 21:24 |
| clarkb | gouthamr: note this is entirely a gerrit thing not a zuul thing | 21:25 |
| clarkb | we could potentially work around it if zuul pushed commits into the repos nstead of gerrit | 21:25 |
| *** eernst has joined #openstack-infra | 21:25 | |
| clarkb | fungi: can you sanity check https://review.openstack.org/#/c/630391/7/inventory/groups.yaml for wiki-dev? want to make sure that won't step on any toes | 21:25 |
| *** jcoufal has quit IRC | 21:25 | |
| clarkb | fungi: if that doesn't conflict with your work there I can approve | 21:26 |
| *** e0ne has joined #openstack-infra | 21:26 | |
| *** kjackal has quit IRC | 21:27 | |
| fungi | clarkb: it'll be fine | 21:28 |
| clarkb | fungi: tahnks for checking. | 21:28 |
| fungi | i've booted wiki-dev02 but still need to propose it to the inventory | 21:28 |
| fungi | so it hasn't puppeted yet | 21:28 |
| clarkb | fwiw lists-dev and groups-dev won't match there because they don't have digited servers yet | 21:29 |
| *** wolverineav has joined #openstack-infra | 21:29 | |
| *** janki has quit IRC | 21:29 | |
| clarkb | I'll scribble a note to follow up on that | 21:29 |
| *** e0ne has quit IRC | 21:30 | |
| clarkb | cmurphy: ^ fyi on that | 21:31 |
| *** kgiusti has quit IRC | 21:36 | |
| clarkb | I'm actually curious how https://review.openstack.org/#/c/630391/7 manages to merge given that we fixed groups and groups-dev entries in futureparser to drop the digits | 21:38 |
| clarkb | I'll have to check the resulting git log and push a fixup | 21:38 |
| *** eernst has quit IRC | 21:40 | |
| *** eernst_ has joined #openstack-infra | 21:40 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Manage user ssh keys from urls https://review.openstack.org/604932 | 21:44 |
| clarkb | fungi: ^ opendev.org urls. I'm researching the CAA RRs now | 21:45 |
| *** eernst_ has quit IRC | 21:46 | |
| *** xek has quit IRC | 21:46 | |
| *** wolverineav has quit IRC | 21:47 | |
| *** wolverineav has joined #openstack-infra | 21:48 | |
| clarkb | fungi: I'm not finding anything in requests/urllib3/pyOpenSSL docs about checking those records | 21:49 |
| *** markvoelker has joined #openstack-infra | 21:49 | |
| clarkb | this may be something we have to test | 21:49 |
| fungi | clarkb: thanks, and yeah it may be they don't support caa checking yet | 21:49 |
| *** wolverineav has quit IRC | 21:53 | |
| ianw | clarkb: who would be responsible for transferring the signing domain acme-opendev.org to openstack/opendev? | 21:54 |
| ianw | i'm thinking right now getting graphite.opendev.org as the initial letsencrypt target | 21:54 |
| ianw | but only with staging keys, until ready for full production | 21:54 |
| openstackgerrit | Merged openstack-infra/system-config master: vos release project website volumes https://review.openstack.org/641101 | 21:54 |
| openstackgerrit | Merged openstack-infra/system-config master: Remove releases link from gitea https://review.openstack.org/641111 | 21:54 |
| ianw | (i just mean getting the keys on the host ... wiring them up to apache will be a little separate puppet exercise) | 21:55 |
| clarkb | ianw: the foundation uses a registrar manager service thing. jamesmcarthur is our interface with them typically. | 21:55 |
| clarkb | ianw: does one of us already own that domain or do we also need to purchase it? | 21:56 |
| clarkb | (I can work with jamesmcarthur to get that going) | 21:56 |
| clarkb | let me ping him to look here | 21:56 |
| jamesmcarthur | hello | 21:56 |
| clarkb | ohai | 21:56 |
| *** eernst has joined #openstack-infra | 21:56 | |
| clarkb | jamesmcarthur: so one of the things we'd like to simplify letsencrypt cert issuance authorization is a separate dns domain we can delegate to for that | 21:57 |
| clarkb | something like acme-opendev.org | 21:57 |
| jamesmcarthur | sure | 21:57 |
| clarkb | ianw has been driving a lot ofthis work so can probably articulate why this is useful better than I can but basically allows us to have a zone that doesn't get changes made to it reviewed and instead we automate that for cert issuance | 21:58 |
| clarkb | I'm guessing we want to manage that domain in the same way that we do the main one | 21:58 |
| jamesmcarthur | Does someone already own acme-opendev.org? | 21:58 |
| *** eernst has quit IRC | 21:58 | |
| fungi | ianw: any reason we can't just delegate from openstack.org to a subdomain-rooted zone? | 21:58 |
| clarkb | jamesmcarthur: fungi those are both questions I'll have to defer to ianw on | 21:58 |
| openstackgerrit | Merged openstack-infra/system-config master: Upgrade all dev servers to puppet 4 https://review.openstack.org/630391 | 21:58 |
| fungi | er, i mean from opendev.org | 21:58 |
| jamesmcarthur | I was about to say :) | 21:58 |
| ianw | fungi: yeah i did mention it in https://specs.openstack.org/openstack-infra/infra-specs/specs/letsencrypt.html#opendev | 21:59 |
| jamesmcarthur | acme.opendev.org would already be under your control | 21:59 |
| fungi | couldn't we just put acme.opendev.org as the root of a separate zone not managed through code review and cname into there? | 21:59 |
| ianw | we *can*, personally i think it's better to use a separate domain | 21:59 |
| fungi | i'll reread the spec | 21:59 |
| fungi | i think i missed the explanation for why a separate domain improves it, rather than just needing to be a separate zonefile | 21:59 |
| ianw | for example, if we get dns-as-a-service, we can point the signing domain at it, and use the acme client to automatically update without any interference to opendev.org | 22:00 |
| fungi | we can still do exactly that with a subdomain, and even more easily | 22:00 |
| fungi | since we don't have to interface with a registrar to update the delegation | 22:00 |
| *** priteau has quit IRC | 22:01 | |
| fungi | there's nothing special about how an acme.opendev.org zone would be served than an acme-opendev.org zone | 22:01 |
| clarkb | right we can set glue records in our gerrit reviewed zone then delegate that to whereever? | 22:02 |
| fungi | no need for glue records, we just set ns records in opendev.org for the acme.opendev.org zone to the different nameservers | 22:02 |
| ianw | fungi: i'm not sure how that would work, isn't a dns-as-a-service going to want to own the opendev.org domain? | 22:03 |
| fungi | why would it? | 22:03 |
| fungi | tell the dns-as-a-service it owns acme.opendev.org | 22:03 |
| fungi | we delegate to it with ns records for acme in the opendev.org zone | 22:04 |
| clarkb | fungi: oh right they don't have to be in the same domain | 22:04 |
| clarkb | fungi: we'd also have to set up dnssec? | 22:05 |
| clarkb | but that should all be doable | 22:05 |
| jamesmcarthur | Yeah, I would stick with a sub-domain here if possible. | 22:05 |
| dmsimard | Eh "Introducing Kraken, an Open Source Peer-to-Peer Docker Registry": https://eng.uber.com/introducing-kraken/ | 22:07 |
| ianw | hrm, personally i still like the separation of powers of two domains, let me think some more | 22:08 |
| fungi | ianw: it makes sense to have two domains if you want two completely different sets of people/software controlling them. if you trust and want the folks who control the opendev.org domain to also have control over acme.opendev.org then that works fine. systems you give access to update acme.opendev.org should have no way to break anything else in opendev.org outside that subdomain | 22:10 |
| fungi | if the concern was that folks controlling opendev.org should have no control over acme-opendev.org that's when a separate domain would also provide a separation of powers | 22:11 |
| *** wolverineav has joined #openstack-infra | 22:12 | |
| *** eharney has quit IRC | 22:14 | |
| ianw | fungi: ok, let me see if i can come up with something that seems like it will roll out a valid configuration using a subdomain | 22:14 |
| fungi | it ought to look identical other than containing an additional . right? | 22:22 |
| *** markvoelker has quit IRC | 22:22 | |
| *** sdake has quit IRC | 22:28 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Update query for bug 1818560 https://review.openstack.org/641147 | 22:28 |
| openstack | bug 1818560 in OpenStack Compute (nova) "Nova test_report_client uses nova conf when starting placement intercept, causing missing config opts" [Critical,Fix released] https://launchpad.net/bugs/1818560 - Assigned to Eric Fried (efried) | 22:28 |
| *** sdake has joined #openstack-infra | 22:32 | |
| *** sdake has quit IRC | 22:32 | |
| clarkb | I'm keeping an eye for puppet 4 install/runs on review-dev | 22:35 |
| clarkb | I think the next run in a few minutes will be with puppet4 | 22:36 |
| *** sdake has joined #openstack-infra | 22:37 | |
| clarkb | http://paste.openstack.org/show/747305/ puppet4 on review dev | 22:39 |
| clarkb | the unresolved dependencies for those puppet modules are something we should look into as is why did rc.d change for gerrit | 22:39 |
| clarkb | I'm going to check the other servers now since ^ shouldn't represent any emergencies | 22:39 |
| *** jamesmcarthur has quit IRC | 22:41 | |
| clarkb | http://paste.openstack.org/show/747306/ translate-dev had a thing chagne that I wouldn't have expected. Storyboard-dev is cleaner, http://paste.openstack.org/show/747307/. Both don't know what ::project_config_ref is | 22:43 |
| clarkb | since ::project_config_ref is common to all three I'll start there | 22:43 |
| clarkb | cmurphy: ^ fyi there are things we should investigate and possibly clean up | 22:43 |
| *** gfidente|afk has quit IRC | 22:44 | |
| *** sdake has quit IRC | 22:45 | |
| *** wolverineav has quit IRC | 22:47 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-project_config master: Check if project_config_ref is defined before using it https://review.openstack.org/641152 | 22:51 |
| clarkb | fix 1: | 22:51 |
| *** sdake has joined #openstack-infra | 22:51 | |
| *** munimeha1 has quit IRC | 22:52 | |
| *** rlandy is now known as rlandy|bbl | 22:54 | |
| *** tkajinam has joined #openstack-infra | 22:54 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zone-opendev.org master: Add acme.opendev.org https://review.openstack.org/641154 | 22:55 |
| *** jamesmcarthur has joined #openstack-infra | 22:57 | |
| *** jamesmcarthur has quit IRC | 22:58 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add unmanaged domains, add acme.opendev.org https://review.openstack.org/641155 | 22:58 |
| *** jamesmcarthur has joined #openstack-infra | 22:58 | |
| *** sdake has quit IRC | 23:02 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Install latest puppet-vcsrepo https://review.openstack.org/641156 | 23:02 |
| clarkb | fix 2: | 23:02 |
| fungi | "This message is to inform you that the host your cloud server 'graphite.openstack.org' resides on alerted our monitoring systems at 23:02 UTC." | 23:03 |
| fungi | ianw: ^ now there's irony for you | 23:03 |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Update query for bug 1818560 https://review.openstack.org/641147 | 23:04 |
| openstack | bug 1818560 in OpenStack Compute (nova) "Nova test_report_client uses nova conf when starting placement intercept, causing missing config opts" [Critical,Fix released] https://launchpad.net/bugs/1818560 - Assigned to Eric Fried (efried) | 23:04 |
| *** yamamoto has joined #openstack-infra | 23:05 | |
| ianw | fungi: hrm, if it's messed it up in any way we can rm it | 23:08 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Bump puppet/staging to 1.0.1 to satisfy mysql module dep https://review.openstack.org/641157 | 23:08 |
| clarkb | fix 3 | 23:08 |
| ianw | all it's currently doing is rejecting random stats packets sent at it from externals :) | 23:08 |
| jamesmcarthur | sorry ianw and fungi: I dropped off for a few. If y'all need assistance in the domain reg department, just hit me up. | 23:09 |
| *** yamamoto has quit IRC | 23:10 | |
| ianw | jamesmcarthur: ok, thanks, will proceed with subdomain for now, will ping if anything changes | 23:11 |
| jamesmcarthur | roger! | 23:12 |
| fungi | thanks jamesmcarthur! | 23:13 |
| clarkb | puppet-gerrit actually does say to create those rc.* files | 23:14 |
| clarkb | so its fine that it did, but I'm not sure why they weren't already there | 23:14 |
| *** hwoarang has quit IRC | 23:14 | |
| clarkb | probably because S04gerrit is there instaed? | 23:14 |
| clarkb | on prod gerrit we have S04gerrit and S90gerrit with puppet3 | 23:16 |
| clarkb | with review-dev we have only S04gerrit with puppet4. I wonder if puppet is cleaning the S90 values up | 23:16 |
| clarkb | because S04 supercedes | 23:16 |
| fungi | otherwise it could double-start i suppose | 23:17 |
| clarkb | I think the fix here is going to be dropping the explicit symlinks in puppet-gerrit and letting the service ersources manage that | 23:17 |
| clarkb | Looking at the wildfly change now | 23:17 |
| *** hwoarang has joined #openstack-infra | 23:18 | |
| *** markvoelker has joined #openstack-infra | 23:19 | |
| *** wolverineav has joined #openstack-infra | 23:24 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Test gitea project creation playbook https://review.openstack.org/641158 | 23:25 |
| corvus | mordred: ^ | 23:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Test gitea project creation playbook https://review.openstack.org/641158 | 23:26 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Test gitea project creation playbook https://review.openstack.org/641158 | 23:27 |
| mordred | corvus: \o/ | 23:28 |
| ianw | fungi: hrm, testing Add acme.opendev.org https://review.openstack.org/641154 is going to be a pain. the roles clone from git repos, but don't know to use the zuul checkout | 23:29 |
| *** wolverineav has quit IRC | 23:30 | |
| ianw | s/the roles/the dns roles/ | 23:30 |
| clarkb | ok the translate change is interesting. We are writing a config with username and password for wildfly. This change seems to have either added a user or modified its password? I'm looking at the file bucket now to see if I can tell | 23:30 |
| corvus | ianw: i would think that you just don't want an acme zone file in that repo | 23:31 |
| corvus | ianw: i think you'd just want to have le-ansible write that file, and, once that exists on the actual production server, add the NS records to the zone | 23:32 |
| *** ekultails has quit IRC | 23:32 | |
| corvus | ianw: (otherwise, the system-config dns roles are going to fight with le-ansible over that file) | 23:32 |
| ianw | corvus: my thinking is that /etc/named.conf needs to have a reference to at least a empty template to be valid | 23:33 |
| *** sdake has joined #openstack-infra | 23:33 | |
| clarkb | file_line doesn't seem to use the clientbucket? | 23:33 |
| ianw | corvus: in terms of fighting, the next change marks it as "unmanaged" | 23:33 |
| ianw | ^ Add unmanaged domains, add acme.opendev.org https://review.openstack.org/641155 | 23:33 |
| corvus | ianw: well that changes things :) | 23:34 |
| clarkb | well translate-dev01 is stable now and that config setting should be fine to update so I think I'll just ignore that one. | 23:34 |
| clarkb | Now to fix puppet-gerrit | 23:34 |
| clarkb | oh hrm we don't actually have a gerrit service resource? then what is deleting S90gerrit | 23:35 |
| *** derekh has quit IRC | 23:36 | |
| *** jamesmcarthur has quit IRC | 23:37 | |
| corvus | ianw: that seems like a reasonable implementation. i'm not sure it's going to be worth it to build full integration testing of proposed data changes. however, i think it's safe to land the zone addition in 641154 but without the NS records. perhaps that's enough to satisfy the tests? | 23:37 |
| clarkb | ok best guess is it is actually the puppet install that removes S90gerrit since I think we install puppet4 each time through | 23:37 |
| clarkb | and that might trigger something in systemd that does a cleanup of redundant init things. Now my question is where does S04gerrit come from? | 23:38 |
| *** wolverineav has joined #openstack-infra | 23:38 | |
| *** jamesmcarthur has joined #openstack-infra | 23:38 | |
| clarkb | my guess for that is systemd init.d support | 23:38 |
| clarkb | if you put an init script in /etc/init.d/ does systemd daemon-reload automatically enable that? | 23:38 |
| clarkb | fungi: ianw ^ you may know? | 23:38 |
| fungi | i don't really know the first thing about systemd | 23:39 |
| ianw | hrm, my instinct says no but ... yeah ... | 23:39 |
| *** jamesmcarthur has quit IRC | 23:40 | |
| ianw | corvus: yeah, i think having that file there would at least allow testinfra to check bind started (synatx ok) and that files are deployed | 23:41 |
| ianw | i might be able to do a run using the review git URLs somehow, i'll try that first | 23:42 |
| corvus | clarkb, fungi, mordred: good news everyone! also some bad news | 23:43 |
| fungi | if futurama is to be believed, "good news everyone" is almost always followed by bad news | 23:44 |
| corvus | the good news is that the buildset registry is working (now that the v2 manifests are in place on the gitea images, the apparent registry problem from earlier is gone, so that hypothesis was correct) | 23:44 |
| corvus | the other good news is that the latest runs of a series of changes designed to show that the system is working with dependent changes also seem to be working, in that the sha of the image deployed in change #2 is the same sha previously built in change #1 | 23:45 |
| corvus | the bad news is that in order to *really* show it was working, i also tried to break the image and deploy it, but apparently our testinfra tests of zuul-preview weren't able to detect that breakage, so they passed | 23:46 |
| openstackgerrit | Clark Boylan proposed openstack-infra/openstack-zuul-jobs master: Add puppet-vcsrepo as required project on puppet jobs https://review.openstack.org/641161 | 23:46 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Install latest puppet-vcsrepo https://review.openstack.org/641156 | 23:46 |
| clarkb | corvus: progress at least | 23:47 |
| mordred | corvus: oh. well - that's some lovely good news | 23:47 |
| corvus | it looks like if you tell docker to bind a port, it actually listens on that port, even if the container isn't | 23:48 |
| clarkb | infra-root cmurphy https://review.openstack.org/#/q/owner:cboylan+topic:puppet-4+status:open is a set of changes to clean up the puppet-4 stuff I noticed | 23:48 |
| clarkb | cmurphy: I think we want to cleanup all of the puppet module dependencies before deploying more puppet-4 | 23:48 |
| clarkb | cmurphy: once those initial dep fixes go in I'll generate the tree and we can go through it and fix the rest | 23:49 |
| clarkb | as for Gerrit I think we should be cautious for now and just let it create the symlinks that will be cleaned up by something | 23:49 |
| clarkb | and when we understand that better fix it properly | 23:49 |
| clarkb | also I'm beginning to think we should puppet-4 things in batches like that so we can monitor and address unexpected behaviors | 23:50 |
| *** markvoelker has quit IRC | 23:52 | |
| clarkb | I'm going to do a systemctl daemon-reload once puppet puts S90gerrit back and see if that cleans up S90gerrit in favor os S04gerrit | 23:54 |
| corvus | clarkb, mordred: is it perhaps possible that EXPOSE is meaningless in host networking? | 23:55 |
| clarkb | corvus: oh that could be it | 23:56 |
| corvus | okay, so i probably didn't break zuul-preview hard enough :) | 23:56 |
| mordred | corvus: entirely likely | 23:56 |
| fungi | try harder! | 23:56 |
| clarkb | corvus: the process running in the host network namespace is gonna bind to what it binds to regardless of docker network magic | 23:56 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-preview master: DNM: break dockerfile https://review.openstack.org/640901 | 23:57 |
| corvus | clarkb, mordred, fungi: ^ think that'll break enough? | 23:57 |
| clarkb | corvus: that should break when the container is started but not at build time I would expect | 23:57 |
| clarkb | so good enough if you want runtime breakage | 23:58 |
| corvus | yep, that's what i'm looking for | 23:58 |
| corvus | working build, failing deploy | 23:58 |
| clarkb | then ya that should break as expected (at least based on experiences trying to exec bash in an image with only sh) | 23:58 |
| *** wolverineav has quit IRC | 23:59 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-preview master: DNM: fix dockerfile https://review.openstack.org/640904 | 23:59 |
| *** wolverineav has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!