| clarkb | ianw: any idea why this task shows changed http://paste.openstack.org/show/729010/ ? | 00:01 |
|---|---|---|
| clarkb | doens't do that for the centos nodes fwiw | 00:01 |
| ianw | ooohhh, that uses apt: with a url | 00:01 |
| ianw | i wonder if apt doesn't keep track of that | 00:01 |
| clarkb | oh it may just set it each time rather than checking | 00:02 |
| ianw | oh that sucks | 00:02 |
| mnaser | (it shouldn't) | 00:02 |
| clarkb | looking at the logs I've realized I may need to clean up system-config on the new mirror as soon as the stop cloning system-config change merges | 00:03 |
| clarkb | I don't think this is urgent either just a small cleanup since it didn't fail on the sjc1 mirror like the other nodes | 00:04 |
| clarkb | overall though seems to be working | 00:04 |
| ianw | the apt module page explicitly gives an example of using a deb: https://... line -- i had assumed it would be idempotent | 00:04 |
| ianw | i guess it can't really know. unless we do something like cache the repo .deb file and then stat it or something ... urgh | 00:06 |
| clarkb | ianw: can we check if the deb is installed already? | 00:07 |
| openstackgerrit | Merged openstack-infra/project-config master: Start booting instances at vexxhost-sjc1 https://review.openstack.org/596830 | 00:08 |
| fungi | apt expects you to configure entries for package repositories in your sources list (with optional pinning if you want to make sure you only allow specific packages from them). it provides its own caching and freshness checking | 00:08 |
| ianw | fungi: the apt.puppetlabs.com repo .debs are probably just a convenience? we could certainly move to manual repo installation (see: initial code was cargo-cult import of install_puppet.sh) | 00:10 |
| clarkb | wow is it already 00:10UTC? time flies when you are fixing puppet! | 00:10 |
| mnaser | hah | 00:12 |
| clarkb | I'm going to have to step out and parent for a bit now. But this is good progress. I think we are close to being able to launch arbitrary xenial nodes and have them be puppeted now (there are a few oustanding fixes to launch node that I wrote iirc) | 00:15 |
| clarkb | ianw: thanks for the help on the puppet installation piece | 00:15 |
| * mnaser waits for new vms to go up | 00:15 | |
| fungi | and i've now received another 28 e-mails from/about the zk servers | 00:15 |
| fungi | i wonder if it's going to do this every ansible pulse from now on | 00:16 |
| *** ansmith has quit IRC | 00:16 | |
| clarkb | fungi: yes, we may want to remove the become: yes entries from install-puppet | 00:16 |
| *** dingyichen has joined #openstack-infra | 00:16 | |
| clarkb | fungi: https://review.openstack.org/#/c/597308/2 removes one of them | 00:16 |
| fungi | thanks! | 00:16 |
| clarkb | (I'm not in a great spot to push that change up right now) | 00:16 |
| fungi | and looks like it's already approved | 00:17 |
| clarkb | fungi: I htink there are others fwiw | 00:17 |
| ianw | fungi: sorry i've missed this discussion. what's up with become: yes ? | 00:19 |
| *** ansmith has joined #openstack-infra | 00:19 | |
| *** anteaya has joined #openstack-infra | 00:19 | |
| openstackgerrit | Merged openstack-infra/system-config master: Don't git clone system-config in puppet-install https://review.openstack.org/597308 | 00:20 |
| fungi | ianw: the "new" zk servers are misconfigured and don't know their fqdns. when sudo runs on them it tries to resolve the hostnames they think they have, fails to be able to do that, and sends root a security warning by e-mail | 00:21 |
| fungi | unfortunately since they don't know their fqdns they also attempt to directly deliver those to the sysadmin forward addresses using a helo which isn't in dns | 00:22 |
| fungi | so then a lot of our sysadmins have addresses at mtas which reject e-mail from such misconfigured servers | 00:22 |
| fungi | the zk servers of course want root to know about these delivery failures, so they send an ndr | 00:23 |
| ianw | oh, and so the new puppet role exacerbates this because i've put in "become: yes" calls? | 00:24 |
| fungi | seems so. now instead of a few each night because of cron, we get a couple dozen or more every hour | 00:24 |
| ianw | i put them there because as also zuul roles (which may not implicitly be run as root as we do for playbooks/roles), i think they need them? | 00:24 |
| fungi | i'm definitely not questioning the need for become: yes | 00:25 |
| fungi | more whether we should be keeping these zk servers up and running (seems like we're still not actually using them?) | 00:25 |
| fungi | and if we do decide we need to keep them up and running, i guess we can fix whatever the various hostname misconfigurations are on them | 00:26 |
| *** jamesmcarthur has joined #openstack-infra | 00:27 | |
| *** longkb has joined #openstack-infra | 00:32 | |
| *** rlandy is now known as rlandy|bbl | 00:33 | |
| *** anteaya has quit IRC | 00:36 | |
| mnaser | ooo | 00:37 |
| mnaser | i see vms getting spun up | 00:37 |
| *** rh-jelabarre has quit IRC | 00:38 | |
| *** gyee has quit IRC | 00:42 | |
| *** bobh has joined #openstack-infra | 00:43 | |
| *** mriedem has quit IRC | 00:44 | |
| mnaser | only 2 vms though | 00:45 |
| mnaser | infra-root: anyone around to check things out in nodepool? | 00:45 |
| ianw | mnaser: one tick ... | 00:49 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [wip] rotate ansible logfiles https://review.openstack.org/597333 | 00:49 |
| ianw | hrm, seeing limestone failiures | 00:52 |
| ianw | 2018-08-29 00:51:40,395 ERROR nodepool.NodeLauncher-0001617694: Launch failed for node centos-7-limestone-regionone-0001617694 | 00:52 |
| ianw | and {"forbidden": {"message": "Maximum number of ports exceeded", "code": 403} on packethost | 00:53 |
| ianw | sigh ... did i really want to look! :) | 00:53 |
| ianw | tonyb: what is that doing in the check pipeline? i can't see it could get secrets to copy anything there? | 00:56 |
| tonyb | ianw: It's just triggered on any jobs that touch bindep.txt | 00:57 |
| ianw | mnaser: ok ... Predicted remaining provider quota: {'compute': {'cores': -4, 'instances': 7, 'ram': 26624}} | 00:57 |
| tonyb | ianw: If that isn't correct I don't mind removing that config | 00:57 |
| mnaser | darn | 00:57 |
| mnaser | of course | 00:57 |
| mnaser | ianw: one second | 00:57 |
| mnaser | ianw: boom (a good boom hopefully) | 00:58 |
| mnaser | quota might be cached though | 00:59 |
| mnaser | Bumped volume quota too | 00:59 |
| ianw | mnaser: ok, i'm just tailing a "grep -i sjc" on the launcher logs ... still currently getting those messages but as you say, might take a bit for shade to realise | 01:00 |
| mnaser | So maybe a nodepool restart might be necessary | 01:00 |
| mnaser | Last time I think clarkb restated nodepool to get rid of the cache | 01:01 |
| tonyb | ianw: when you're not talking to mnaser ... it's being run because of: http://git.openstack.org/cgit/openstack/requirements/tree/.zuul.d/project.yaml#n10 which as I said in the rlease channel if that's wrong I'm happy to remove it | 01:02 |
| *** jcoufal has quit IRC | 01:02 | |
| ianw | tonyb: i'm guessing that's to indicate if the builds are working? do we have a separate publish step? we must i imagine? or why is afs getting installed here? | 01:03 |
| *** yamamoto has quit IRC | 01:03 | |
| *** yamamoto has joined #openstack-infra | 01:03 | |
| ianw | mnaser: hrm, the launcher has been around for a few days, so not sure if he restarted that at least | 01:03 |
| mnaser | ianw: oh I mean the quota thing, the last time it happened, it was restarted to get it to | 01:04 |
| mnaser | Forget cache | 01:04 |
| mnaser | But don’t take my word for it | 01:04 |
| clarkb | ya shade aggressively caches things | 01:05 |
| clarkb | I cant remember if quota had that issie though definitely other things like flavor listings rhough | 01:05 |
| ianw | clarkb: yeah, any idea how long it takes to timeout? nodepool-launcher has been running since aug07 on nl03 which is where i'm looking | 01:05 |
| clarkb | in the case of flavors never, it loaded them once and never reloads unless the provider is reconfigured | 01:08 |
| ianw | mnaser: ok, restarted launcher ... did it just send a bunch of jobs your way? | 01:10 |
| fungi | any infra-puppet-core up for approving https://review.openstack.org/596398 ? if so, i can keep an eye on the meetbot restart | 01:12 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [wip] rotate ansible logfiles https://review.openstack.org/597333 | 01:13 |
| ianw | fungi: lgtm ... if you want more irc things to review also see https://review.openstack.org/#/c/589001/ | 01:13 |
| *** fungi has quit IRC | 01:18 | |
| *** fungi has joined #openstack-infra | 01:18 | |
| *** calbers has quit IRC | 01:19 | |
| *** kambiz has quit IRC | 01:19 | |
| *** Shrews has quit IRC | 01:19 | |
| *** odyssey4me has quit IRC | 01:19 | |
| *** bgmccollum has quit IRC | 01:19 | |
| *** gary_perkins has quit IRC | 01:19 | |
| *** dosaboy has quit IRC | 01:19 | |
| *** dtruong has quit IRC | 01:19 | |
| *** beagles has quit IRC | 01:19 | |
| *** sdoran has quit IRC | 01:19 | |
| *** dtantsur|afk has quit IRC | 01:19 | |
| *** beagles has joined #openstack-infra | 01:19 | |
| *** odyssey4me has joined #openstack-infra | 01:20 | |
| ianw | TASK [emit-job-header : Print job information] thank you whoever got this in!!!! ++ | 01:20 |
| *** kambiz has joined #openstack-infra | 01:21 | |
| *** calbers has joined #openstack-infra | 01:21 | |
| *** dtantsur has joined #openstack-infra | 01:22 | |
| *** bgmccollum has joined #openstack-infra | 01:22 | |
| *** Bhujay has joined #openstack-infra | 01:25 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [wip] rotate ansible logfiles https://review.openstack.org/597333 | 01:26 |
| *** dpawlik has joined #openstack-infra | 01:29 | |
| *** auristor has quit IRC | 01:29 | |
| *** dpawlik has quit IRC | 01:33 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [wip] rotate ansible logfiles https://review.openstack.org/597333 | 01:36 |
| openstackgerrit | Merged openstack-infra/puppet-meetbot master: Wait to join channels until identified by NickServ https://review.openstack.org/596398 | 01:36 |
| *** bobh has quit IRC | 01:41 | |
| *** jamesmcarthur has quit IRC | 01:42 | |
| *** hongbin has joined #openstack-infra | 01:48 | |
| *** dkehn has quit IRC | 01:49 | |
| *** pas-ha has quit IRC | 01:49 | |
| *** eglute has quit IRC | 01:49 | |
| *** simondodsley_ has quit IRC | 01:49 | |
| *** ianw has quit IRC | 01:49 | |
| *** SergeyLukjanov has quit IRC | 01:49 | |
| *** andreykurilin has quit IRC | 01:49 | |
| *** zigo has quit IRC | 01:49 | |
| *** spsurya has quit IRC | 01:49 | |
| *** jamielennox has quit IRC | 01:49 | |
| *** geekinutah has quit IRC | 01:49 | |
| *** rm_work has quit IRC | 01:49 | |
| *** persia has quit IRC | 01:49 | |
| *** andreykurilin has joined #openstack-infra | 01:49 | |
| *** SergeyLukjanov has joined #openstack-infra | 01:51 | |
| *** ianw has joined #openstack-infra | 01:53 | |
| *** jamielennox has joined #openstack-infra | 01:53 | |
| *** rm_work has joined #openstack-infra | 01:55 | |
| *** persia has joined #openstack-infra | 01:55 | |
| *** eernst has quit IRC | 01:57 | |
| *** Shrews has joined #openstack-infra | 02:00 | |
| *** auristor has joined #openstack-infra | 02:00 | |
| *** lewo has quit IRC | 02:01 | |
| *** vtapia has joined #openstack-infra | 02:01 | |
| *** ianw has quit IRC | 02:03 | |
| *** ianw has joined #openstack-infra | 02:04 | |
| ianw | huh i guess i didn't have sasl setup right | 02:05 |
| *** larainema has joined #openstack-infra | 02:11 | |
| fungi | ianw: fwiw, it seems like when they force evacuate servers the stampeding herd overloads their auth backend and i get sasl timeouts | 02:12 |
| fungi | i need to see if weechat has an option to just keep reconnecting if sasl doesn't succeed | 02:13 |
| *** rlandy|bbl is now known as rlandy | 02:13 | |
| *** psachin has joined #openstack-infra | 02:16 | |
| *** bobh has joined #openstack-infra | 02:16 | |
| *** jamesmcarthur has joined #openstack-infra | 02:17 | |
| *** hamzy has quit IRC | 02:17 | |
| *** hamzy has joined #openstack-infra | 02:18 | |
| *** eumel8 has quit IRC | 02:19 | |
| *** bgmccollum has quit IRC | 02:19 | |
| mnaser | I’m out of my hotel. I’ll be back soon | 02:19 |
| *** bgmccollum has joined #openstack-infra | 02:25 | |
| *** jamesmcarthur has quit IRC | 02:26 | |
| pabelanger | ianw: yah, we managed to land nodepool info in emit-job-header today | 02:26 |
| *** openstack has joined #openstack-infra | 02:51 | |
| *** zzzeek has joined #openstack-infra | 02:52 | |
| *** ChanServ sets mode: +o openstack | 02:52 | |
| *** dtroyer has joined #openstack-infra | 02:57 | |
| *** jhesketh has joined #openstack-infra | 02:57 | |
| *** jamesmcarthur has joined #openstack-infra | 02:59 | |
| *** openstackstatus has joined #openstack-infra | 03:00 | |
| *** ChanServ sets mode: +v openstackstatus | 03:00 | |
| *** lastmikoi has joined #openstack-infra | 03:02 | |
| fungi | #status log updated chat.freenode.net entry in eavesdrop.o.o's /etc/hosts file from no-longer-active 82.96.96.11 to 38.229.70.22 (card.freenode.net) and restarted the openstack-meetbot and statusbot services | 03:05 |
| openstackstatus | fungi: finished logging | 03:05 |
| *** bobh has quit IRC | 03:06 | |
| fungi | #status log the openstack meetbot is now delaying channel joins until nickserv confirms identification; all channels should once again be logged | 03:06 |
| openstackstatus | fungi: finished logging | 03:06 |
| *** graphene has quit IRC | 03:08 | |
| *** psachin has quit IRC | 03:13 | |
| *** psachin has joined #openstack-infra | 03:13 | |
| *** bobh has joined #openstack-infra | 03:17 | |
| *** diablo_rojo has quit IRC | 03:21 | |
| *** bobh has quit IRC | 03:24 | |
| *** rlandy has quit IRC | 03:30 | |
| *** ramishra has joined #openstack-infra | 03:37 | |
| *** jamesmcarthur has quit IRC | 03:49 | |
| *** hongbin has quit IRC | 03:51 | |
| *** jamesmcarthur has joined #openstack-infra | 03:52 | |
| *** spsurya has joined #openstack-infra | 03:52 | |
| *** nicolasbock has quit IRC | 03:58 | |
| *** udesale has joined #openstack-infra | 03:59 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [wip] rotate ansible logfiles https://review.openstack.org/597333 | 04:04 |
| AJaeger | config-core, https://review.openstack.org/588842 and https://review.openstack.org/#/c/592970 are two python3-first changes to remove jobs from project-config, please review | 04:11 |
| AJaeger | config-core, and two small job removal changes from openstack-zuul-jobs, please add to your review queue: https://review.openstack.org/#/c/596984/ and https://review.openstack.org/#/c/597169/ | 04:12 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [wip] rotate ansible logfiles https://review.openstack.org/597333 | 04:13 |
| *** jamesmcarthur has quit IRC | 04:15 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add job page https://review.openstack.org/597048 | 04:20 |
| *** jamesmcarthur has joined #openstack-infra | 04:21 | |
| *** jamesmcarthur has quit IRC | 04:26 | |
| *** jamesmcarthur has joined #openstack-infra | 04:28 | |
| *** dpawlik has joined #openstack-infra | 04:29 | |
| *** jamesmcarthur has quit IRC | 04:33 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add job page https://review.openstack.org/597048 | 04:35 |
| *** jamesmcarthur has joined #openstack-infra | 04:36 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [wip] rotate ansible logfiles https://review.openstack.org/597333 | 04:37 |
| *** jamesmcarthur has quit IRC | 04:40 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Infra repo retirement - step 1 https://review.openstack.org/597370 | 04:43 |
| mnaser | ianw: back, i still see 2 vms though | 04:43 |
| mnaser | dunno if its for a lack of load or | 04:43 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Infra repo retirement - step 1 https://review.openstack.org/597370 | 04:44 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Infra repo retirement - step 1 https://review.openstack.org/597370 | 04:45 |
| ianw | mnaser: the last thing i see is | 04:48 |
| ianw | 2018-08-29 04:47:58,451 DEBUG nodepool.driver.NodeRequestHandler[nl03-30075-PoolWorker.vexxhost-sjc1-main]: Predicted remaining provider quota: {'compute': {'cores': -4, 'ram': 26624, 'instances': 7}} | 04:48 |
| ianw | so what's with the "-4" cores? | 04:49 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove obsolete legacy-js-generator-openstack-nodejs6-npm-docs https://review.openstack.org/597371 | 04:49 |
| *** dpawlik has quit IRC | 04:49 | |
| *** dpawlik has joined #openstack-infra | 04:50 | |
| *** diablo_rojo has joined #openstack-infra | 04:53 | |
| *** dpawlik has quit IRC | 04:56 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [wip] rotate ansible logfiles https://review.openstack.org/597333 | 05:00 |
| *** jamesmcarthur has joined #openstack-infra | 05:01 | |
| openstackgerrit | Merged openstack-infra/git-review master: Use doc/requirements and openstackdocstheme https://review.openstack.org/594186 | 05:03 |
| openstackgerrit | Merged openstack-infra/git-review master: Start using reno for releasenotes https://review.openstack.org/593670 | 05:03 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [wip] rotate ansible logfiles https://review.openstack.org/597333 | 05:05 |
| *** jamesmcarthur has quit IRC | 05:06 | |
| *** dtruong has joined #openstack-infra | 05:09 | |
| *** janki has joined #openstack-infra | 05:09 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [wip] rotate ansible logfiles https://review.openstack.org/597333 | 05:12 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [wip] rotate ansible logfiles https://review.openstack.org/597333 | 05:17 |
| *** jamesmcarthur has joined #openstack-infra | 05:27 | |
| *** ykarel has joined #openstack-infra | 05:28 | |
| *** jamesmcarthur has quit IRC | 05:31 | |
| *** jamesmcarthur has joined #openstack-infra | 05:33 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: roles/kerberos-client: fix defaults https://review.openstack.org/597374 | 05:34 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config master: Update kerberos client variable names https://review.openstack.org/597375 | 05:36 |
| AJaeger | ianw, frickler, mnaser , could you put https://review.openstack.org/588842 , https://review.openstack.org/#/c/596984/ , and https://review.openstack.org/#/c/597169/ on your review list, please? | 05:39 |
| *** cjloader has joined #openstack-infra | 05:40 | |
| *** ykarel has quit IRC | 05:42 | |
| *** alexchadin has joined #openstack-infra | 05:47 | |
| *** ykarel has joined #openstack-infra | 05:52 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [wip] rotate ansible logfiles https://review.openstack.org/597333 | 06:00 |
| *** jamesmcarthur has quit IRC | 06:00 | |
| *** dpawlik has joined #openstack-infra | 06:01 | |
| *** ccamacho has joined #openstack-infra | 06:03 | |
| *** dpawlik has quit IRC | 06:03 | |
| *** dpawlik has joined #openstack-infra | 06:03 | |
| *** dpawlik has quit IRC | 06:04 | |
| *** dpawlik has joined #openstack-infra | 06:04 | |
| *** udesale has quit IRC | 06:07 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove obsolete legacy-dg-hooks-dsvm https://review.openstack.org/596984 | 06:12 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy pbr jobs https://review.openstack.org/597169 | 06:12 |
| *** ykarel has quit IRC | 06:23 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [wip] rotate ansible logfiles https://review.openstack.org/597333 | 06:23 |
| openstackgerrit | Merged openstack-infra/project-config master: remove job settings for Oslo repositories https://review.openstack.org/588842 | 06:24 |
| *** alexchadin has quit IRC | 06:26 | |
| mnaser | ianw: i sent cores to -1 | 06:31 |
| *** alexchadin has joined #openstack-infra | 06:33 | |
| AJaeger | thanks, ianw ! | 06:38 |
| *** markvoelker has joined #openstack-infra | 06:39 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/activity-board master: Retire repo https://review.openstack.org/597389 | 06:45 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/err2d2 master: Retire err2d2 https://review.openstack.org/597392 | 06:48 |
| *** pcaruana has joined #openstack-infra | 06:50 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/puppet-vinz master: Retire puppet-vinz https://review.openstack.org/597396 | 06:54 |
| *** udesale has joined #openstack-infra | 06:55 | |
| *** ykarel has joined #openstack-infra | 06:55 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/vinz master: Retire vinz https://review.openstack.org/597397 | 06:55 |
| *** aojea has joined #openstack-infra | 06:55 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/vinz-webclient master: Retire vinz-webclient https://review.openstack.org/597398 | 06:56 |
| *** eumel8 has joined #openstack-infra | 07:00 | |
| *** jamesmcarthur has joined #openstack-infra | 07:00 | |
| *** rcernin has quit IRC | 07:00 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/system-config master: Remove obsolete pypi-mirror https://review.openstack.org/597399 | 07:01 |
| *** apetrich has joined #openstack-infra | 07:02 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/activity-board master: Retire repo https://review.openstack.org/597389 | 07:03 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/pypi-mirror master: Retire pypi-mirror https://review.openstack.org/597401 | 07:03 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/puppet-vinz master: Retire puppet-vinz https://review.openstack.org/597396 | 07:04 |
| *** ykarel has quit IRC | 07:05 | |
| *** shardy has joined #openstack-infra | 07:07 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Infra repo retirement - step 1 https://review.openstack.org/597370 | 07:08 |
| *** ykarel has joined #openstack-infra | 07:10 | |
| *** ykarel has quit IRC | 07:10 | |
| *** ykarel has joined #openstack-infra | 07:10 | |
| *** ykarel has quit IRC | 07:11 | |
| *** ykarel has joined #openstack-infra | 07:12 | |
| *** jtomasek has joined #openstack-infra | 07:13 | |
| *** ykarel has quit IRC | 07:13 | |
| *** longkb has quit IRC | 07:14 | |
| *** r-mibu has joined #openstack-infra | 07:14 | |
| *** takamatsu has quit IRC | 07:16 | |
| tonyb | is ask.o.o down? or is it slow because of backups or som such? | 07:18 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Infra repo retirement of 13 repos - step 1 https://review.openstack.org/597370 | 07:19 |
| *** electrofelix has joined #openstack-infra | 07:20 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Infra repo retirement of 13 repos - step 1 https://review.openstack.org/597370 | 07:22 |
| *** jesusaur has quit IRC | 07:23 | |
| *** jesusaur has joined #openstack-infra | 07:27 | |
| *** udesale has quit IRC | 07:31 | |
| *** jpich has joined #openstack-infra | 07:37 | |
| *** tosky has joined #openstack-infra | 07:38 | |
| *** longkb has joined #openstack-infra | 07:39 | |
| *** fdegir has joined #openstack-infra | 07:41 | |
| *** ykarel has joined #openstack-infra | 07:49 | |
| *** threestrands has quit IRC | 07:50 | |
| *** jpena|off is now known as jpena | 07:51 | |
| frickler | tonyb: works for me when checking now. probably short outage during daily cron jobs | 07:55 |
| tonyb | frickler: thanks | 07:56 |
| AJaeger | config-core, one more review, please - helping with python3-first: https://review.openstack.org/#/c/597035/ | 07:56 |
| *** e0ne has joined #openstack-infra | 07:56 | |
| *** zigo has joined #openstack-infra | 07:57 | |
| *** Emine has joined #openstack-infra | 07:58 | |
| *** e0ne has quit IRC | 08:00 | |
| *** jamesmcarthur_ has joined #openstack-infra | 08:01 | |
| *** graphene has joined #openstack-infra | 08:02 | |
| *** jamesmcarthur has quit IRC | 08:03 | |
| AJaeger | ttx, is openstack-infra/releasestatus and openstack-infra/puppet-releasestatus still used anywhere? | 08:06 |
| *** owalsh_ is now known as owalsh | 08:10 | |
| *** shabeeb has joined #openstack-infra | 08:11 | |
| *** ccamacho has quit IRC | 08:13 | |
| *** ccamacho has joined #openstack-infra | 08:14 | |
| *** hashar has joined #openstack-infra | 08:26 | |
| *** graphene has quit IRC | 08:27 | |
| *** ykarel is now known as ykarel|away | 08:27 | |
| *** alexchadin has quit IRC | 08:27 | |
| *** graphene has joined #openstack-infra | 08:28 | |
| *** ykarel|away has quit IRC | 08:29 | |
| AJaeger | config-core, congress has finished moving in-repos, we can move forward with https://review.openstack.org/592970 - please review | 08:29 |
| openstackgerrit | Markus Hosch proposed openstack-infra/nodepool master: Add metric for image build result https://review.openstack.org/590412 | 08:31 |
| *** e0ne has joined #openstack-infra | 08:38 | |
| openstackgerrit | Markus Hosch proposed openstack-infra/nodepool master: Add list of metrics provided to statsd https://review.openstack.org/590233 | 08:45 |
| *** longkb has quit IRC | 08:51 | |
| *** ccamacho has quit IRC | 08:55 | |
| *** ccamacho has joined #openstack-infra | 08:56 | |
| *** ykarel has joined #openstack-infra | 08:56 | |
| *** sshnaidm|afk is now known as sshnaidm | 08:59 | |
| *** noama has joined #openstack-infra | 08:59 | |
| *** dpawlik has quit IRC | 09:03 | |
| *** dpawlik has joined #openstack-infra | 09:04 | |
| *** gema has joined #openstack-infra | 09:07 | |
| *** wendar has joined #openstack-infra | 09:10 | |
| *** Bhujay has quit IRC | 09:10 | |
| *** dpawlik has quit IRC | 09:24 | |
| ttx | AJaeger: no it's not used anymore | 09:26 |
| *** dpawlik has joined #openstack-infra | 09:28 | |
| AJaeger | thanks for confirming, ttx | 09:29 |
| *** holser_ has joined #openstack-infra | 09:30 | |
| *** gfidente has joined #openstack-infra | 09:31 | |
| *** ykarel is now known as ykarel|away | 09:36 | |
| openstackgerrit | Markus Hosch proposed openstack-infra/nodepool master: Add metric for image build result https://review.openstack.org/590412 | 09:40 |
| *** ykarel|away has quit IRC | 09:43 | |
| *** priteau has joined #openstack-infra | 09:46 | |
| *** sambetts|afk has joined #openstack-infra | 09:52 | |
| *** udesale has joined #openstack-infra | 09:52 | |
| *** diablo_rojo has quit IRC | 09:53 | |
| *** ykarel has joined #openstack-infra | 09:57 | |
| openstackgerrit | Frank Kloeker proposed openstack-infra/puppet-ptgbot master: [WIP] Make ptgbot fit for Stein https://review.openstack.org/597437 | 10:04 |
| openstackgerrit | Merged openstack-infra/project-config master: remove job settings for congress repositories https://review.openstack.org/592970 | 10:10 |
| openstackgerrit | Merged openstack-infra/project-config master: Remove job settings for monasca-agent https://review.openstack.org/597035 | 10:11 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: [wip] rotate ansible logfiles https://review.openstack.org/597333 | 10:11 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/releasestatus master: Retire releasestatus https://review.openstack.org/597439 | 10:21 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/puppet-releasestatus master: Retire puppet-releasestatus https://review.openstack.org/597440 | 10:21 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/system-config master: Remove releasestatus configuration https://review.openstack.org/597442 | 10:26 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Infra repo retirement of 14 repos - step 1 https://review.openstack.org/597370 | 10:27 |
| *** psachin has quit IRC | 10:28 | |
| *** graphene has quit IRC | 10:38 | |
| *** graphene has joined #openstack-infra | 10:40 | |
| *** yamamoto_ has joined #openstack-infra | 10:41 | |
| *** yamamoto has quit IRC | 10:45 | |
| *** AJaeger has quit IRC | 10:49 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add logrotate role and rotate ansible log files https://review.openstack.org/597333 | 10:51 |
| *** shabeeb has quit IRC | 10:51 | |
| *** shabeeb has joined #openstack-infra | 10:51 | |
| *** AJaeger has joined #openstack-infra | 10:52 | |
| *** shabeeb has quit IRC | 10:56 | |
| *** srp has joined #openstack-infra | 10:56 | |
| *** srp has quit IRC | 10:56 | |
| *** srp has joined #openstack-infra | 10:57 | |
| *** shabeeb has joined #openstack-infra | 10:59 | |
| *** longkb has joined #openstack-infra | 11:01 | |
| *** snapiri has joined #openstack-infra | 11:01 | |
| *** srp has quit IRC | 11:01 | |
| *** electrofelix has quit IRC | 11:02 | |
| *** d0ugal has quit IRC | 11:03 | |
| *** udesale has quit IRC | 11:04 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add logrotate role and rotate ansible log files https://review.openstack.org/597333 | 11:07 |
| *** nicolasbock has joined #openstack-infra | 11:11 | |
| *** electrofelix has joined #openstack-infra | 11:14 | |
| openstackgerrit | Rafael Folco proposed openstack-infra/project-config master: Switch to emit-zuul-job-header-test role https://review.openstack.org/597236 | 11:14 |
| *** gary_perkins has joined #openstack-infra | 11:15 | |
| *** r-mibu has quit IRC | 11:16 | |
| *** ginopc has joined #openstack-infra | 11:16 | |
| *** d0ugal has joined #openstack-infra | 11:17 | |
| *** viks__ has quit IRC | 11:17 | |
| openstackgerrit | Rafael Folco proposed openstack-infra/zuul-jobs master: [DNM] base-test #597235 https://review.openstack.org/597449 | 11:20 |
| *** dpawlik has quit IRC | 11:21 | |
| *** jcoufal has joined #openstack-infra | 11:24 | |
| openstackgerrit | Rafael Folco proposed openstack-infra/project-config master: Switch to emit-zuul-job-header-test role https://review.openstack.org/597236 | 11:28 |
| *** shabeeb has quit IRC | 11:32 | |
| *** shabeeb has joined #openstack-infra | 11:32 | |
| *** ginopc has quit IRC | 11:34 | |
| openstackgerrit | Rafael Folco proposed openstack-infra/zuul-jobs master: Set zuul job start time fact https://review.openstack.org/597235 | 11:35 |
| *** snapiri has quit IRC | 11:35 | |
| *** shabeeb has quit IRC | 11:37 | |
| *** jpena is now known as jpena|lunch | 11:37 | |
| openstackgerrit | Merged openstack-infra/devstack-gate master: Add gate job for stable/rocky https://review.openstack.org/596280 | 11:42 |
| *** longkb has quit IRC | 11:42 | |
| *** tpsilva has joined #openstack-infra | 11:43 | |
| *** shabeeb has joined #openstack-infra | 11:48 | |
| *** aojea has quit IRC | 11:51 | |
| *** ssbarnea|ruck has joined #openstack-infra | 11:54 | |
| *** shabeeb has quit IRC | 11:55 | |
| *** shabeeb has joined #openstack-infra | 11:56 | |
| *** vivsoni has quit IRC | 11:58 | |
| *** vivsoni has joined #openstack-infra | 11:59 | |
| *** bobh has joined #openstack-infra | 12:03 | |
| *** ssbarnea|ruck has quit IRC | 12:03 | |
| *** rosmaita has joined #openstack-infra | 12:03 | |
| *** rh-jelabarre has joined #openstack-infra | 12:03 | |
| *** vivsoni has quit IRC | 12:05 | |
| *** vivsoni has joined #openstack-infra | 12:05 | |
| *** vivsoni has quit IRC | 12:06 | |
| *** janki has quit IRC | 12:06 | |
| *** vivsoni has joined #openstack-infra | 12:07 | |
| *** ccamacho has quit IRC | 12:07 | |
| *** ccamacho has joined #openstack-infra | 12:09 | |
| *** bobh has quit IRC | 12:10 | |
| *** beagles has quit IRC | 12:12 | |
| pabelanger | infra-root: I've manually fixed /etc/hosts on zk0[123], should stop sudo spam | 12:13 |
| *** beagles has joined #openstack-infra | 12:13 | |
| *** dpawlik has joined #openstack-infra | 12:14 | |
| *** udesale has joined #openstack-infra | 12:14 | |
| *** udesale has quit IRC | 12:16 | |
| *** udesale has joined #openstack-infra | 12:19 | |
| *** sthussey has joined #openstack-infra | 12:20 | |
| *** ykarel_ has joined #openstack-infra | 12:20 | |
| *** ykarel has quit IRC | 12:22 | |
| *** signed8bit_Zzz is now known as signed8bit | 12:22 | |
| *** trown|outtypewww is now known as trown | 12:25 | |
| openstackgerrit | Rafael Folco proposed openstack-infra/zuul-jobs master: [DNM] base-test #597235 https://review.openstack.org/597449 | 12:26 |
| *** udesale has joined #openstack-infra | 12:28 | |
| *** udesale has quit IRC | 12:28 | |
| *** sshnaidm is now known as sshnaidm|afk | 12:28 | |
| *** rlandy has joined #openstack-infra | 12:29 | |
| *** janki has joined #openstack-infra | 12:30 | |
| *** udesale has joined #openstack-infra | 12:31 | |
| *** yamamoto has joined #openstack-infra | 12:33 | |
| openstackgerrit | Nguyen Hai proposed openstack-infra/project-config master: remove job settings for dragonflow repositories https://review.openstack.org/597475 | 12:33 |
| *** ykarel_ is now known as ykarel | 12:36 | |
| *** jpich has quit IRC | 12:36 | |
| *** yamamoto_ has quit IRC | 12:36 | |
| *** kgiusti has joined #openstack-infra | 12:37 | |
| *** boden has joined #openstack-infra | 12:40 | |
| *** mriedem has joined #openstack-infra | 12:42 | |
| openstackgerrit | Nguyen Hai proposed openstack-infra/project-config master: remove job settings for senlin repositories https://review.openstack.org/597502 | 12:45 |
| *** jpena|lunch is now known as jpena | 12:46 | |
| *** eharney has joined #openstack-infra | 12:47 | |
| *** jamesmcarthur_ has quit IRC | 12:50 | |
| *** jamesmcarthur has joined #openstack-infra | 12:50 | |
| *** rfolco has joined #openstack-infra | 12:54 | |
| *** ginopc has joined #openstack-infra | 12:55 | |
| *** ccamacho has quit IRC | 12:58 | |
| *** ccamacho has joined #openstack-infra | 12:59 | |
| *** sshnaidm|afk is now known as sshnaidm | 13:02 | |
| *** dtantsur is now known as dtantsur|bbl | 13:04 | |
| *** pcaruana has quit IRC | 13:04 | |
| *** snapiri has joined #openstack-infra | 13:08 | |
| *** jamesmcarthur has quit IRC | 13:15 | |
| openstackgerrit | Rafael Folco proposed openstack-infra/zuul-jobs master: [WIP] Set zuul job start time fact https://review.openstack.org/597513 | 13:17 |
| openstackgerrit | Dan Radez proposed openstack-infra/project-config master: adding readthedocs to networking-ansible https://review.openstack.org/597194 | 13:18 |
| openstackgerrit | Nguyen Hai proposed openstack-infra/project-config master: remove job settings for rally repositories https://review.openstack.org/597517 | 13:18 |
| *** auristor has quit IRC | 13:30 | |
| *** srp has joined #openstack-infra | 13:37 | |
| *** shabeeb has quit IRC | 13:40 | |
| *** auristor has joined #openstack-infra | 13:41 | |
| *** rajinir has joined #openstack-infra | 13:42 | |
| *** dkehn has joined #openstack-infra | 13:45 | |
| *** jamesmcarthur has joined #openstack-infra | 13:46 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul-packaging master: Retire zuul-packaging https://review.openstack.org/597531 | 13:52 |
| *** markvoelker has quit IRC | 13:53 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Infra repo retirement of 15 repos - step 1 https://review.openstack.org/597370 | 13:53 |
| *** signed8bit is now known as signed8bit_Zzz | 13:57 | |
| *** dtantsur|bbl is now known as dtantsur | 13:58 | |
| *** signed8bit_Zzz is now known as signed8bit | 14:00 | |
| *** knikolla has joined #openstack-infra | 14:02 | |
| AJaeger | config-core, next change to move jobs out - searchlight finished python3-first: please review https://review.openstack.org/#/c/594033/ | 14:04 |
| openstackgerrit | Trinh Nguyen proposed openstack-infra/irc-meetings master: Update Searchlight team meeting https://review.openstack.org/597537 | 14:05 |
| *** lbragstad has joined #openstack-infra | 14:07 | |
| *** srp has quit IRC | 14:07 | |
| fungi | thanks pabelanger! my inbox thanks you mightily | 14:09 |
| *** ykarel is now known as ykarel|afk | 14:11 | |
| dtroyer | fungi: would you have a minute to look at https://review.openstack.org/#/c/596426/ for me? | 14:14 |
| *** ykarel|afk has quit IRC | 14:15 | |
| AJaeger | TheJulia: do you want to change https://review.openstack.org/596844 as suggested by clarkb ? | 14:22 |
| TheJulia | AJaeger: I can, just on calls for the next two hours. | 14:23 |
| TheJulia | I will try to do so in a little bit | 14:23 |
| AJaeger | TheJulia: not urgent from my side ;) thanks | 14:23 |
| *** bobh has joined #openstack-infra | 14:28 | |
| openstackgerrit | Rafael Folco proposed openstack-infra/zuul-jobs master: Set zuul job start time fact https://review.openstack.org/597235 | 14:29 |
| *** shabeeb has joined #openstack-infra | 14:29 | |
| fungi | dtroyer: sure thing | 14:29 |
| fungi | #status log reset /opt/etherpad-lite/etherpad-lite on etherpad-dev.o.o to the 1.7.0 release and restarted the etherpad-lite service for testing | 14:29 |
| openstackstatus | fungi: finished logging | 14:29 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Update etherpad.o.o to etherpad-lite 1.7.0 https://review.openstack.org/597544 | 14:30 |
| fungi | infra-root: ^ we may want to slightly prioritize that | 14:30 |
| TheJulia | AJaeger: pushing up now | 14:33 |
| openstackgerrit | Julia Kreger proposed openstack-infra/project-config master: Add virtualbmc-core group https://review.openstack.org/596844 | 14:33 |
| AJaeger | TheJulia: LGTM | 14:34 |
| *** shabeeb has quit IRC | 14:36 | |
| AJaeger | fungi, here're two small changes, could you review them as well, please? https://review.openstack.org/#/c/594033/ and https://review.openstack.org/#/c/588980/ | 14:36 |
| AJaeger | thanks, fungi ! | 14:40 |
| openstackgerrit | Merged openstack-infra/project-config master: Add pushMerge for starlingx-release https://review.openstack.org/596426 | 14:41 |
| *** markvoelker has joined #openstack-infra | 14:44 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: move legacy-tempest-dsvm-full-zmq to openstack/devstack-plugin-zmq https://review.openstack.org/588980 | 14:45 |
| openstackgerrit | Merged openstack-infra/project-config master: remove job settings for searchlight repositories https://review.openstack.org/594033 | 14:48 |
| *** pcaruana has joined #openstack-infra | 14:50 | |
| dtroyer | thanks fungi! | 14:50 |
| *** ykarel has joined #openstack-infra | 14:57 | |
| fungi | i'm testing https://etherpad-dev.openstack.org/p/things for any obvious problems we might run into with 597544, if anybody wants to help | 14:57 |
| *** pbourke has quit IRC | 15:01 | |
| *** pbourke has joined #openstack-infra | 15:04 | |
| *** janki has quit IRC | 15:04 | |
| *** knikolla has quit IRC | 15:14 | |
| *** knikolla has joined #openstack-infra | 15:15 | |
| *** dklyle has quit IRC | 15:22 | |
| *** dklyle has joined #openstack-infra | 15:23 | |
| *** ykarel has quit IRC | 15:25 | |
| *** ykarel has joined #openstack-infra | 15:27 | |
| *** bramwelt has joined #openstack-infra | 15:27 | |
| *** Emine has quit IRC | 15:27 | |
| *** jtomasek has quit IRC | 15:29 | |
| openstackgerrit | Merged openstack-infra/puppet-ptgbot master: Push configuration for Denver PTG4 https://review.openstack.org/596765 | 15:33 |
| clarkb | corvus: followup on logstash firewall rules from yesterday, elastic-recheck looks caught up so I think that fixed the outstanding issues. | 15:36 |
| clarkb | mnaser: how is sjc1 looking? | 15:36 |
| *** ykarel_ has joined #openstack-infra | 15:39 | |
| *** graphene has quit IRC | 15:41 | |
| corvus | clarkb: \o/ | 15:42 |
| corvus | what should we be breaking now? | 15:42 |
| *** ykarel has quit IRC | 15:42 | |
| *** graphene has joined #openstack-infra | 15:43 | |
| clarkb | corvus: cloud launcher is what I'm hoping I'll get too before the end of the day. Before that I want to follow up on winterscale thread, draw up a list of trusty nodes that needs upgrades (will help identify the ansible python2 needs as well), and sign the new gpg release key | 15:43 |
| clarkb | pabelanger is working on clouds.yaml stuff so that should be up soon hopefully | 15:43 |
| pabelanger | Yup, I was going to base it atop of https://review.openstack.org/597185/ which collects testinfra logs | 15:44 |
| *** fnordahl has joined #openstack-infra | 15:45 | |
| *** r-daneel has joined #openstack-infra | 15:48 | |
| *** markvoelker has quit IRC | 15:48 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add job page https://review.openstack.org/597048 | 15:49 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add config-errors notifications drawer https://review.openstack.org/597147 | 15:49 |
| clarkb | http://grafana.openstack.org/d/nuvIH5Imk/nodepool-vexxhost?orgId=1 seems we aren't using much of the vexxhost quota yet. I may start there as a finish up to yesterday's work | 15:51 |
| clarkb | but first tea and breakfast | 15:51 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add config-errors notifications drawer https://review.openstack.org/597147 | 15:52 |
| *** diablo_rojo has joined #openstack-infra | 15:53 | |
| *** rosmaita has quit IRC | 15:57 | |
| *** efried is now known as efried_rollin | 15:57 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Install limestone CA on openstack using hosts https://review.openstack.org/593029 | 15:58 |
| *** jpich has joined #openstack-infra | 15:59 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Generate junit.xml file for testinfra https://review.openstack.org/597185 | 16:01 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Install limestone CA on openstack using hosts https://review.openstack.org/593029 | 16:01 |
| *** slaweq has quit IRC | 16:11 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add private key storage migration https://review.openstack.org/592213 | 16:12 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add job page https://review.openstack.org/597048 | 16:13 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: remove job settings for OpenStack-Helm repositories https://review.openstack.org/597574 | 16:14 |
| *** ramishra has quit IRC | 16:15 | |
| AJaeger | dhellmann: https://review.openstack.org/593264 needs rebasing - removing the WIP alone is not enough... | 16:15 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Install limestone CA on hosts using openstacksdk https://review.openstack.org/593029 | 16:15 |
| AJaeger | dhellmann: want to do that? That's the mistral one... | 16:16 |
| *** slaweq has joined #openstack-infra | 16:16 | |
| dhellmann | AJaeger : I was going to let Nguyen do it | 16:16 |
| *** ykarel_ is now known as ykarel|away | 16:16 | |
| *** udesale has quit IRC | 16:16 | |
| dhellmann | I'm working on proposing the missing cleanup patches right now | 16:16 |
| AJaeger | dhellmann: fine with me | 16:16 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: remove job settings for OpenStackAnsible repositories https://review.openstack.org/597576 | 16:17 |
| dhellmann | AJaeger : we're ready to go ahead with https://review.openstack.org/#/c/593706 | 16:18 |
| AJaeger | config-core, please review ^ | 16:19 |
| *** ginopc has quit IRC | 16:19 | |
| AJaeger | dhellmann: LGTM | 16:19 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: remove job settings for OpenStackClient repositories https://review.openstack.org/597577 | 16:20 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: base-test: iptables: allow zuul console streaming https://review.openstack.org/597305 | 16:20 |
| *** slaweq has quit IRC | 16:21 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: remove job settings for OpenStackSDK repositories https://review.openstack.org/597579 | 16:21 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: remove job settings for ec2-api repositories https://review.openstack.org/597582 | 16:23 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: remove job settings for freezer repositories https://review.openstack.org/597583 | 16:24 |
| *** hashar is now known as hasharAway | 16:28 | |
| *** jpena is now known as jpena|off | 16:31 | |
| *** jpich has quit IRC | 16:32 | |
| *** trown is now known as trown|lunch | 16:33 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Change project.private_key to private_secrets_key https://review.openstack.org/597302 | 16:34 |
| *** slaweq has joined #openstack-infra | 16:35 | |
| clarkb | dhellmann: is going to win the most lines of code removed award for this cycle :) | 16:36 |
| * dhellmann hopes the trophy comes with a beer | 16:36 | |
| *** psachin has joined #openstack-infra | 16:39 | |
| *** r-daneel has quit IRC | 16:40 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Install limestone CA on hosts using openstacksdk https://review.openstack.org/593029 | 16:42 |
| clarkb | mnaser: ianw still seeing 2018-08-29 16:44:09,149 DEBUG nodepool.driver.NodeRequestHandler[nl03-30075-PoolWorker.vexxhost-sjc1-main]: Predicted remaining provider quota: {'compute': {'cores': -4, 'ram': 26624, 'instances': 7}} with sjc1 | 16:44 |
| clarkb | I'll manually list quotas ina bit to check them myself, but found an unrelated bug in nodepool that I want to fix first | 16:44 |
| *** rkukura_ has joined #openstack-infra | 16:45 | |
| clarkb | corvus: Shrews http://paste.openstack.org/show/729064/ is the unrelated bug if curious | 16:45 |
| openstackgerrit | Merged openstack-infra/project-config master: remove job settings for Technical Committee repositories https://review.openstack.org/593706 | 16:45 |
| clarkb | I think zuul is giving hold expiries in string? | 16:45 |
| *** ykarel|away has quit IRC | 16:46 | |
| *** rkukura has quit IRC | 16:47 | |
| *** rkukura_ is now known as rkukura | 16:47 | |
| corvus | oh interesting | 16:47 |
| corvus | maybe that's a bug in zuul? | 16:47 |
| clarkb | all of the defaults are in int so this must've been human command argument passed as a string I think | 16:48 |
| clarkb | corvus: ya | 16:48 |
| clarkb | I'm sorting out a fix | 16:48 |
| corvus | i'm guessing the feature's author just used the nodepool default | 16:48 |
| *** slaweq has quit IRC | 16:49 | |
| clarkb | ya need a type=int in the argparse, fix up in a sec | 16:49 |
| fungi | dhellmann: it ought to come with a mandatory vacation | 16:49 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: remove job settings for kolla repositories https://review.openstack.org/597590 | 16:51 |
| corvus | we can "reward" people with mandatory "vacations"? that sounds useful ;) | 16:51 |
| *** shardy has quit IRC | 16:52 | |
| dhellmann | that should definitely be a thing | 16:52 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: remove job settings for magnum repositories https://review.openstack.org/597591 | 16:52 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: The argparse type of node-hold-expiration is int https://review.openstack.org/597592 | 16:53 |
| corvus | clarkb: in related news, i guess i have some nodes i need to manually clean up eh? | 16:53 |
| corvus | i'll do that now | 16:53 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: remove job settings for murano repositories https://review.openstack.org/597593 | 16:53 |
| *** dtantsur is now known as dtantsur|afk | 16:54 | |
| clarkb | corvus: ya or update the zk to integers and nodepool will do it for you | 16:54 |
| clarkb | now to investigate sjc1 quotas | 16:54 |
| *** ccamacho has quit IRC | 16:54 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: remove job settings for qinling repositories https://review.openstack.org/597597 | 16:55 |
| *** psachin has quit IRC | 16:56 | |
| *** e0ne has quit IRC | 16:56 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: remove job settings for storlets repositories https://review.openstack.org/597598 | 16:57 |
| *** gyee has joined #openstack-infra | 16:59 | |
| *** diablo_rojo has quit IRC | 16:59 | |
| clarkb | mnaser: http://paste.openstack.org/show/729067/ relevant output from limits show --absoluve against openstackjenkins-vexxhost account with suggested changes | 17:02 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: remove job settings for winstackers repositories https://review.openstack.org/597603 | 17:05 |
| *** snapiri has quit IRC | 17:05 | |
| corvus | clarkb: what's the core count for our flavor? | 17:05 |
| *** zul has quit IRC | 17:06 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: remove job settings for zaqar repositories https://review.openstack.org/597604 | 17:06 |
| clarkb | corvus: 8 coes | 17:08 |
| clarkb | *cores | 17:08 |
| corvus | clarkb: ok, so because nova is returning an apparently incorrect max value, when nodepool wants to create the next instance, it calculates 4-8=-4 remaining | 17:09 |
| clarkb | yup, the max value is what our quota is I think | 17:11 |
| clarkb | the quota is just out of sync with what we are trying to boot | 17:11 |
| *** rabel has joined #openstack-infra | 17:16 | |
| *** graphene has quit IRC | 17:16 | |
| rabel | hi. is this the correct channel for questions regarding tempest? | 17:16 |
| clarkb | rabel: typically #openstack-qa is a better lcoation since tempest is a qa team project, but many of them livehere too and some of us may know the answers if you want ot ask here as well | 17:18 |
| *** graphene has joined #openstack-infra | 17:18 | |
| rabel | thx. i will ask there | 17:18 |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool master: Force node hold expiration to integer type https://review.openstack.org/597609 | 17:24 |
| *** graphene has quit IRC | 17:31 | |
| *** diablo_rojo has joined #openstack-infra | 17:31 | |
| *** graphene has joined #openstack-infra | 17:33 | |
| mnaser | clarkb: i realized what i've been doing | 17:34 |
| mnaser | quota update on wrong region | 17:34 |
| fungi | hah, that certainly would yield unexpected results | 17:35 |
| mnaser | should be fixed now | 17:36 |
| mnaser | not sure if you have to do something to get rid of cached values | 17:36 |
| clarkb | mnaser: I'll check | 17:36 |
| clarkb | I do see the max is set to -1 which I think means infinity now | 17:37 |
| clarkb | looking to see if nodepool has noticed | 17:37 |
| corvus | clarkb: nodepool caches for 5 minutes, or until it hits an unexpected quota error | 17:38 |
| clarkb | corvus: gotcha will wait a few minutes then | 17:38 |
| *** shabeeb has joined #openstack-infra | 17:38 | |
| *** bhavikdbavishi has joined #openstack-infra | 17:38 | |
| openstackgerrit | Ben Nemec proposed openstack-dev/pbr master: Support provides_extra metadata https://review.openstack.org/597612 | 17:39 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add private key storage migration https://review.openstack.org/592213 | 17:43 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Change project.private_key to private_secrets_key https://review.openstack.org/597302 | 17:43 |
| *** jamesmcarthur has quit IRC | 17:43 | |
| *** Emine has joined #openstack-infra | 17:44 | |
| corvus | clarkb, pabelanger, fungi: https://review.openstack.org/597305 is ready and makes run-base console log nicer | 17:44 |
| clarkb | corvus: looking | 17:44 |
| *** jtomasek has joined #openstack-infra | 17:45 | |
| clarkb | corvus: base is what the role always sets, extra is the per service set, then test is any test rules? | 17:45 |
| corvus | clarkb: base is what we set globally. nothing now, but we could factor 22 out of the role and put it in base, for example. | 17:46 |
| corvus | (to maybe clarify -- the role itself knows nothing about this, this is infrastructure we build outside the role, and then pass it in as a single parameter to the role) | 17:47 |
| *** graphene has quit IRC | 17:48 | |
| corvus | we do the same thing with the aliases file for exim. the role only knows about one list of aliases, but we create that list by combining several other lists (base: our root alias + individual service aliases) | 17:48 |
| clarkb | got it, I did leave a question comment on the change that is mostly to do with ansible's behavuor | 17:49 |
| *** graphene has joined #openstack-infra | 17:50 | |
| corvus | clarkb: late-binding | 17:50 |
| *** trown|lunch is now known as trown|brb | 17:50 | |
| *** trown|brb is now known as trown | 17:50 | |
| clarkb | ansible variable precedence is magic | 17:51 |
| clarkb | +2 | 17:51 |
| *** noama has quit IRC | 17:52 | |
| *** slaweq has joined #openstack-infra | 17:53 | |
| *** jamesmcarthur has joined #openstack-infra | 17:57 | |
| *** diablo_rojo has quit IRC | 17:57 | |
| openstackgerrit | Merged openstack-infra/zuul master: The argparse type of node-hold-expiration is int https://review.openstack.org/597592 | 17:58 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: remove job settings for mistral repositories https://review.openstack.org/593264 | 18:01 |
| *** slaweq has quit IRC | 18:03 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add system-config-run-eavesdrop https://review.openstack.org/597617 | 18:05 |
| corvus | clarkb, pabelanger, fungi: ^ that should theoretically give us a little more confidence in the iptables streaming change -- it should exercise the extra tcp ports thing (eavesdrop sets an extra tcp port of 80) | 18:06 |
| AJaeger | dhellmann: looking at magnum, I see that you migrate install-guide jobs. That shouldn't be done, those run only on older branches, see http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/jobs.yaml#n803 | 18:06 |
| dhellmann | AJaeger : the migration script deals with that. If a job applies to multiple branches it is added to only those branches. | 18:06 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add system-config-run-eavesdrop https://review.openstack.org/597617 | 18:07 |
| AJaeger | dhellmann: the install-guide-jobs template uses those jobs. I think your script is not aware of that indirection | 18:07 |
| dhellmann | ah | 18:07 |
| AJaeger | dhellmann: https://review.openstack.org/#/c/595476/1/.zuul.yaml | 18:07 |
| AJaeger | that's rocky - but install-guide-jobs job runs only on newton and ocata. | 18:08 |
| dhellmann | it does try to look for job definitions, but it isn't always successful at finding them | 18:08 |
| dhellmann | ok | 18:08 |
| dhellmann | it's probably simplest to just leave that configured in project-config for now | 18:08 |
| AJaeger | yep... | 18:08 |
| corvus | dhellmann: the zuul job api endpoint is live, btw | 18:08 |
| * AJaeger fixes magnum | 18:08 | |
| AJaeger | dhellmann: will you fix the tool? | 18:08 |
| dhellmann | yes | 18:08 |
| corvus | not sure if that's helpful, just thought i'd mention it in case it slipped by unnoticed | 18:09 |
| corvus | dhellmann: http://zuul.openstack.org/api/job/devstack for example | 18:09 |
| corvus | we don't have a project-template api endpoint yet though | 18:09 |
| dhellmann | AJaeger : https://review.openstack.org/597618 | 18:09 |
| dhellmann | corvus : yes, I didn't see that land, so thanks. | 18:10 |
| dhellmann | is that documented somewhere? | 18:10 |
| dhellmann | so I can go look at how to use it | 18:10 |
| *** electrofelix has quit IRC | 18:11 | |
| corvus | dhellmann: nope. still very much under construction, undocumented and subject to change. but i think it's close enough that using it for things like answering "does this job exist, and if so, where is it defined" is relatively low-risk. | 18:11 |
| dhellmann | that looks like it gives me some job info, but not which repos use the job | 18:11 |
| dhellmann | I suppose that query is expensive | 18:11 |
| corvus | dhellmann: yeah, we don't have that yet | 18:12 |
| dhellmann | ok, cool | 18:12 |
| AJaeger | dhellmann: thanks | 18:12 |
| dhellmann | this will be useful for some things; maybe not the cases I'm dealing with today | 18:12 |
| AJaeger | config-core, another python3-first change is ready, please +2A https://review.openstack.org/#/c/593264 | 18:13 |
| corvus | dhellmann: yeah. it'll continue to grow. | 18:13 |
| dhellmann | corvus : yep, I'm looking forward to a day when I don't have to parse yaml files :-) | 18:13 |
| clarkb | `sudo ansible all -m setup -a 'filter=ansible_distribution_release gather_subset=min'` appears to be the way to get a list of distro releases from all our hosts so that we can list which ones are trusty if anyone is wondering :) | 18:17 |
| *** graphene has quit IRC | 18:17 | |
| * clarkb runs this now to find out | 18:17 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add system-config-run-eavesdrop https://review.openstack.org/597617 | 18:18 |
| *** graphene has joined #openstack-infra | 18:18 | |
| *** e0ne has joined #openstack-infra | 18:19 | |
| openstackgerrit | Merged openstack-infra/system-config master: base-test: iptables: allow zuul console streaming https://review.openstack.org/597305 | 18:19 |
| *** zul has joined #openstack-infra | 18:20 | |
| clarkb | pabelanger: when you have a momemtn I think we still need to clean up the old review.o.o server | 18:20 |
| clarkb | pabelanger: maybe you can take a look at make a recommendation of what we do there (since you drove the upgrade to the new server) | 18:20 |
| *** rh-jelabarre has quit IRC | 18:22 | |
| *** slaweq has joined #openstack-infra | 18:22 | |
| *** e0ne_ has joined #openstack-infra | 18:23 | |
| *** e0ne has quit IRC | 18:25 | |
| EmilienM | hey infra, can someone tell me if something is blocking https://review.openstack.org/#/c/592695/ to land? | 18:27 |
| *** shabeeb has quit IRC | 18:27 | |
| *** srp has joined #openstack-infra | 18:27 | |
| clarkb | EmilienM: no, done | 18:28 |
| EmilienM | clarkb: thank you | 18:28 |
| clarkb | my above ansible command failed to find the release version for birdge.o.o because python ins't installed. Any idea why it failed to find the regular group data for that saying use python3? | 18:28 |
| *** kgiusti has left #openstack-infra | 18:29 | |
| *** rh-jelabarre has joined #openstack-infra | 18:29 | |
| *** kgiusti has joined #openstack-infra | 18:30 | |
| *** slaweq has quit IRC | 18:31 | |
| *** wolverineav has joined #openstack-infra | 18:32 | |
| *** srp has quit IRC | 18:34 | |
| *** graphene has quit IRC | 18:37 | |
| *** graphene has joined #openstack-infra | 18:39 | |
| *** e0ne_ has quit IRC | 18:39 | |
| AJaeger | dhellmann: I gave -1 or fixed myself all install-guide-jobs related changes... | 18:40 |
| dhellmann | AJaeger : thanks | 18:40 |
| *** e0ne has joined #openstack-infra | 18:41 | |
| *** e0ne has quit IRC | 18:44 | |
| *** kgiusti has left #openstack-infra | 18:46 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/puppet-docker_registry master: Retire puppet-docker_registry https://review.openstack.org/597631 | 18:47 |
| *** bhavikdbavishi has quit IRC | 18:47 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Infra repo retirement of 15 repos - step 1 https://review.openstack.org/597370 | 18:48 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Infra repo retirement of 16 repos - step 1 https://review.openstack.org/597370 | 18:49 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: remove job settings for murano repositories https://review.openstack.org/597593 | 18:49 |
| AJaeger | clarkb: I'm waiting for feedback from product group but expect the other two repos to be dead as well - but the above 16 repos can be retired IMHO - feedback welcome. I pushed step 1 out everywhere. (above = https://review.openstack.org/597370). If this is fine, I can do the follow up changes | 18:50 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: remove job settings for ec2-api repositories https://review.openstack.org/597582 | 18:51 |
| clarkb | AJaeger: ok let me look really quickly before your day ends | 18:51 |
| openstackgerrit | Merged openstack-infra/nodepool master: Force node hold expiration to integer type https://review.openstack.org/597609 | 18:51 |
| clarkb | AJaeger: ya lgtm | 18:52 |
| AJaeger | clarkb: great! | 18:52 |
| openstackgerrit | Merged openstack-infra/system-config master: Enable cbs.centos.org mirror https://review.openstack.org/592695 | 18:53 |
| *** graphene has quit IRC | 18:53 | |
| *** graphene has joined #openstack-infra | 18:55 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: remove job settings for magnum repositories https://review.openstack.org/597591 | 18:55 |
| *** e0ne has joined #openstack-infra | 18:56 | |
| clarkb | https://etherpad.openstack.org/p/201808-infra-server-upgrades-and-cleanup a list of servers that need upgrading or cleanup | 18:57 |
| clarkb | I was working on getting etherpad-dev and etherpad.o.o upgraded prior to taking last week off (and I got etehrcalc done). So I'll continue to work on those next week after the release | 18:58 |
| clarkb | but we should start looking at pushing on this a bit more now that launch node from bridge is mostly working | 18:58 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add system-config-run-eavesdrop https://review.openstack.org/597617 | 18:58 |
| corvus | helps to run the job | 18:58 |
| clarkb | infra-root https://review.openstack.org/#/c/596859/ is a change needed to get launch node working (its child in particular) | 18:59 |
| *** jamesdenton has quit IRC | 19:02 | |
| clarkb | corvus: only if you want results :P | 19:04 |
| corvus | everything passed in the last patchset! | 19:05 |
| Shrews | clarkb: hrm, the launch README mentions installing shade, but i don't think that's actually used anywhere | 19:05 |
| clarkb | Shrews: ya I think that is an additional cleanup we will need | 19:05 |
| clarkb | it works if you install it and I was focused on getting to something that works :) | 19:06 |
| Shrews | probably b/c it incidentally installs openstacksdk | 19:06 |
| clarkb | likely | 19:06 |
| clarkb | we probably want to s/shade/openstacksdk/ but would be good to do that in a followup | 19:06 |
| clarkb | (if only bceause what is there does work) | 19:06 |
| *** Emine has quit IRC | 19:08 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: remove job settings for magnum repositories https://review.openstack.org/597591 | 19:08 |
| clarkb | With release this week I don't think we should be focused onr eplacing servers until next week :) | 19:08 |
| clarkb | so its not a huge rush | 19:08 |
| clarkb | and now I must find lunch | 19:08 |
| *** pcaruana has quit IRC | 19:09 | |
| *** ijw has joined #openstack-infra | 19:11 | |
| *** ijw has quit IRC | 19:13 | |
| *** ijw has joined #openstack-infra | 19:13 | |
| *** ijw has quit IRC | 19:14 | |
| *** jamesdenton has joined #openstack-infra | 19:16 | |
| *** kgiusti has joined #openstack-infra | 19:19 | |
| openstackgerrit | Merged openstack-infra/system-config master: Use python3 and modern ansible for launch node https://review.openstack.org/596859 | 19:21 |
| openstackgerrit | Merged openstack-infra/system-config master: Unlink proper path to ansible cache https://review.openstack.org/596873 | 19:21 |
| *** jamesmcarthur has quit IRC | 19:24 | |
| *** jamesmcarthur has joined #openstack-infra | 19:24 | |
| *** rosmaita has joined #openstack-infra | 19:25 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Infra repo retirement of 16 repos - step 3 https://review.openstack.org/597636 | 19:26 |
| *** diablo_rojo has joined #openstack-infra | 19:26 | |
| *** efried_rollin is now known as efried | 19:26 | |
| AJaeger | clarkb: pushed all changes out - including governance one, see https://review.openstack.org/597639 | 19:30 |
| *** mriedem has quit IRC | 19:31 | |
| *** mriedem has joined #openstack-infra | 19:31 | |
| fungi | corvus: the failing test result on 597617,4 seems to show the port 80 rule is not making it into the running ruleset | 19:39 |
| corvus | fungi: then we just might not want to merge the parent :) | 19:39 |
| fungi | still scratching my head, not sure why is isn't | 19:39 |
| corvus | which has merged | 19:40 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/nodepool master: WIP: Implement an Amazon EC2 driver https://review.openstack.org/535558 | 19:41 |
| *** graphene has quit IRC | 19:41 | |
| corvus | the last ansible run started after the change merged -- 2018-08-29T18:45:01+00:00 | 19:42 |
| corvus | so i don't think we broke production | 19:42 |
| fungi | yeah, i'm still successfully hitting http://eavesdrop.openstack.org/ for now | 19:42 |
| corvus | fungi: it's possible we're missing something to either cause that node to be in the eavesdrop group, or for the test to pick up the group_vars | 19:42 |
| *** graphene has joined #openstack-infra | 19:43 | |
| *** graphene has quit IRC | 19:43 | |
| pabelanger | clarkb: re: review.o.o, I think we can just delete it. After we moved the volumes, I don't think we have anything else in it. | 19:43 |
| *** jamesmcarthur has quit IRC | 19:43 | |
| corvus | i'm stuffing my face with a sandwich right now, i'll jump in in a few mins | 19:43 |
| fungi | so, turns out the reason i was having trouble pulling the stein signing key from the keyservers to attest is that the gnupg 2.1 dirmngr daemon had gone sideways. did a `pkill dirmngr` and then `gpg2 --recv-key 0xD943D5D270273ADA` worked fine | 19:44 |
| *** graphene has joined #openstack-infra | 19:45 | |
| fungi | just a tip in case anybody gets similarly stumped by vague "gpg: keyserver receive failed: No data" errors | 19:45 |
| mnaser | infra-root: i see vms going up | 19:50 |
| mnaser | are jobs going through well? | 19:50 |
| mnaser | i guess i can go to logtsash | 19:51 |
| *** jamesmcarthur has joined #openstack-infra | 19:51 | |
| mnaser | time to learn it for the first time again fro the millionth time | 19:51 |
| *** e0ne has quit IRC | 19:56 | |
| *** graphene has quit IRC | 20:00 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/featuretracker master: retire featuretracker https://review.openstack.org/597649 | 20:00 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Install limestone CA on hosts using openstacksdk https://review.openstack.org/593029 | 20:01 |
| *** slaweq has joined #openstack-infra | 20:01 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/puppet-featuretracker master: Retire puppet-featuretracker https://review.openstack.org/597650 | 20:01 |
| *** r-daneel has joined #openstack-infra | 20:01 | |
| *** graphene has joined #openstack-infra | 20:02 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Infra repo retirement of 18 repos - step 1 https://review.openstack.org/597370 | 20:02 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Infra repo retirement of 18 repos - step 3 https://review.openstack.org/597636 | 20:02 |
| AJaeger | clarkb: we can retire featuretracker as well, I added them ^ | 20:03 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove puppet-featuretracker https://review.openstack.org/597651 | 20:04 |
| AJaeger | dhellmann: if you run scripts over infra repos, you can exclude the 18 repos above now... | 20:06 |
| dhellmann | AJaeger : I was going to wait to do the infra repos until after the final release, and coordinate that with clarkb, so that should give us plenty of time to update the governance data | 20:07 |
| AJaeger | good... | 20:08 |
| AJaeger | dhellmann: zuul load is decreasing, I think you can start pushing out what you have queued... | 20:08 |
| dhellmann | looking at the test nodes graph on http://grafana.openstack.org/d/T6vSHcSik/zuul-status?orgId=1 I see the available count is still relatively low, but let me see how many we have for keystone and tripleo | 20:09 |
| AJaeger | config-core, here's one change for python3-first that is ready - please approve https://review.openstack.org/593264 | 20:09 |
| dhellmann | 46 for keystone 112 for tripleo | 20:10 |
| dhellmann | I'll go ahead and do keystone now | 20:10 |
| AJaeger | dhellmann: available means those that are started already, we can start another 150 afterwards... | 20:11 |
| dhellmann | oh, really? ok | 20:11 |
| AJaeger | dhellmann: compare "in use" now and earlier where we basically maxed out | 20:11 |
| dhellmann | then I'll go ahead and do both teams | 20:11 |
| AJaeger | dhellmann: do one - those are nodes, each jobs needs several nodes ;) | 20:11 |
| dhellmann | heh, ok | 20:12 |
| AJaeger | I mean each change needs... | 20:12 |
| dhellmann | just keystone it is then | 20:12 |
| AJaeger | dhellmann: and rest in an hour or two - then the short running jobs (pep, docs) are done | 20:12 |
| AJaeger | unless the zuul experts have other ideas ;) | 20:12 |
| dhellmann | yeah, I'll come back in a few hours and see how things look then | 20:12 |
| dhellmann | I usually try to submit them early in my day, before most folks have started proposing patches | 20:13 |
| AJaeger | or late in your day ;) | 20:13 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add system-config-run-eavesdrop https://review.openstack.org/597617 | 20:14 |
| AJaeger | dhellmann: node requests: 346 ;) | 20:15 |
| corvus | fungi: i think we were missing the bit that groups hosts by hostname ^ | 20:15 |
| pabelanger | looks like issues with packethost again | 20:15 |
| pabelanger | openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://packet.platform9.net/nova/v2.1/ba163363643541d88a737f8af23cfa7f/servers, {"forbidden": {"message": "Maximum number of ports exceeded", "code": 403}} | 20:15 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: remove job settings for keystone repositories https://review.openstack.org/597697 | 20:15 |
| pabelanger | clarkb: guess we leaked ports again^ | 20:15 |
| dhellmann | AJaeger : oof | 20:16 |
| clarkb | pabelanger: arg | 20:17 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Install limestone CA on hosts using openstacksdk https://review.openstack.org/593029 | 20:17 |
| clarkb | I wish I understood that problem | 20:17 |
| clarkb | mnaser: did you figure it out? | 20:17 |
| fungi | corvus: ahh! | 20:17 |
| AJaeger | dhellmann: now up to 540 or so - so, let's wait with tripelo ;) But zuul copies nicely with your stress tests ;) | 20:17 |
| mnaser | clarkb: it should be working? | 20:18 |
| dhellmann | AJaeger : yeah, I'll do tripleo in the morning :-) | 20:18 |
| AJaeger | dhellmann: I suggest last action for the day - they have very long running jobs | 20:19 |
| clarkb | mnaser: ya I think sjc1 is happy now, was wondinerg if you managed to get logstash to say so | 20:19 |
| dhellmann | AJaeger : oh, so late today would be better? | 20:20 |
| *** rh-jelabarre has quit IRC | 20:20 | |
| clarkb | AJaeger: on the beaker-* repos I don't even have +2 we may not have added members to its groups | 20:20 |
| AJaeger | dhellmann: that's my gut feeling. | 20:20 |
| dhellmann | makes sense | 20:20 |
| dhellmann | I'll give it a few hours and come back | 20:20 |
| *** mgagne has joined #openstack-infra | 20:20 | |
| clarkb | I'll swing back around on those when done with these other retirement changes and see if that is the cas | 20:20 |
| AJaeger | clarkb: you can give yourself superpowers ;) | 20:21 |
| AJaeger | clarkb: I used topic https://review.openstack.org/#/q/status:open+++topic:infra-retire | 20:21 |
| AJaeger | infra-root, if you want to review the complete retirement series for the 18 repos, please check topic infra-retire | 20:21 |
| *** kgiusti has left #openstack-infra | 20:23 | |
| *** priteau has quit IRC | 20:23 | |
| AJaeger | clarkb: thanks for reviewing. keep in mind that some repos have jobs refined, the first change needs to merge before we have "noop-jobs" so that Zuul gives a +1. | 20:23 |
| * AJaeger waves good night | 20:23 | |
| clarkb | oh the -1's said it couldn't merge because parent couldn't merge | 20:24 |
| clarkb | maybe my rechecks are too optomistic then | 20:24 |
| AJaeger | clarkb: some of them did... | 20:24 |
| AJaeger | dhellmann: I might beat you with lines removed - see https://review.openstack.org/#/c/597389/ - removes 3203137 lines ;) | 20:25 |
| dhellmann | AJaeger : cheater | 20:25 |
| AJaeger | ;) | 20:25 |
| *** rh-jelabarre has joined #openstack-infra | 20:26 | |
| *** dhill_ has quit IRC | 20:27 | |
| * corvus avoids syncing that change in gertty | 20:28 | |
| clarkb | AJaeger: comment on https://review.openstack.org/#/c/597636/2 | 20:30 |
| *** jamesmcarthur has quit IRC | 20:33 | |
| openstackgerrit | Merged openstack-infra/project-config master: Infra repo retirement of 18 repos - step 1 https://review.openstack.org/597370 | 20:37 |
| *** jcoufal has quit IRC | 20:38 | |
| *** hwoarang has quit IRC | 20:39 | |
| *** jamesmcarthur has joined #openstack-infra | 20:41 | |
| *** jamesmcarthur has quit IRC | 20:44 | |
| openstackgerrit | Slawek Kaplonski proposed openstack-infra/project-config master: Neutron failure rate - include TIMED_OUT jobs on graphs https://review.openstack.org/595763 | 20:45 |
| *** jamesmcarthur has joined #openstack-infra | 20:47 | |
| *** markvoelker has joined #openstack-infra | 20:48 | |
| *** jamesmcarthur has quit IRC | 20:51 | |
| *** jamesmcarthur has joined #openstack-infra | 20:51 | |
| clarkb | pabelanger: I see ~180 leaked ports in packethost. I'm running a show to get all their dns_names to double check they are all nodepool ports and will delete them | 20:52 |
| clarkb | will also look for leaked floating IPs | 20:52 |
| clarkb | running show on 180 ports is not terribly fast | 20:53 |
| *** hasharAway has quit IRC | 20:54 | |
| *** markvoelker has quit IRC | 20:55 | |
| *** bobh has quit IRC | 20:57 | |
| *** trown is now known as trown|outtypewww | 21:05 | |
| *** yamamoto has quit IRC | 21:08 | |
| *** ijw has joined #openstack-infra | 21:11 | |
| ianw | clarkb: sorry that popped up when i looked at the logs yesterday and i didn't look into it | 21:11 |
| clarkb | no problem, its been an ongoing issue with that cloud | 21:12 |
| clarkb | we've seen ti with other fip clouds too | 21:12 |
| ianw | pabelanger: did I manage to convince you about the ansible group v sudo on -> https://review.openstack.org/#/c/596630/ ? | 21:12 |
| *** ijw has quit IRC | 21:14 | |
| *** eharney has quit IRC | 21:15 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Create per-project ssh keys https://review.openstack.org/597711 | 21:15 |
| clarkb | all of the DOWN ports do look deletable so I am deleting them | 21:16 |
| ianw | also appreciate review of logrotate addition https://review.openstack.org/#/c/597333/ . on one hand i'm sure we could drag in some dependency for this sort of thing. otoh, it's simple and self-contained. and with the history of puppet modules, we don't exactly do a great job at keeping up with things when we do pull in such dependencies as libraries | 21:17 |
| *** wolverineav has quit IRC | 21:20 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add system-config-run-eavesdrop https://review.openstack.org/597617 | 21:21 |
| clarkb | ianw: left a comment, I may have missed that so didn't -1 but if I'm right should be addressed I think | 21:23 |
| ianw | clarkb: replied, i did leave that out on purpose such that it wasn't spending time checking for the package every time we installed a logrotate file | 21:25 |
| clarkb | ianw: ya documenting that it assumes the package is there is probably a good idea, where do we install it for bridge (where we are using it first?) | 21:26 |
| clarkb | maybe we need to add logrotate to the vars/ package lists in base-server role? | 21:26 |
| prometheanfire | http://logs.openstack.org/96/594496/1/check/build-wheel-mirror-centos-7/70ec312/job-output.txt.gz makes me think that somethings wrong with the image, or something | 21:27 |
| clarkb | http://logs.openstack.org/96/594496/1/check/build-wheel-mirror-centos-7/70ec312/job-output.txt.gz#_2018-08-29_19_16_29_664394 is the error | 21:29 |
| clarkb | I don't think its the image, the job should make that directory if it needs it I think | 21:29 |
| clarkb | oh but it never gets that far because http://logs.openstack.org/96/594496/1/check/build-wheel-mirror-centos-7/70ec312/job-output.txt.gz#_2018-08-29_19_16_22_186849 happens | 21:29 |
| clarkb | ianw: ^ y ou probably know things about that one | 21:29 |
| *** boden has quit IRC | 21:30 | |
| *** boden has joined #openstack-infra | 21:30 | |
| *** boden has joined #openstack-infra | 21:31 | |
| *** boden has quit IRC | 21:31 | |
| *** boden has joined #openstack-infra | 21:32 | |
| ianw | clarbk / prometheanfire: i do know about it, and even fixed it -> https://review.openstack.org/#/c/597375/ :) | 21:32 |
| prometheanfire | ianw: lol, followed | 21:32 |
| clarkb | ianw wins, I will review that now | 21:32 |
| *** boden has quit IRC | 21:32 | |
| ianw | prometheanfire: but, what didn't get last afternoon, is why this job is running in the check queue | 21:32 |
| *** boden has joined #openstack-infra | 21:32 | |
| ianw | do you know the history there? | 21:33 |
| prometheanfire | you want the review that triggered it? | 21:33 |
| *** boden has quit IRC | 21:33 | |
| *** boden has joined #openstack-infra | 21:33 | |
| *** boden has quit IRC | 21:34 | |
| ianw | prometheanfire: no, the bigger picture of why it's running. because it can't get access to the secrets to put stuff in AFS unless it runs as a post job | 21:34 |
| *** boden has joined #openstack-infra | 21:34 | |
| *** boden has quit IRC | 21:34 | |
| *** boden has joined #openstack-infra | 21:35 | |
| *** boden has quit IRC | 21:35 | |
| *** boden has joined #openstack-infra | 21:36 | |
| *** boden has quit IRC | 21:36 | |
| *** boden has joined #openstack-infra | 21:36 | |
| *** bobh has joined #openstack-infra | 21:37 | |
| *** boden has quit IRC | 21:37 | |
| *** boden has joined #openstack-infra | 21:37 | |
| *** boden has quit IRC | 21:38 | |
| *** boden has joined #openstack-infra | 21:38 | |
| ianw | ahhh ... " This also add support to test wheel-mirror builds." @ Ia6c1113d119b01e7189e4ada41e10865c4ec0066 | 21:38 |
| *** boden has quit IRC | 21:38 | |
| *** boden has joined #openstack-infra | 21:39 | |
| *** boden has quit IRC | 21:39 | |
| *** boden has joined #openstack-infra | 21:39 | |
| *** boden has quit IRC | 21:40 | |
| *** boden has joined #openstack-infra | 21:40 | |
| openstackgerrit | Merged openstack-infra/project-config master: Update kerberos client variable names https://review.openstack.org/597375 | 21:40 |
| *** boden has quit IRC | 21:41 | |
| *** boden has joined #openstack-infra | 21:41 | |
| *** boden has quit IRC | 21:41 | |
| *** boden has joined #openstack-infra | 21:42 | |
| *** boden has quit IRC | 21:42 | |
| *** boden has joined #openstack-infra | 21:43 | |
| *** boden has joined #openstack-infra | 21:43 | |
| *** boden has joined #openstack-infra | 21:44 | |
| *** mriedem is now known as mriedem_away | 21:44 | |
| *** boden has quit IRC | 21:45 | |
| *** boden has joined #openstack-infra | 21:45 | |
| *** boden has quit IRC | 21:45 | |
| ianw | prometheanfire / tonyb: ok, i see; the build jobs all have a different post run. so we don't need to install the afs client on the build job -- that will cut about 12 minutes off the job too. i'll sort something out today to clear it all up | 21:46 |
| *** boden has joined #openstack-infra | 21:46 | |
| ianw | but we'll also want to keep an eye on the post publish jobs too | 21:46 |
| tonyb | ianw: Thanks | 21:46 |
| *** boden has quit IRC | 21:46 | |
| tonyb | ianw: happy to watch the post pipeline | 21:47 |
| *** boden has joined #openstack-infra | 21:47 | |
| prometheanfire | ianw: thanks, sorry, got sidetracked by meeting | 21:47 |
| clarkb | ok packethost is doing more things now after deleting leaked ports, will have to watch and see if it continues | 21:48 |
| *** rcernin has joined #openstack-infra | 21:49 | |
| *** mriedem_away has quit IRC | 21:49 | |
| *** boden has quit IRC | 21:51 | |
| *** yamamoto has joined #openstack-infra | 21:53 | |
| clarkb | current errors in packethost appear to be timeouts, will watch and see fi that improves as it may just be related to image copies since we haven't booted much there recently | 21:55 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Updated Summit Json cache https://review.openstack.org/597718 | 21:56 |
| *** graphene has quit IRC | 21:59 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Updated Summit Json cache https://review.openstack.org/597718 | 21:59 |
| *** graphene has joined #openstack-infra | 22:01 | |
| *** jamesmcarthur has quit IRC | 22:01 | |
| clarkb | mnaser: so I don't forget probably the thing for us to follow up on is whether or not we are leaking volumes in sjc1 ya? | 22:03 |
| *** yamamoto has quit IRC | 22:03 | |
| clarkb | if I look at a current volume list all volumes are in use so that looks good right now | 22:03 |
| *** r-daneel_ has joined #openstack-infra | 22:05 | |
| *** r-daneel has quit IRC | 22:05 | |
| *** r-daneel_ is now known as r-daneel | 22:05 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add system-config-run-eavesdrop https://review.openstack.org/597617 | 22:09 |
| *** bobh has quit IRC | 22:12 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Create per-project ssh keys https://review.openstack.org/597711 | 22:14 |
| *** ijw has joined #openstack-infra | 22:17 | |
| *** slaweq has quit IRC | 22:17 | |
| *** gema has quit IRC | 22:18 | |
| *** threestrands has joined #openstack-infra | 22:19 | |
| *** threestrands has quit IRC | 22:19 | |
| *** tpsilva has quit IRC | 22:22 | |
| *** threestrands has joined #openstack-infra | 22:22 | |
| tonyb | ianw: Okay i'm back so when you're free I'm happy to look at whatever I can to help | 22:25 |
| clarkb | fungi: I just approved the etherpad upgrade | 22:26 |
| clarkb | commit message mentioned security fixes so figured we shouldn't wait on that | 22:26 |
| fungi | clarkb: thanks!!! | 22:31 |
| clarkb | fungi: also I was able to recv-key the Stein key | 22:32 |
| clarkb | so thats all signed now | 22:32 |
| fungi | awesome. i had no problem after i tried killing the lingering dirmngr process | 22:33 |
| fungi | so presumably my signature is showing up on it | 22:33 |
| fungi | i still have the key generation doc update half drafter | 22:33 |
| fungi | drafted | 22:33 |
| clarkb | the attestation docs don't change (of couse its 94% local anyway) | 22:34 |
| fungi | yup | 22:36 |
| fungi | though the default key length is 50% more with the gnupg on bionic | 22:37 |
| fungi | gnupg increased the rsa default from 2048 to 3072 bits | 22:37 |
| *** markvoelker has joined #openstack-infra | 22:46 | |
| clarkb | I'm looking at running cloud launcher on bridge.o.o, it needs to run with all-clouds.yaml yaml file. Does anyone know if that will pollute our global ansible cache with instance info for the cloud accounts we don't want normal ansible cron talking to? | 22:46 |
| * ianw googles the ascii shrug guy | 22:47 | |
| openstackgerrit | Merged openstack-infra/system-config master: Update etherpad.o.o to etherpad-lite 1.7.0 https://review.openstack.org/597544 | 22:47 |
| clarkb | also I think I can't run it out of bridge.yaml for a similar reason, so this needs to be its own cron and separte ansible invocation | 22:48 |
| clarkb | ya doesn't look like I can temporarily tell the ansible modules to use a different client config file, but still trying to confirm that | 22:49 |
| clarkb | I'll convert it to be a cron like we run already | 22:50 |
| clarkb | s/already/previously/ | 22:50 |
| *** diablo_rojo has quit IRC | 22:53 | |
| *** gfidente has quit IRC | 22:55 | |
| *** r-daneel has quit IRC | 22:56 | |
| *** rlandy is now known as rlandy|bbl | 22:57 | |
| clarkb | ianw: did you see the followup comment on https://review.openstack.org/#/c/597333/ ? I'm going to base cloud launcher on this chagne too so that I can set up a log rotation | 23:01 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Provide per-project ssh key to executor https://review.openstack.org/597727 | 23:01 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Run cloud launcher on bridge.o.o https://review.openstack.org/597728 | 23:03 |
| clarkb | I think that will do it but it depends on us getting all-clouds.yaml written so I will WIP until pabelanger has those patches up and I can depends on | 23:03 |
| *** dingyichen has quit IRC | 23:04 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add system-config-run-eavesdrop https://review.openstack.org/597617 | 23:05 |
| *** ijw has quit IRC | 23:06 | |
| clarkb | in interesting news, github is scanning openstack/requirements:global-requirements.txt for security vulnerabilities | 23:07 |
| clarkb | prometheanfire: ^ fyi I mentioned to keystone but pysam2 should be bumped to pysaml2 ~> 4.5.0 according to github | 23:07 |
| fungi | neat | 23:10 |
| *** slaweq has joined #openstack-infra | 23:11 | |
| *** markvoelker has quit IRC | 23:12 | |
| *** tosky has quit IRC | 23:13 | |
| mnaser | clarkb: yup it looks like its good so far | 23:13 |
| mnaser | clarkb: maybe we can bump to 50? | 23:13 |
| mnaser | if we can check that jobs are healthy | 23:13 |
| clarkb | mnaser: sure let me take a look at logstash | 23:14 |
| *** slaweq has quit IRC | 23:15 | |
| ianw | clarkb: yeah, adding to base seems good | 23:22 |
| clarkb | mnaser: filename:"job-output.txt" AND message:"Upload logs to log server" AND node_provider:"vexxhost-sjc1" AND build_status:"FAILURE" is my query, says 23 failures the vast majority are kata-runsh | 23:22 |
| clarkb | mnaser: I'm going to dig into failures now, I'm not up to speed on status of kata ci right now | 23:22 |
| mnaser | clarkb: but the openstack side seems okay i assume? | 23:22 |
| clarkb | ya only a small number of failures there and all appear to be incheck | 23:23 |
| clarkb | http://logs.openstack.org/22/622/1a8368f50e32fa7dba045d2a22e79a0a820763a4/third-party-check/kata-runsh/a8faaff/job-output.txt#_2018-08-29_20_34_22_940044 is legit failure in kata runsh | 23:23 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/project-config master: Bump vexxhost-sjc1 to 50 servers https://review.openstack.org/597731 | 23:24 |
| clarkb | http://logs.openstack.org/78/597678/1/check/build-openstack-sphinx-docs/8cfa182/job-output.txt#_2018-08-29_20_41_48_955862 also looks like a valid fail (incompatible requirements) | 23:24 |
| mnaser | clarkb: ^ | 23:24 |
| clarkb | http://logs.openstack.org/86/596986/1/check/tripleo-ci-centos-7-containers-multinode/7979318/job-output.txt#_2018-08-29_20_09_17_823672 is a tripleo timeout | 23:24 |
| clarkb | http://logs.openstack.org/21/597721/1/check/openstack-ansible-deploy-aio_distro_lxc-opensuse-423/d8c271b/job-output.txt#_2018-08-29_22_39_18_642829 looks like valid failure in ansible playbooks things | 23:25 |
| clarkb | so ya I think we can bump it up | 23:25 |
| clarkb | not seeing anything that says broken cloud other than the timeout could indicate a slow node? | 23:25 |
| mnaser | doubt its a slow node | 23:25 |
| mnaser | the cloud is honestly empty | 23:25 |
| mnaser | lol | 23:25 |
| mnaser | clarkb: but its possible that it was a cold cache? | 23:26 |
| clarkb | mnaser: or maybe you are using qemu instead of kvm? :P I seriously doubt that though | 23:26 |
| clarkb | cold caches could be at fault, and that will only get better as we use the cloud | 23:26 |
| mnaser | if that was the case i dont think the we'd have any jobs passing at all | 23:26 |
| mnaser | :P | 23:26 |
| clarkb | mnaser: it has been known to happen on more than one cloud because bioses disable vtx by default iirc | 23:27 |
| clarkb | but you seem to know how to run a cloud so I doubt that would hit you | 23:27 |
| fungi | mnaser definitely seems to know how to run a cloud | 23:28 |
| mnaser | :) | 23:29 |
| mnaser | and also double checked, it's on lol | 23:29 |
| *** holser_ has quit IRC | 23:29 | |
| clarkb | ianw: fungi: care to review https://review.openstack.org/597731 to kick it up to 50 instances in sjc1? | 23:30 |
| fungi | lgtm | 23:31 |
| fungi | i'm planning to stick a personal vm there pretty soon anyway | 23:31 |
| clarkb | danke | 23:31 |
| fungi | was interested to see that's the region my account defaulted to | 23:31 |
| mnaser | dat tripleo gate queue | 23:35 |
| ianw | clarkb: did we figure out the strangely odd quota results nodepool had? | 23:36 |
| clarkb | ianw: yes mnaser bumped the wrong region | 23:36 |
| mnaser | darn | 23:37 |
| clarkb | once the correct region was updated everything went happy as soon as shade caches expired | 23:37 |
| mnaser | i got worried with that comment out of context | 23:37 |
| mnaser | thinking i bumped ca-ymq-1 in my last change | 23:37 |
| mnaser | lol | 23:37 |
| ianw | ohh, cool, that neatly explains why the cache was never expiring then :) | 23:37 |
| ianw | it's hard to tell the difference between a cached invalid value and an invalid cache value | 23:38 |
| openstackgerrit | Merged openstack-infra/project-config master: Bump vexxhost-sjc1 to 50 servers https://review.openstack.org/597731 | 23:40 |
| *** sthussey has quit IRC | 23:44 | |
| clarkb | I believe http://status.openstack.org/elastic-recheck/#1384373 is all github | 23:54 |
| clarkb | which is painful | 23:54 |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add logrotate role and rotate ansible log files https://review.openstack.org/597333 | 23:54 |
| clarkb | was looking at e-r to see if it would show anything different about sjc1 | 23:54 |
| *** signed8bit is now known as signed8bit_Zzz | 23:55 | |
| clarkb | ianw: ^ lgtm now thanks | 23:56 |
| clarkb | fungi: that is curious stein key still not showing up on mit keyserver? | 23:58 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: remove job settings for tripleo repositories https://review.openstack.org/597822 | 23:59 |
| clarkb | ubuntu has it I guess syncing is just slow | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!