| *** wolverineav has joined #openstack-infra | 00:01 | |
| fungi | clarkb: if contents of /home/wildfly turn out to be important to maintain, we should either mount a cinder volume there or symlink into one | 00:01 |
|---|---|---|
| clarkb | fungi: ++ | 00:01 |
| clarkb | fungi: part of me is also curious why the sysv compat layer in systemd isn't working. But I probably don't want to know | 00:02 |
| fungi | don't stare directly into the systemd | 00:03 |
| *** pahuang has quit IRC | 00:03 | |
| fungi | when you look into the systemd long enough, it begins to look back | 00:03 |
| *** inc0 has quit IRC | 00:03 | |
| clarkb | supposedly if you try to start a systemd service and no unit file exists systemd checks to see if an init script lives in /etc/init.d/ with the same name | 00:04 |
| clarkb | but that doesn't seem tp have happened here and it just errors | 00:04 |
| *** sflanigan has quit IRC | 00:05 | |
| *** wolverineav has quit IRC | 00:06 | |
| clarkb | hrm I wonder if you have to reload systemd's config after installing the init.d file | 00:08 |
| clarkb | bkero: ^ do you know? | 00:08 |
| *** rbuzatu has quit IRC | 00:08 | |
| *** rbuzatu has joined #openstack-infra | 00:09 | |
| clarkb | yup that appears to be it, locally, if I systemctl daemon-reload its happy | 00:12 |
| *** wolverineav has joined #openstack-infra | 00:13 | |
| *** rbuzatu has quit IRC | 00:14 | |
| clarkb | that implies puppet isn't doing that before starting services? that seems annoying | 00:14 |
| clarkb | nibalizer: crinkle ^ any idea how puppet handles that? | 00:14 |
| *** wolverineav has quit IRC | 00:14 | |
| *** kgiusti has joined #openstack-infra | 00:14 | |
| *** kgiusti has left #openstack-infra | 00:15 | |
| *** Sukhdev has quit IRC | 00:16 | |
| clarkb | I have to say thats kind of annoying from a "compat" layer because its not compat unless you do the systemd specific thing first : | 00:17 |
| clarkb | er :( | 00:17 |
| *** pahuang has joined #openstack-infra | 00:20 | |
| EmilienM | I'm failing to find why npm can't be found when building centos7 images for nodepool | 00:20 |
| EmilienM | I tried to install in my centos7 VM and could run npm out of the box | 00:21 |
| EmilienM | while it's failing in our infra, when trying to deploy stackviz | 00:21 |
| *** sflanigan has joined #openstack-infra | 00:22 | |
| *** sflanigan has joined #openstack-infra | 00:22 | |
| clarkb | EmilienM: it looks like it is installing it 2016-11-22 15:59:41,627 INFO nodepool.image.build.centos-7: installing npm from stackviz | 00:22 |
| clarkb | EmilienM: but npm isn't ending up in the PATH maybe | 00:22 |
| *** wolverineav has joined #openstack-infra | 00:23 | |
| clarkb | EmilienM: what does which npm say on your centos7 box? | 00:23 |
| EmilienM | clarkb: /usr/bin/npm | 00:24 |
| EmilienM | that's why I'm confused | 00:24 |
| EmilienM | I'm downloading it from EPEL | 00:24 |
| EmilienM | version: 1:3.10.8-1.6.9.1.1.el7 | 00:24 |
| *** aeng has quit IRC | 00:25 | |
| EmilienM | and it works fine on fedora | 00:27 |
| *** wolverineav has quit IRC | 00:28 | |
| clarkb | EmilienM: I would do a local build with vm centos stackviz elements and set a breakpoint right before the npm install command to see what is goign on | 00:30 |
| clarkb | EmilienM: you can just add `bash` to that element prior to the command and it should drop you into an interactive shell | 00:31 |
| EmilienM | clarkb: i'll try that | 00:32 |
| *** wolverineav has joined #openstack-infra | 00:33 | |
| clarkb | EmilienM: you will need to set an elements path to include stackviz since its not part of dib itself | 00:34 |
| clarkb | but otherwise that should work well for debugging | 00:34 |
| EmilienM | yep it's in project-config iirc | 00:34 |
| clarkb | yup | 00:34 |
| clarkb | project-config/nodepool/elements | 00:34 |
| *** psilvad has joined #openstack-infra | 00:38 | |
| *** wolverineav has quit IRC | 00:38 | |
| *** Julien-zte has joined #openstack-infra | 00:43 | |
| *** tuanluong has joined #openstack-infra | 00:43 | |
| *** sdake has quit IRC | 00:43 | |
| *** ijw has quit IRC | 00:45 | |
| *** sdake has joined #openstack-infra | 00:45 | |
| clarkb | looking at https://github.com/biemond/biemond-wildfly/blob/master/templates/wildfly.service.erb more closely its running the init script via the unit file which is something systemd is already supposed to have support for :( | 00:45 |
| clarkb | I don't even | 00:45 |
| clarkb | jeblair: where am I on my systemd quota? | 00:46 |
| jeblair | clarkb: i only count 2, but i wasn't fully paying attention! | 00:46 |
| openstackgerrit | Adam Gandelman proposed openstack-infra/zuul: Re-enable test_success_pattern as test_success_url https://review.openstack.org/400455 | 00:48 |
| *** mmandel has joined #openstack-infra | 00:50 | |
| *** Julien-zte has quit IRC | 00:50 | |
| *** Julien-z_ has joined #openstack-infra | 00:50 | |
| jeblair | SotK, Zara: i deleted the orphan tasks (with no stories) from storyboard-dev (this is the case that shouldn't happen now that the cascade change merged) | 00:52 |
| clarkb | EmilienM: when you are done tracking down that npm thing maybe you know why systemd + puppet behave this way? | 00:52 |
| openstackgerrit | James E. Blair proposed openstack/boartty: Add beginning of support for boards/worklists https://review.openstack.org/401015 | 00:53 |
| EmilienM | clarkb: sorry, what is the context? | 00:53 |
| *** wolverineav has joined #openstack-infra | 00:54 | |
| *** mmandel has quit IRC | 00:54 | |
| ianw | guess what the recent centos build failures seem to be about -- not pulling in systemd, and something's changed so that it gets missed | 00:54 |
| openstackgerrit | Merged openstack/boartty: Add beginning of support for boards/worklists https://review.openstack.org/401015 | 00:54 |
| clarkb | EmilienM: putting a script in /etc/init.d/foo, tell puppet to ensure foo is enabled and started. This failes beacuse service foo start says there is no foo.service file, which is correct. Appears you need to do a systemctl daemon-reload so that systemd knows the sysv init script is there then it works | 00:55 |
| *** wolverineav has quit IRC | 00:55 | |
| clarkb | reading the systemd puppet provider it doesn't look like it has any concept of daemon-reload so not sure how any of this works with puppet (likely I am missing something important) | 00:56 |
| *** yolanda has joined #openstack-infra | 00:56 | |
| EmilienM | clarkb: there isn't, we have the same problem in tripleo and we shamelessly run an exec to reload it... :( | 00:56 |
| *** mriedem has joined #openstack-infra | 00:57 | |
| clarkb | EmilienM: basically have service resource require exec to daemon-reload? | 00:57 |
| clarkb | and exec refresh-only on file resource? | 00:57 |
| EmilienM | clarkb: something like this, yes | 00:57 |
| clarkb | jeblair: see puppet service management is silly too :) | 00:57 |
| clarkb | EmilienM: thanks | 00:57 |
| EmilienM | clarkb: yes ^ | 00:57 |
| EmilienM | it is... that's why I dislike doing it with puppet and fix packaging directly | 00:57 |
| *** mdrabe has quit IRC | 00:57 | |
| jeblair | we should write a... no. | 00:58 |
| clarkb | with that I think I need to step away from the computation device | 00:59 |
| *** tovin07 has joined #openstack-infra | 00:59 | |
| *** tovin07 has joined #openstack-infra | 01:00 | |
| *** ijw has joined #openstack-infra | 01:00 | |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul: Skip individual test_requirements tests https://review.openstack.org/401018 | 01:01 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul: Re-enable requirement-newer-than tests https://review.openstack.org/401019 | 01:01 |
| EmilienM | jeblair: lol, say it! | 01:01 |
| clarkb | now the really scary question is does puppet enterprise fix that/ | 01:01 |
| clarkb | I don't want to know | 01:01 |
| *** tovin07 has quit IRC | 01:01 | |
| *** Julien-zte has joined #openstack-infra | 01:01 | |
| *** tovin07 has joined #openstack-infra | 01:01 | |
| *** tovin07 has joined #openstack-infra | 01:02 | |
| *** Julien-z_ has quit IRC | 01:02 | |
| EmilienM | clarkb: I don't see how it would fix that | 01:02 |
| clarkb | EmilienM: the systemd service provider should issue a daemon-reload if the unit is not listed but otherwise is required to accomplish some action | 01:03 |
| clarkb | EmilienM: then only fail if it still doesn't exist afterwards | 01:03 |
| EmilienM | clarkb: honestly, it should be fixed in packaging directly | 01:04 |
| *** wolverineav has joined #openstack-infra | 01:04 | |
| clarkb | EmilienM: no | 01:04 |
| clarkb | EmilienM: beacuse you can use systemd without packages | 01:04 |
| EmilienM | managing init scripts with puppet is terrible | 01:04 |
| clarkb | regardless | 01:04 |
| EmilienM | (we do it also) | 01:04 |
| clarkb | its a perfectly valid thing to do | 01:04 |
| clarkb | there is nothing about service management that implies packaging | 01:04 |
| clarkb | (I think having packages that do the right thing is also a great idea, but it shouldn't be necessary) | 01:05 |
| *** sdake has quit IRC | 01:05 | |
| clarkb | there are many reasons for this one of which is the git server in centos has a broken unit file | 01:05 |
| clarkb | so you have to provide your own and upstream wouldn't patch it because stable or something | 01:06 |
| clarkb | but maybe that has since changed | 01:06 |
| *** sdake has joined #openstack-infra | 01:06 | |
| clarkb | but systemd also allows for defniing user services and that doesn't fit with the system packaging model | 01:06 |
| jeblair | i just discovered '.mode line' in sqlite3. | 01:08 |
| *** wolverineav has quit IRC | 01:09 | |
| jeblair | you can actually see the column names in the results of a select | 01:09 |
| *** pvaneck has quit IRC | 01:09 | |
| ianychoi | clarkb, thanks a lot for helping upgrade translate-dev.o.o. Now I acknowledge systemd problem but I do not have good knowledge on dealing with this.. | 01:10 |
| *** wolverineav has joined #openstack-infra | 01:14 | |
| openstackgerrit | James E. Blair proposed openstack/boartty: Fix board sync errors https://review.openstack.org/401023 | 01:17 |
| openstackgerrit | Merged openstack/boartty: Fix board sync errors https://review.openstack.org/401023 | 01:18 |
| *** pahuang has quit IRC | 01:18 | |
| ianychoi | clarkb, ah and have a good thanksgiving holiday! :) | 01:18 |
| *** wolverineav has quit IRC | 01:18 | |
| *** zhurong has joined #openstack-infra | 01:19 | |
| *** pahuang has joined #openstack-infra | 01:21 | |
| *** newmember has quit IRC | 01:23 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Stop using documentation network range https://review.openstack.org/395529 | 01:24 |
| *** wolverineav has joined #openstack-infra | 01:24 | |
| *** sdake has quit IRC | 01:27 | |
| pabelanger | jeblair: just seen your comment now after I left a comment on 400998. I think we might have some issues with nodepool-launcher service being enabled on reboot | 01:27 |
| pabelanger | but haven't tested yet | 01:27 |
| pabelanger | jeblair: but, we don't need common. And try your approach | 01:28 |
| jeblair | pabelanger: either way; was just thinking it might get us there faster | 01:28 |
| jeblair | (i'm sure we'll end up in the same place) | 01:28 |
| *** wolverineav has quit IRC | 01:28 | |
| *** yanyanhu has joined #openstack-infra | 01:29 | |
| *** sdake has joined #openstack-infra | 01:29 | |
| *** wolverineav has joined #openstack-infra | 01:30 | |
| *** Alagar has joined #openstack-infra | 01:31 | |
| openstackgerrit | James E. Blair proposed openstack/boartty: Support search by story number https://review.openstack.org/401026 | 01:31 |
| pabelanger | jeblair: Yup, I'm happy to give your patches a try. And iterate forward if issues | 01:31 |
| openstackgerrit | Merged openstack/boartty: Support search by story number https://review.openstack.org/401026 | 01:32 |
| *** gildub has quit IRC | 01:33 | |
| *** dave-mccowan has joined #openstack-infra | 01:34 | |
| *** wolverineav has quit IRC | 01:35 | |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul: Re-enable requirement-newer-than tests https://review.openstack.org/401019 | 01:37 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul: Re-enable requirement-older-than tests https://review.openstack.org/401027 | 01:37 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul: Re-enable requirement-username tests https://review.openstack.org/401028 | 01:37 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul: Re-enable requirement-email tests https://review.openstack.org/401029 | 01:37 |
| openstackgerrit | James E. Blair proposed openstack/boartty: Support search by tag https://review.openstack.org/401030 | 01:40 |
| openstackgerrit | Merged openstack/boartty: Support search by tag https://review.openstack.org/401030 | 01:40 |
| jeblair | okay, that ^ will let me see all the stories tagged 'zuulv3' | 01:40 |
| *** wolverineav has joined #openstack-infra | 01:41 | |
| jeblair | (and more importantly, make a dashboard out of it) | 01:41 |
| *** wolverineav has quit IRC | 01:41 | |
| *** mtanino has joined #openstack-infra | 01:43 | |
| *** kaisers has quit IRC | 01:43 | |
| *** kaisers has joined #openstack-infra | 01:44 | |
| *** vaidy has joined #openstack-infra | 01:46 | |
| *** isviridov_away has joined #openstack-infra | 01:47 | |
| *** sdake has quit IRC | 01:50 | |
| *** sdake has joined #openstack-infra | 01:53 | |
| *** Goneri has quit IRC | 02:00 | |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Revise ACLs for networking-[onos|l2gw] projects https://review.openstack.org/399798 | 02:00 |
| *** EricGonczer_ has quit IRC | 02:09 | |
| *** Goneri has joined #openstack-infra | 02:10 | |
| *** EricGonczer_ has joined #openstack-infra | 02:10 | |
| *** dave-mcc_ has joined #openstack-infra | 02:11 | |
| *** dave-mccowan has quit IRC | 02:12 | |
| *** dave-mccowan has joined #openstack-infra | 02:13 | |
| *** sdake has quit IRC | 02:14 | |
| *** sdake has joined #openstack-infra | 02:15 | |
| *** dave-mcc_ has quit IRC | 02:16 | |
| *** psilvad has quit IRC | 02:16 | |
| *** dave-mccowan has quit IRC | 02:18 | |
| *** tqtran has quit IRC | 02:21 | |
| *** sflanigan has quit IRC | 02:23 | |
| *** gongysh has joined #openstack-infra | 02:24 | |
| *** fguillot has quit IRC | 02:27 | |
| *** jascott1 has quit IRC | 02:28 | |
| *** jamielennox is now known as jamielennox|away | 02:30 | |
| *** thorst has joined #openstack-infra | 02:32 | |
| *** thorst has quit IRC | 02:32 | |
| *** thorst has joined #openstack-infra | 02:33 | |
| *** aeng has joined #openstack-infra | 02:33 | |
| *** sflanigan has joined #openstack-infra | 02:35 | |
| *** sdake has quit IRC | 02:36 | |
| *** sdake has joined #openstack-infra | 02:37 | |
| *** dave-mccowan has joined #openstack-infra | 02:39 | |
| *** thorst has quit IRC | 02:40 | |
| *** thorst has joined #openstack-infra | 02:41 | |
| *** wolverineav has joined #openstack-infra | 02:42 | |
| *** hurgleburgler has joined #openstack-infra | 02:42 | |
| bkero | clarkb: you need to systemd daemon-reload | 02:43 |
| *** thorst has quit IRC | 02:45 | |
| *** asettle has joined #openstack-infra | 02:45 | |
| *** mriedem has quit IRC | 02:46 | |
| *** wolverineav has quit IRC | 02:46 | |
| *** dave-mccowan has quit IRC | 02:48 | |
| *** asettle has quit IRC | 02:50 | |
| *** yamamoto has joined #openstack-infra | 02:51 | |
| *** wolverineav has joined #openstack-infra | 02:52 | |
| *** Alagar has quit IRC | 02:52 | |
| *** yamamoto has quit IRC | 02:52 | |
| *** Alagar has joined #openstack-infra | 02:53 | |
| *** Alagar has quit IRC | 02:55 | |
| *** wolverineav has quit IRC | 02:56 | |
| *** sdake has quit IRC | 02:57 | |
| *** Alagar has joined #openstack-infra | 02:58 | |
| *** jamielennox|away is now known as jamielennox | 02:59 | |
| *** sdake has joined #openstack-infra | 02:59 | |
| *** mmandel has joined #openstack-infra | 03:02 | |
| *** wolverineav has joined #openstack-infra | 03:02 | |
| *** adriant has quit IRC | 03:03 | |
| *** zhurong has quit IRC | 03:04 | |
| *** wolverineav has quit IRC | 03:06 | |
| *** mmandel has quit IRC | 03:06 | |
| *** zhurong has joined #openstack-infra | 03:07 | |
| *** thorst has joined #openstack-infra | 03:09 | |
| *** thorst has quit IRC | 03:10 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Trace package install in package-installs-v2 https://review.openstack.org/401048 | 03:11 |
| *** Nrk9t1x has joined #openstack-infra | 03:12 | |
| *** wolverineav has joined #openstack-infra | 03:12 | |
| *** Nrk9t1x has quit IRC | 03:15 | |
| *** wolverineav has quit IRC | 03:16 | |
| *** Nrk9t1x has joined #openstack-infra | 03:17 | |
| *** sdake has quit IRC | 03:19 | |
| *** sdake has joined #openstack-infra | 03:21 | |
| *** wolverineav has joined #openstack-infra | 03:22 | |
| *** cschwede has quit IRC | 03:23 | |
| *** wolverineav has quit IRC | 03:26 | |
| *** chandankumar has joined #openstack-infra | 03:27 | |
| *** cschwede has joined #openstack-infra | 03:28 | |
| *** yamahata has quit IRC | 03:29 | |
| *** aeng has quit IRC | 03:31 | |
| *** wolverineav has joined #openstack-infra | 03:31 | |
| *** aeng has joined #openstack-infra | 03:32 | |
| *** sflanigan has quit IRC | 03:32 | |
| *** liusheng has quit IRC | 03:33 | |
| *** zhurong has quit IRC | 03:35 | |
| *** zhurong has joined #openstack-infra | 03:36 | |
| *** thorst has joined #openstack-infra | 03:36 | |
| *** csomerville has quit IRC | 03:36 | |
| *** wolverineav has quit IRC | 03:36 | |
| *** thorst has quit IRC | 03:37 | |
| *** adriant has joined #openstack-infra | 03:37 | |
| *** sdake has quit IRC | 03:41 | |
| *** wolverineav has joined #openstack-infra | 03:41 | |
| *** links has joined #openstack-infra | 03:42 | |
| *** sdake has joined #openstack-infra | 03:43 | |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul: Re-enable requirement-older-than tests https://review.openstack.org/401027 | 03:43 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul: Re-enable requirement-username tests https://review.openstack.org/401028 | 03:43 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul: Re-enable requirement-email tests https://review.openstack.org/401029 | 03:43 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul: Re-enable requirement-newer-than tests https://review.openstack.org/401019 | 03:43 |
| *** EricGonczer_ has quit IRC | 03:45 | |
| *** wolverineav has quit IRC | 03:46 | |
| *** armax has quit IRC | 03:46 | |
| *** sflanigan has joined #openstack-infra | 03:49 | |
| *** wolverineav has joined #openstack-infra | 03:51 | |
| *** psachin has joined #openstack-infra | 03:52 | |
| *** pcrews has quit IRC | 03:53 | |
| *** wolverineav has quit IRC | 03:56 | |
| *** ijw has quit IRC | 03:56 | |
| *** wolverineav has joined #openstack-infra | 04:01 | |
| *** markvoelker has quit IRC | 04:02 | |
| *** sdake has quit IRC | 04:03 | |
| *** sdake has joined #openstack-infra | 04:05 | |
| *** wolverineav has quit IRC | 04:06 | |
| *** flwang1 has quit IRC | 04:08 | |
| *** gildub has joined #openstack-infra | 04:10 | |
| *** jgriffith is now known as jgriffith_away | 04:10 | |
| *** Julien-zte has quit IRC | 04:11 | |
| *** Goneri has quit IRC | 04:11 | |
| *** wolverineav has joined #openstack-infra | 04:11 | |
| *** wolverineav has quit IRC | 04:12 | |
| *** esikachev has joined #openstack-infra | 04:12 | |
| *** Julien-zte has joined #openstack-infra | 04:12 | |
| *** esikachev has quit IRC | 04:16 | |
| *** aeng is now known as aeng_mtg | 04:18 | |
| *** wolverineav has joined #openstack-infra | 04:21 | |
| *** yamamoto has joined #openstack-infra | 04:21 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Trace package install in package-installs-v2 https://review.openstack.org/401048 | 04:22 |
| *** yamamoto has quit IRC | 04:23 | |
| *** Alagar has quit IRC | 04:24 | |
| *** ijw has joined #openstack-infra | 04:24 | |
| *** sdake has quit IRC | 04:25 | |
| *** wolverineav has quit IRC | 04:25 | |
| *** sdake has joined #openstack-infra | 04:28 | |
| *** ijw has quit IRC | 04:29 | |
| *** yamahata has joined #openstack-infra | 04:29 | |
| *** wolverineav has joined #openstack-infra | 04:31 | |
| *** wolverineav has quit IRC | 04:35 | |
| *** mhayden has quit IRC | 04:38 | |
| *** wolverineav has joined #openstack-infra | 04:40 | |
| *** yamahata has quit IRC | 04:43 | |
| *** Alagar has joined #openstack-infra | 04:44 | |
| *** tlian has quit IRC | 04:44 | |
| *** wolverineav has quit IRC | 04:45 | |
| openstackgerrit | Khai Do proposed openstack-infra/infra-specs: Gerrit upgrade https://review.openstack.org/388190 | 04:47 |
| *** sdake has quit IRC | 04:49 | |
| *** mhayden has joined #openstack-infra | 04:50 | |
| *** wolverineav has joined #openstack-infra | 04:50 | |
| *** EricGonczer_ has joined #openstack-infra | 04:51 | |
| openstackgerrit | Aparna proposed openstack/diskimage-builder: Update hpssacli to ssacli in proliant-tools element https://review.openstack.org/396504 | 04:52 |
| *** sdake has joined #openstack-infra | 04:53 | |
| *** yamamoto has joined #openstack-infra | 04:54 | |
| *** yamamoto has quit IRC | 04:54 | |
| *** wolverineav has quit IRC | 04:55 | |
| *** Nrk9t1x has quit IRC | 04:57 | |
| *** EricGonczer_ has quit IRC | 04:58 | |
| *** rbuzatu has joined #openstack-infra | 04:58 | |
| *** wolverineav has joined #openstack-infra | 05:00 | |
| *** jogo has joined #openstack-infra | 05:02 | |
| *** jogo has quit IRC | 05:02 | |
| *** jogo has joined #openstack-infra | 05:02 | |
| *** rbuzatu has quit IRC | 05:02 | |
| *** pahuang has quit IRC | 05:03 | |
| *** wolverin_ has joined #openstack-infra | 05:03 | |
| openstackgerrit | Renat Akhmerov proposed openstack-infra/project-config: Add mistral gate for mysql unit tests https://review.openstack.org/400535 | 05:04 |
| *** sree has joined #openstack-infra | 05:04 | |
| *** wolverineav has quit IRC | 05:05 | |
| *** sflanigan has quit IRC | 05:05 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Trace package install in package-installs-v2 https://review.openstack.org/401048 | 05:05 |
| *** pahuang has joined #openstack-infra | 05:05 | |
| *** yamahata has joined #openstack-infra | 05:05 | |
| *** Jeffrey4l has quit IRC | 05:06 | |
| *** newmember has joined #openstack-infra | 05:08 | |
| *** aeng_mtg has quit IRC | 05:08 | |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul: Re-enable requirement-older-than tests https://review.openstack.org/401027 | 05:09 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul: Re-enable requirement-username tests https://review.openstack.org/401028 | 05:09 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul: Re-enable requirement-email tests https://review.openstack.org/401029 | 05:09 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul: Re-enable requirement-newer-than tests https://review.openstack.org/401019 | 05:09 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul: Re-enable requirement vote tests https://review.openstack.org/401061 | 05:09 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul: Re-enable requirement status tests https://review.openstack.org/401062 | 05:09 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul: Re-enable requirement reject tests https://review.openstack.org/401063 | 05:09 |
| *** dingyichen has quit IRC | 05:10 | |
| *** wolverin_ has quit IRC | 05:12 | |
| *** aeng_mtg has joined #openstack-infra | 05:13 | |
| *** pgadiya has joined #openstack-infra | 05:13 | |
| *** dingyichen has joined #openstack-infra | 05:14 | |
| *** pcrews has joined #openstack-infra | 05:16 | |
| *** Alagar has quit IRC | 05:16 | |
| *** sflanigan has joined #openstack-infra | 05:17 | |
| *** sflanigan has joined #openstack-infra | 05:17 | |
| *** Alagar has joined #openstack-infra | 05:20 | |
| *** cshastri has joined #openstack-infra | 05:20 | |
| *** wolverineav has joined #openstack-infra | 05:22 | |
| *** jogo has quit IRC | 05:22 | |
| *** senk has joined #openstack-infra | 05:24 | |
| *** wolverineav has quit IRC | 05:27 | |
| *** senk_ has joined #openstack-infra | 05:28 | |
| *** vikrant has joined #openstack-infra | 05:30 | |
| *** sandanar has joined #openstack-infra | 05:30 | |
| *** senk has quit IRC | 05:31 | |
| *** sdake_ has joined #openstack-infra | 05:31 | |
| *** sandanar_ has joined #openstack-infra | 05:34 | |
| *** sdake has quit IRC | 05:35 | |
| *** senk has joined #openstack-infra | 05:37 | |
| *** hurgleburgler has quit IRC | 05:38 | |
| *** sandanar has quit IRC | 05:38 | |
| *** senk_ has quit IRC | 05:40 | |
| *** wolverineav has joined #openstack-infra | 05:42 | |
| *** wolverineav has quit IRC | 05:46 | |
| *** gongysh has quit IRC | 05:49 | |
| *** wolverineav has joined #openstack-infra | 05:52 | |
| *** Alagar has quit IRC | 05:52 | |
| *** wolverineav has quit IRC | 05:56 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Trace package install in package-installs-v2 https://review.openstack.org/401048 | 05:58 |
| *** chandankumar has quit IRC | 05:59 | |
| *** wolverineav has joined #openstack-infra | 06:01 | |
| *** adriant has quit IRC | 06:02 | |
| *** yamamoto has joined #openstack-infra | 06:02 | |
| *** markvoelker has joined #openstack-infra | 06:02 | |
| *** wolverineav has quit IRC | 06:06 | |
| *** liusheng has joined #openstack-infra | 06:07 | |
| *** GnomeKing has joined #openstack-infra | 06:07 | |
| *** markvoelker has quit IRC | 06:08 | |
| *** GnomeKing is now known as Bobba | 06:10 | |
| Bobba | Please could someone add os-xenapi-core as a member to os-xenapi-release (https://review.openstack.org/#/admin/groups/1650,members) thanks! | 06:10 |
| *** wolverineav has joined #openstack-infra | 06:11 | |
| *** Alagar has joined #openstack-infra | 06:12 | |
| *** esikachev has joined #openstack-infra | 06:12 | |
| *** wolverineav has quit IRC | 06:16 | |
| *** Nrk9t1x has joined #openstack-infra | 06:17 | |
| *** esikachev has quit IRC | 06:17 | |
| *** aeng_mtg is now known as aeng_afk | 06:18 | |
| *** jogo has joined #openstack-infra | 06:20 | |
| *** jogo has quit IRC | 06:20 | |
| *** jogo has joined #openstack-infra | 06:20 | |
| *** wolverineav has joined #openstack-infra | 06:21 | |
| *** wolverineav has quit IRC | 06:26 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Trace package install in package-installs-v2 https://review.openstack.org/401048 | 06:30 |
| *** wolverineav has joined #openstack-infra | 06:31 | |
| *** tojuvone has quit IRC | 06:34 | |
| *** yamamoto has quit IRC | 06:34 | |
| *** wolverineav has quit IRC | 06:36 | |
| *** abregman has joined #openstack-infra | 06:36 | |
| *** nadya has joined #openstack-infra | 06:37 | |
| *** nadya has quit IRC | 06:38 | |
| *** esikachev has joined #openstack-infra | 06:38 | |
| *** wolverineav has joined #openstack-infra | 06:41 | |
| *** yamamoto has joined #openstack-infra | 06:43 | |
| *** yamamoto has quit IRC | 06:43 | |
| *** oanson has joined #openstack-infra | 06:43 | |
| *** wolverin_ has joined #openstack-infra | 06:44 | |
| *** hrubi has quit IRC | 06:44 | |
| *** wolverineav has quit IRC | 06:45 | |
| *** senk has quit IRC | 06:46 | |
| *** kzaitsev_mb has quit IRC | 06:48 | |
| *** wolverin_ has quit IRC | 06:49 | |
| *** yamamoto has joined #openstack-infra | 06:51 | |
| *** yamamoto has quit IRC | 06:57 | |
| *** wolverineav has joined #openstack-infra | 07:02 | |
| *** wolverineav has quit IRC | 07:06 | |
| *** gildub has quit IRC | 07:07 | |
| *** qwertyco has joined #openstack-infra | 07:08 | |
| *** tjones has quit IRC | 07:09 | |
| *** nadya has joined #openstack-infra | 07:12 | |
| *** Guest2_ has joined #openstack-infra | 07:12 | |
| *** nadya has quit IRC | 07:12 | |
| *** tesseract has joined #openstack-infra | 07:12 | |
| *** Alagar has quit IRC | 07:12 | |
| *** Alagar has joined #openstack-infra | 07:12 | |
| *** tesseract is now known as Guest59528 | 07:13 | |
| *** senk has joined #openstack-infra | 07:14 | |
| *** wolverineav has joined #openstack-infra | 07:16 | |
| *** ccamacho has joined #openstack-infra | 07:17 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Trace package install in package-installs-v2 https://review.openstack.org/401048 | 07:18 |
| *** mmandel has joined #openstack-infra | 07:18 | |
| *** wolverineav has quit IRC | 07:20 | |
| *** mmandel has quit IRC | 07:22 | |
| *** e0ne has joined #openstack-infra | 07:24 | |
| *** isaacb has joined #openstack-infra | 07:25 | |
| *** wolverineav has joined #openstack-infra | 07:26 | |
| *** nadya has joined #openstack-infra | 07:27 | |
| *** gouthamr has joined #openstack-infra | 07:30 | |
| *** chandankumar has joined #openstack-infra | 07:31 | |
| *** wolverineav has quit IRC | 07:32 | |
| *** dingyichen has quit IRC | 07:35 | |
| *** yamahata has quit IRC | 07:39 | |
| *** rbuzatu_ has joined #openstack-infra | 07:42 | |
| *** drifterza has joined #openstack-infra | 07:44 | |
| *** e0ne has quit IRC | 07:45 | |
| *** jascott1 has joined #openstack-infra | 07:47 | |
| *** s-shiono has joined #openstack-infra | 07:47 | |
| *** Hal1 has joined #openstack-infra | 07:51 | |
| *** wolverineav has joined #openstack-infra | 07:52 | |
| *** matrohon has joined #openstack-infra | 07:56 | |
| AJaeger | ianw: could you review https://review.openstack.org/392940 and https://review.openstack.org/399667 , please? | 07:57 |
| *** wolverineav has quit IRC | 07:57 | |
| *** s-shiono has quit IRC | 07:58 | |
| *** wolverineav has joined #openstack-infra | 07:59 | |
| *** nherciu has joined #openstack-infra | 08:01 | |
| *** wolverineav has quit IRC | 08:04 | |
| openstackgerrit | Zhipeng Huang proposed openstack-infra/project-config: Rename project Nomad to Cyborg based upon team voting and consensus at https://etherpad.openstack.org/p/nomad-ocata-design-session https://review.openstack.org/401115 | 08:05 |
| *** ad_rien_ has joined #openstack-infra | 08:06 | |
| *** yamamoto has joined #openstack-infra | 08:08 | |
| AJaeger | Why didn't they come up with that two days ago? ;( ^ | 08:11 |
| *** dimtruck is now known as zz_dimtruck | 08:11 | |
| *** rbuzatu_ has quit IRC | 08:11 | |
| *** berendt has joined #openstack-infra | 08:12 | |
| *** Guest2_ has quit IRC | 08:14 | |
| *** ihrachys has joined #openstack-infra | 08:14 | |
| *** s-shiono has joined #openstack-infra | 08:16 | |
| *** yamamoto has quit IRC | 08:19 | |
| *** maishsk has joined #openstack-infra | 08:19 | |
| *** maishsk has quit IRC | 08:19 | |
| *** nmagnezi has joined #openstack-infra | 08:20 | |
| *** senk has quit IRC | 08:20 | |
| *** ralonsoh has joined #openstack-infra | 08:20 | |
| *** jpena|off is now known as jpena | 08:20 | |
| *** derekjhyang has quit IRC | 08:21 | |
| *** senk has joined #openstack-infra | 08:21 | |
| *** wolverineav has joined #openstack-infra | 08:21 | |
| openstackgerrit | Rob Cresswell proposed openstack-infra/project-config: Update Horizons selenium-headless test https://review.openstack.org/399868 | 08:21 |
| openstackgerrit | Merged openstack-infra/system-config: Check number of meetbot channels https://review.openstack.org/392940 | 08:22 |
| *** yamamoto has joined #openstack-infra | 08:23 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Trace package install in package-installs-v2 https://review.openstack.org/401048 | 08:24 |
| *** senk_ has joined #openstack-infra | 08:25 | |
| openstackgerrit | Rob Cresswell proposed openstack-infra/project-config: Update Horizons selenium-headless test https://review.openstack.org/399868 | 08:26 |
| *** wolverineav has quit IRC | 08:26 | |
| openstackgerrit | Rob Cresswell proposed openstack-infra/project-config: Update Horizons selenium-headless test https://review.openstack.org/399868 | 08:26 |
| robcresswell | Sorry for the noise, still failing to wrap my head around the various yaml changes. | 08:27 |
| *** dtantsur|afk is now known as dtantsur | 08:27 | |
| *** e0ne has joined #openstack-infra | 08:28 | |
| *** senk has quit IRC | 08:28 | |
| *** yamamoto has quit IRC | 08:29 | |
| *** jaosorior has joined #openstack-infra | 08:29 | |
| *** yamamoto has joined #openstack-infra | 08:30 | |
| *** shardy has joined #openstack-infra | 08:30 | |
| *** jpich has joined #openstack-infra | 08:30 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Run checklang on all branches https://review.openstack.org/399667 | 08:31 |
| openstackgerrit | Zhipeng Huang proposed openstack-infra/project-config: Rename project Nomad to Cyborg based upon team voting and consensus https://review.openstack.org/401115 | 08:33 |
| *** ssbarnea has joined #openstack-infra | 08:33 | |
| *** e0ne has quit IRC | 08:34 | |
| *** yamamoto has quit IRC | 08:35 | |
| *** dtardivel has joined #openstack-infra | 08:41 | |
| *** amoralej|off is now known as amoralej | 08:43 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: WIP TLS everywhere job https://review.openstack.org/391738 | 08:43 |
| openstackgerrit | Rob Cresswell proposed openstack-infra/project-config: Update Horizons selenium-headless test https://review.openstack.org/399868 | 08:44 |
| *** ihrachys has quit IRC | 08:45 | |
| *** isaacb has quit IRC | 08:45 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Trace package install in package-installs-v2 https://review.openstack.org/401048 | 08:47 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Special case dib-python in dib-lint https://review.openstack.org/401127 | 08:47 |
| *** Alagar has quit IRC | 08:48 | |
| *** makowals has quit IRC | 08:49 | |
| *** makowals_ has joined #openstack-infra | 08:49 | |
| *** yuval has joined #openstack-infra | 08:51 | |
| *** rossella__ has joined #openstack-infra | 08:57 | |
| *** isaacb has joined #openstack-infra | 08:57 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Special case dib-python in dib-lint https://review.openstack.org/401127 | 08:59 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Trace package install in package-installs-v2 https://review.openstack.org/401048 | 08:59 |
| AJaeger | ianw, sorry, founda bug in 399667 and fixed it - could you review again, please? Thanks! | 08:59 |
| *** zzzeek has quit IRC | 09:00 | |
| *** sree has quit IRC | 09:01 | |
| *** gongysh has joined #openstack-infra | 09:01 | |
| *** sree has joined #openstack-infra | 09:01 | |
| *** arxcruz has joined #openstack-infra | 09:01 | |
| *** zzzeek has joined #openstack-infra | 09:02 | |
| *** sree is now known as sreeram_v | 09:04 | |
| *** gouthamr has quit IRC | 09:04 | |
| *** asettle has joined #openstack-infra | 09:05 | |
| *** sreeram_v is now known as sree | 09:05 | |
| *** martinkopec has joined #openstack-infra | 09:08 | |
| *** sree is now known as sreeram_v | 09:10 | |
| *** rbuzatu has joined #openstack-infra | 09:11 | |
| *** ggnel_t has quit IRC | 09:17 | |
| *** tpsilva has joined #openstack-infra | 09:19 | |
| *** ihrachys has joined #openstack-infra | 09:20 | |
| *** ihrachys has quit IRC | 09:20 | |
| *** ihrachys has joined #openstack-infra | 09:20 | |
| openstackgerrit | Michal Dulko proposed openstack-infra/project-config: Fix Cinder's multibackend ZeroMQ job https://review.openstack.org/401150 | 09:21 |
| dulek | AJaeger, yolanda: Turns out the zmq job merged yesterday was using old envvar, that wasn't really working. Patch above should fix that. ^ | 09:22 |
| *** rbuzatu has quit IRC | 09:24 | |
| *** rbuzatu has joined #openstack-infra | 09:24 | |
| *** _degorenko|afk is now known as degorenko | 09:25 | |
| *** Hal1 has quit IRC | 09:25 | |
| *** sshnaidm|away is now known as sshnaidm | 09:26 | |
| *** wolverineav has joined #openstack-infra | 09:29 | |
| *** dizquierdo has joined #openstack-infra | 09:30 | |
| *** sreeram_v is now known as sree | 09:30 | |
| *** rossella__ has quit IRC | 09:31 | |
| *** lucas-afk is now known as lucasagomes | 09:31 | |
| *** rossella__ has joined #openstack-infra | 09:31 | |
| *** sree is now known as sreeram_v | 09:32 | |
| *** ggnel_t has joined #openstack-infra | 09:32 | |
| *** markvoelker has joined #openstack-infra | 09:32 | |
| *** lin_yang has quit IRC | 09:33 | |
| *** john-davidge has joined #openstack-infra | 09:34 | |
| *** Julien-zte has quit IRC | 09:34 | |
| *** markvoelker has quit IRC | 09:37 | |
| strigazi_AFK | AJaeger, yolanda, Good Morning, we can merge this: https://review.openstack.org/#/c/400598/ | 09:37 |
| *** strigazi_AFK is now known as strigazi | 09:39 | |
| *** derekh has joined #openstack-infra | 09:41 | |
| *** yuval has quit IRC | 09:43 | |
| Qiming | guys, I'm seeing some strange errors from gate jobs complaining that service endpoint not found, we haven't done any changes to gate jobs for quite a few weeks | 09:44 |
| Qiming | this error is happening starting a few hours ago | 09:45 |
| *** NBziouech has joined #openstack-infra | 09:45 | |
| Qiming | is there any infra side change we should be aware of, e.g. adapt gate jobs accordingly? | 09:45 |
| *** ccamacho is now known as ccamacho|afk | 09:47 | |
| therve | Qiming, Is it Heat related? | 09:47 |
| therve | If it's using the heatclient, we got a regression in the latest release which is being worked on | 09:47 |
| openstackgerrit | Lee Yarwood proposed openstack-infra/elastic-recheck: Add query for grenade ssh key failure bug 1644150 https://review.openstack.org/401156 | 09:47 |
| *** pilgrimstack has joined #openstack-infra | 09:47 | |
| openstack | bug 1644150 in grenade "Failed to connect to the host via ssh: Host key verification failed" [Undecided,New] https://launchpad.net/bugs/1644150 | 09:47 |
| Qiming | therve, seems not | 09:49 |
| Qiming | we were not able to get senlin public endpoint when calling tempest tests, not sure if something changed in tempest, infra, devstack ... ? | 09:50 |
| Qiming | thanks for asking, therve | 09:50 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: WIP TLS everywhere job https://review.openstack.org/391738 | 09:51 |
| *** pilgrimstack1 has joined #openstack-infra | 09:51 | |
| *** pilgrimstack has quit IRC | 09:51 | |
| *** dtantsur is now known as dtantsur|bbl | 10:00 | |
| *** wolverineav has quit IRC | 10:00 | |
| *** zhurong has quit IRC | 10:01 | |
| AJaeger | strigazi: are you sure it works? Then I'll +2A | 10:01 |
| AJaeger | Qiming: better ask on #openstack-qa | 10:01 |
| *** ggnel_t has quit IRC | 10:02 | |
| AJaeger | strigazi: Ah, see your comment now - will +2A | 10:02 |
| Qiming | thanks, AJaeger, was doubting if any changes at infra side is making senlin jobs failing, will keep digging | 10:03 |
| strigazi | AJaeger, thanks | 10:03 |
| *** gildub has joined #openstack-infra | 10:03 | |
| *** hashar has joined #openstack-infra | 10:07 | |
| AJaeger | Qiming, I'm not aware of anything - but I guesss you need to debug a bit first ;) | 10:08 |
| Qiming | AJaeger, therve, have figured out the reason ... devstack has removed the definition of 'KEYSTONE_CATALOG_BACKEND' 8 hours ago, and that was (for unknown reasons) affecting senlin plugin setup at gate side, fixed... | 10:08 |
| openstackgerrit | Merged openstack-infra/project-config: [magnum] Update fedora-atomic build job to 24 https://review.openstack.org/400598 | 10:08 |
| Qiming | thanks anyway | 10:08 |
| AJaeger | Qiming: glad to hear! | 10:08 |
| *** yanyanhu has quit IRC | 10:11 | |
| *** electrofelix has joined #openstack-infra | 10:14 | |
| *** ggnel_t has joined #openstack-infra | 10:15 | |
| *** gongysh has quit IRC | 10:15 | |
| *** wolverineav has joined #openstack-infra | 10:20 | |
| *** jamielennox is now known as jamielennox|away | 10:21 | |
| *** esikachev has quit IRC | 10:23 | |
| *** sambetts|afk is now known as sambetts | 10:24 | |
| *** wolverineav has quit IRC | 10:25 | |
| *** ihrachys has quit IRC | 10:28 | |
| *** claudiub|2 has joined #openstack-infra | 10:29 | |
| *** tojuvone has joined #openstack-infra | 10:30 | |
| *** ldnunes has joined #openstack-infra | 10:31 | |
| openstackgerrit | Andy McCrae proposed openstack-infra/project-config: Fix OpenStack-Ansible xenial gate jobs https://review.openstack.org/401184 | 10:31 |
| *** geguileo has joined #openstack-infra | 10:34 | |
| *** jascott1_ has joined #openstack-infra | 10:34 | |
| *** jascott1 has quit IRC | 10:35 | |
| *** pkoniszewski has quit IRC | 10:35 | |
| *** isq has quit IRC | 10:35 | |
| *** jmorgan1 has quit IRC | 10:35 | |
| geguileo | Hi, Cinder gate is kind of blocked, could we get another core to review https://review.openstack.org/394496? | 10:36 |
| geguileo | We have over 1K failures a day on the multinode grenade job... | 10:36 |
| geguileo | Thanks! | 10:36 |
| *** newmember has quit IRC | 10:37 | |
| *** qwertyco has quit IRC | 10:40 | |
| *** tuanluong has quit IRC | 10:40 | |
| *** yolanda has quit IRC | 10:41 | |
| *** jascott1 has joined #openstack-infra | 10:41 | |
| *** panda|Zz is now known as panda | 10:43 | |
| *** jmorgan1 has joined #openstack-infra | 10:44 | |
| *** tosky has joined #openstack-infra | 10:44 | |
| *** pkoniszewski has joined #openstack-infra | 10:44 | |
| openstackgerrit | Pascal Mazon proposed openstack-infra/project-config: Add 6WIND Virtual Accelerator Charm https://review.openstack.org/400621 | 10:44 |
| *** mmoster_ has quit IRC | 10:44 | |
| *** jascott1_ has quit IRC | 10:45 | |
| *** ihrachys has joined #openstack-infra | 10:46 | |
| *** wolverineav has joined #openstack-infra | 10:48 | |
| *** ccamacho|afk is now known as ccamacho | 10:50 | |
| *** wolverineav has quit IRC | 10:53 | |
| *** ihrachys has quit IRC | 10:53 | |
| *** nadya has quit IRC | 10:53 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: WIP TLS everywhere job https://review.openstack.org/391738 | 10:55 |
| openstackgerrit | Vitaly Gridnev proposed openstack-infra/project-config: [sahara] finish transitioning scenario jobs to xenial https://review.openstack.org/400814 | 10:59 |
| openstackgerrit | Vitaly Gridnev proposed openstack-infra/project-config: [sahara] finish transitioning scenario jobs to xenial https://review.openstack.org/400814 | 10:59 |
| *** igormarnat has quit IRC | 11:00 | |
| *** dteselkin has quit IRC | 11:00 | |
| *** katyafervent_awa has quit IRC | 11:01 | |
| *** asilenkov has quit IRC | 11:01 | |
| *** rakhmerov has quit IRC | 11:01 | |
| *** ativelkov has quit IRC | 11:01 | |
| *** ativelkov has joined #openstack-infra | 11:03 | |
| *** nadya has joined #openstack-infra | 11:04 | |
| openstackgerrit | Ian Y. Choi proposed openstack-infra/system-config: Switch translate-dev to use openstackid https://review.openstack.org/393405 | 11:05 |
| *** rakhmerov has joined #openstack-infra | 11:05 | |
| *** asilenkov has joined #openstack-infra | 11:05 | |
| *** katyafervent_awa has joined #openstack-infra | 11:06 | |
| *** dteselkin has joined #openstack-infra | 11:07 | |
| *** vikrant is now known as vikrant|T | 11:07 | |
| *** igormarnat has joined #openstack-infra | 11:07 | |
| *** wolverineav has joined #openstack-infra | 11:08 | |
| *** mmandel has joined #openstack-infra | 11:08 | |
| *** Hal1 has joined #openstack-infra | 11:09 | |
| *** yolanda has joined #openstack-infra | 11:11 | |
| *** wolverineav has quit IRC | 11:12 | |
| *** mmandel has quit IRC | 11:12 | |
| *** Jeffrey4l has joined #openstack-infra | 11:14 | |
| *** yuval has joined #openstack-infra | 11:15 | |
| *** yuval has quit IRC | 11:15 | |
| *** sreeram_v has quit IRC | 11:18 | |
| *** ad_rien_ has quit IRC | 11:18 | |
| *** sreeram_v has joined #openstack-infra | 11:19 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci: DO NOT MERGE Transition to quickstart: simulate transition https://review.openstack.org/399257 | 11:21 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci: Transition to quickstart: add toci_gate_oooq.sh and jobs configurations https://review.openstack.org/399256 | 11:21 |
| *** sreeram_v has quit IRC | 11:23 | |
| *** wolverineav has joined #openstack-infra | 11:27 | |
| *** wolverineav has quit IRC | 11:31 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/devstack-gate: WIP Add setup_host playbook https://review.openstack.org/391519 | 11:33 |
| *** ccamacho is now known as ccamacho|lunch | 11:34 | |
| openstackgerrit | Isaac Beckman proposed openstack-infra/project-config: selective caching of git repos https://review.openstack.org/322487 | 11:35 |
| *** dtantsur|bbl is now known as dtantsur | 11:36 | |
| openstackgerrit | Merged openstack-infra/project-config: Set Cinder multinode grenade on master to xenial https://review.openstack.org/394496 | 11:39 |
| *** boden has joined #openstack-infra | 11:40 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix Cinder's multibackend ZeroMQ job https://review.openstack.org/401150 | 11:42 |
| openstackgerrit | Merged openstack-infra/project-config: Run checklang on all branches https://review.openstack.org/399667 | 11:43 |
| openstackgerrit | Merged openstack-infra/project-config: Add mistral gate for mysql unit tests https://review.openstack.org/400535 | 11:43 |
| openstackgerrit | Merged openstack-infra/project-config: networking-odl: disable rally job for pre-ocata release https://review.openstack.org/399283 | 11:43 |
| openstackgerrit | Merged openstack-infra/project-config: retire django-openstack-auth-kerberos https://review.openstack.org/399649 | 11:43 |
| openstackgerrit | Merged openstack-infra/project-config: Revise ACLs for networking-[onos|l2gw] projects https://review.openstack.org/399798 | 11:46 |
| openstackgerrit | Merged openstack-infra/project-config: Fix different zmq configurations for functional tests https://review.openstack.org/400823 | 11:46 |
| *** Nrk9t1x has quit IRC | 11:46 | |
| openstackgerrit | Merged openstack-infra/project-config: All Python35 issues should be fixed for MoltenIron https://review.openstack.org/400820 | 11:47 |
| openstackgerrit | Merged openstack-infra/project-config: Remove duplicated Horizon test runs https://review.openstack.org/400084 | 11:47 |
| openstackgerrit | Merged openstack-infra/project-config: [Manila] Adds job for Manila UI DevStack plugin https://review.openstack.org/399768 | 11:47 |
| *** wolverineav has joined #openstack-infra | 11:47 | |
| amrith | g'morning infra, is tarballs.openstack.org a single thing, or is it really distributed; one per CI cloud? | 11:47 |
| openstackgerrit | Merged openstack-infra/project-config: tooz: make Python 3.5 job voting https://review.openstack.org/399624 | 11:48 |
| openstackgerrit | Merged openstack-infra/project-config: [rally] switch all jobs to xenial and unblock gates https://review.openstack.org/398953 | 11:48 |
| openstackgerrit | Andy McCrae proposed openstack-infra/project-config: Fix OpenStack-Ansible xenial gate jobs https://review.openstack.org/401184 | 11:50 |
| *** psilvad has joined #openstack-infra | 11:50 | |
| *** e0ne has joined #openstack-infra | 11:50 | |
| amrith | AJaeger, g'afternoon, would you know whether ^^ | 11:51 |
| *** wolverineav has quit IRC | 11:52 | |
| *** claudiub has joined #openstack-infra | 11:52 | |
| openstackgerrit | Sergey Reshetnyak proposed openstack-infra/project-config: Add new project: fuel-ccp-rally https://review.openstack.org/401220 | 11:52 |
| *** nicolasbock has joined #openstack-infra | 11:54 | |
| *** claudiub|2 has quit IRC | 11:54 | |
| *** ssbarnea_ has joined #openstack-infra | 11:55 | |
| openstackgerrit | Isaac Beckman proposed openstack-infra/puppet-jenkins: Customize jenkins.default by template https://review.openstack.org/262787 | 11:55 |
| *** yolanda has quit IRC | 11:55 | |
| *** yolanda has joined #openstack-infra | 11:55 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/devstack-gate: WIP Add setup_host playbook https://review.openstack.org/391519 | 11:56 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci: DO NOT MERGE Transition to quickstart: simulate transition https://review.openstack.org/399257 | 11:58 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci: Transition to quickstart: add toci_gate_oooq.sh and jobs configurations https://review.openstack.org/399256 | 11:58 |
| *** ssbarnea has quit IRC | 11:58 | |
| *** mkoderer has joined #openstack-infra | 11:59 | |
| openstackgerrit | Tuan Luong-Anh proposed openstack/diskimage-builder: Replace six.iteritems() with .items() https://review.openstack.org/400514 | 12:03 |
| *** openstackgerrit has quit IRC | 12:03 | |
| *** openstackgerrit has joined #openstack-infra | 12:03 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: WIP TLS everywhere job https://review.openstack.org/391738 | 12:04 |
| *** jascott1 has quit IRC | 12:04 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/devstack-gate: WIP Add setup_host playbook https://review.openstack.org/391519 | 12:04 |
| *** raildo has joined #openstack-infra | 12:05 | |
| *** wolverineav has joined #openstack-infra | 12:07 | |
| *** jcoufal has joined #openstack-infra | 12:09 | |
| *** gildub has quit IRC | 12:11 | |
| *** wolverineav has quit IRC | 12:11 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/devstack-gate: WIP Add setup_host playbook https://review.openstack.org/391519 | 12:14 |
| *** lseki has joined #openstack-infra | 12:14 | |
| *** ssbarnea has joined #openstack-infra | 12:14 | |
| *** gongysh has joined #openstack-infra | 12:15 | |
| *** jordanP has joined #openstack-infra | 12:16 | |
| *** gongysh has quit IRC | 12:17 | |
| *** ssbarnea_ has quit IRC | 12:18 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Full quickstart gate run on OVB https://review.openstack.org/381094 | 12:19 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: DONT MERGE: Test quickstart experimental job https://review.openstack.org/397609 | 12:19 |
| *** abregman is now known as abregman|mtg | 12:21 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Full quickstart gate run on OVB https://review.openstack.org/381094 | 12:21 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: DONT MERGE: Test quickstart experimental job https://review.openstack.org/397609 | 12:21 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/devstack-gate: WIP Add setup_host playbook https://review.openstack.org/391519 | 12:21 |
| *** lucasagomes is now known as lucas-hungry | 12:21 | |
| lseki | guys, I have a question about zuul pipeline precedence | 12:22 |
| lseki | if there's a high priority pipeline that is triggered all the time, a lower precedence pipeline may never be started | 12:23 |
| lseki | is that correct? | 12:23 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci: DO NOT MERGE Transition to quickstart: simulate transition https://review.openstack.org/399257 | 12:25 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci: Transition to quickstart: add toci_gate_oooq.sh and jobs configurations https://review.openstack.org/399256 | 12:25 |
| *** rtheis has joined #openstack-infra | 12:26 | |
| *** rtheis has quit IRC | 12:26 | |
| *** pgadiya has quit IRC | 12:26 | |
| *** gouthamr has joined #openstack-infra | 12:32 | |
| dulek | All the gate-grenade-dsvm-cinder-multinode-ubuntu-xenial jobs in Cinder are hanging in queued state, even though some patches run for 1,5 hour. I'm scared I've broke something with https://review.openstack.org/#/c/394496/ | 12:33 |
| lseki | @jeblair ^ | 12:33 |
| dulek | Could anyone help me investigate? | 12:33 |
| *** ssbarnea has quit IRC | 12:35 | |
| *** ssbarnea has joined #openstack-infra | 12:35 | |
| *** wolverineav has joined #openstack-infra | 12:36 | |
| *** nadya has quit IRC | 12:37 | |
| pabelanger | lseki: yes, that is correct | 12:38 |
| *** shardy is now known as shardy_lunch | 12:39 | |
| pabelanger | lseki: you can setup a window-floor to help control how many changes to run at 1 time in a pipeline, we do this with gate as not to starve other pipelines | 12:39 |
| *** ihrachys has joined #openstack-infra | 12:40 | |
| *** weshay_ has quit IRC | 12:40 | |
| *** mmoster_ has joined #openstack-infra | 12:40 | |
| pabelanger | lseki: note, the pipeline needs to be a DependentPipelineManagers | 12:41 |
| *** wolverineav has quit IRC | 12:41 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/devstack-gate: Add role for fix_etc_hosts https://review.openstack.org/383052 | 12:45 |
| *** nadya has joined #openstack-infra | 12:45 | |
| *** nadya has quit IRC | 12:46 | |
| *** oanson has quit IRC | 12:53 | |
| *** links has quit IRC | 12:54 | |
| *** wolverineav has joined #openstack-infra | 12:56 | |
| *** dprince has joined #openstack-infra | 12:56 | |
| *** sreeram_v has joined #openstack-infra | 12:57 | |
| *** ssbarnea has quit IRC | 12:58 | |
| *** ssbarnea has joined #openstack-infra | 12:58 | |
| *** wolverineav has quit IRC | 13:00 | |
| *** sparkiegeek has joined #openstack-infra | 13:02 | |
| *** sreeram_v has quit IRC | 13:02 | |
| sparkiegeek | hi, apologies in advance if I'm asking in the wrong place. I'm trying to get a patch up for review, and had troubles signing the CLA - I did it through Gerrit, filled in my details (inc. mailing address) and got errors about saving contact information | 13:04 |
| lseki | @pabelanger great! I'll use it | 13:04 |
| lseki | thx | 13:04 |
| *** nherciu_ has joined #openstack-infra | 13:04 | |
| sparkiegeek | I tried several times, and consistently got errors, so then I tried without my mailing info and it went through. But now "git review" complains that it needs contact information :/ | 13:05 |
| *** wolverineav has joined #openstack-infra | 13:07 | |
| *** dkehn has quit IRC | 13:08 | |
| *** ccamacho|lunch is now known as ccamacho | 13:08 | |
| *** dkehn has joined #openstack-infra | 13:08 | |
| *** nherciu has quit IRC | 13:08 | |
| *** mtanino has quit IRC | 13:09 | |
| *** adarazs has quit IRC | 13:12 | |
| openstackgerrit | Vitaly Gridnev proposed openstack-infra/project-config: [sahara] finish transitioning scenario jobs to xenial https://review.openstack.org/400814 | 13:13 |
| *** ihrachys has quit IRC | 13:17 | |
| *** adarazs has joined #openstack-infra | 13:18 | |
| openstackgerrit | Mike Kolesnik proposed openstack-infra/project-config: Fix functional tests for networking-odl https://review.openstack.org/401255 | 13:18 |
| *** Guest59528 has quit IRC | 13:19 | |
| *** amoralej is now known as amoralej|lunch | 13:20 | |
| *** ad_rien_ has joined #openstack-infra | 13:21 | |
| *** ad_rien_ has quit IRC | 13:21 | |
| *** ad_rien_ has joined #openstack-infra | 13:22 | |
| *** ad_rien_ has quit IRC | 13:22 | |
| openstackgerrit | Valeriy Ponomaryov proposed openstack-infra/project-config: [Manila] Fix new gate-manila-ui-dsvm-nv job https://review.openstack.org/401257 | 13:23 |
| *** lucas-hungry is now known as lucasagomes | 13:23 | |
| *** NBziouech has quit IRC | 13:24 | |
| openstackgerrit | Valeriy Ponomaryov proposed openstack-infra/project-config: [Manila] Fix new gate-manila-ui-dsvm-nv job https://review.openstack.org/401257 | 13:24 |
| *** ad_rien_ has joined #openstack-infra | 13:24 | |
| *** yamamoto has joined #openstack-infra | 13:25 | |
| *** newmember has joined #openstack-infra | 13:26 | |
| *** NBziouech has joined #openstack-infra | 13:27 | |
| *** markvoelker has joined #openstack-infra | 13:28 | |
| *** shardy_lunch is now known as shardy | 13:28 | |
| *** yolanda has quit IRC | 13:29 | |
| *** charlledaniel has joined #openstack-infra | 13:34 | |
| *** Julien-zte has joined #openstack-infra | 13:34 | |
| *** abregman|mtg is now known as abregman | 13:35 | |
| *** dkehn has quit IRC | 13:35 | |
| *** dkehn has joined #openstack-infra | 13:35 | |
| openstackgerrit | Igor Belikov proposed openstack-infra/jenkins-job-builder: Add Pipeline job type and deprecate Workflow https://review.openstack.org/323744 | 13:35 |
| charlledaniel | Good Morning OpenStack. Does anyone know any admin from https://review.openstack.org? I have two accounts there and I wanted to delete one of them, but I don't know how. | 13:36 |
| *** xyang1 has joined #openstack-infra | 13:36 | |
| dulek | All the gate-grenade-dsvm-cinder-multinode-ubuntu-xenial jobs in Cinder are hanging in queued state, even though some patches run for 1,5 hour. I'm scared I've broke something with https://review.openstack.org/#/c/394496/ | 13:41 |
| dulek | Could anyone help with investigation? | 13:41 |
| *** baoli has joined #openstack-infra | 13:43 | |
| *** baoli has quit IRC | 13:43 | |
| *** baoli has joined #openstack-infra | 13:43 | |
| *** jamesdenton has joined #openstack-infra | 13:44 | |
| *** jpena is now known as jpena|lunch | 13:45 | |
| *** dmsimard|away is now known as dmsimard | 13:45 | |
| *** snarwade has quit IRC | 13:45 | |
| *** mdrabe has joined #openstack-infra | 13:46 | |
| *** nadya has joined #openstack-infra | 13:46 | |
| openstackgerrit | Tung Doan proposed openstack-infra/project-config: Tacker changes to xenial node https://review.openstack.org/400780 | 13:48 |
| *** esikachev has joined #openstack-infra | 13:48 | |
| *** tlian has joined #openstack-infra | 13:49 | |
| *** hrubi has joined #openstack-infra | 13:50 | |
| *** derekh is now known as derekh_afk | 13:51 | |
| *** nadya has quit IRC | 13:51 | |
| *** trash has quit IRC | 13:52 | |
| *** jamesdenton has quit IRC | 13:55 | |
| *** abregman has quit IRC | 13:55 | |
| fungi | dulek: it looks like there are no workers registered to run that job since roughly when https://review.openstack.org/394496 got applied. i'm still digging to find the cause | 13:56 |
| *** gordc has joined #openstack-infra | 13:57 | |
| dulek | fungi: Thanks! In the same time ubuntu-xenial-2-node jobs from Neutron are working just fine. | 13:57 |
| * sparkiegeek 's issue was due to mis-matches in email addresses between my www.openstack.org account and review.openstack.org account - now resolved | 13:57 | |
| openstackgerrit | Igor Belikov proposed openstack-infra/jenkins-job-builder: Add Pipeline job type and deprecate Workflow https://review.openstack.org/323744 | 13:58 |
| *** sparkiegeek has left #openstack-infra | 13:58 | |
| *** ramishra has quit IRC | 13:59 | |
| *** ramishra has joined #openstack-infra | 14:01 | |
| *** kgiusti has joined #openstack-infra | 14:02 | |
| *** Zara has quit IRC | 14:04 | |
| *** yankcrime has quit IRC | 14:04 | |
| *** SotK has quit IRC | 14:04 | |
| *** jascott1 has joined #openstack-infra | 14:04 | |
| *** fguillot has joined #openstack-infra | 14:05 | |
| fungi | we have 24 ubuntu-xenial-2-node workers running jobs, with reasonable runtimes | 14:05 |
| *** SotK has joined #openstack-infra | 14:05 | |
| fungi | looking at gearman registrations, i see build:gate-grenade-dsvm-neutron-multinode-ubuntu-xenial:ubuntu-xenial-2-node and build:gate-grenade-dsvm-neutron-multinode-ubuntu-xenial in neutron's case | 14:06 |
| *** _nick has joined #openstack-infra | 14:07 | |
| *** abregman has joined #openstack-infra | 14:07 | |
| fungi | but in cinder's case only build:gate-grenade-dsvm-cinder-multinode-ubuntu-xenial is registered | 14:07 |
| fungi | so for some reason the node-specific registration isn't present | 14:07 |
| *** thorst has joined #openstack-infra | 14:08 | |
| *** jascott1 has quit IRC | 14:09 | |
| *** Zara has joined #openstack-infra | 14:09 | |
| openstackgerrit | Andy McCrae proposed openstack-infra/project-config: Fix OpenStack-Ansible xenial gate jobs https://review.openstack.org/401184 | 14:10 |
| *** ijw has joined #openstack-infra | 14:10 | |
| *** pcaruana has quit IRC | 14:14 | |
| *** ijw has quit IRC | 14:15 | |
| *** jamesdenton has joined #openstack-infra | 14:15 | |
| openstackgerrit | Jakub Libosvar proposed openstack-infra/elastic-recheck: Add query for grenade multinode failure, bug 1643485 https://review.openstack.org/401279 | 14:15 |
| openstack | bug 1643485 in neutron "Stable Newton grenade multinode jobs fail with "Failed to connect to the host via ssh: Host key verification failed."" [Critical,New] https://launchpad.net/bugs/1643485 | 14:15 |
| *** jlibosva has joined #openstack-infra | 14:15 | |
| dulek | fungi: Hm, I've tried to write the patch the same way as Neutron one. | 14:15 |
| fungi | dulek: yeah, i see that. i'm not immediately spotting anything wrong with the patch but i'm checking out how it actually parses at this point | 14:16 |
| dulek | fungi: Okay, I'll try looking through the patch again. | 14:17 |
| *** mhickey has joined #openstack-infra | 14:18 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci: logging: extract journal to compressed text file, before gathering https://review.openstack.org/400817 | 14:19 |
| *** abregman is now known as abregman|mtg | 14:19 | |
| *** thorst has quit IRC | 14:19 | |
| *** annegentle has joined #openstack-infra | 14:19 | |
| *** yolanda has joined #openstack-infra | 14:21 | |
| *** vikrant|T has quit IRC | 14:21 | |
| *** amoralej|lunch is now known as amoralej | 14:22 | |
| *** cshastri has quit IRC | 14:22 | |
| *** thorst has joined #openstack-infra | 14:22 | |
| *** mriedem has joined #openstack-infra | 14:23 | |
| openstackgerrit | Tung Doan proposed openstack-infra/project-config: Tacker changes to xenial node https://review.openstack.org/400780 | 14:23 |
| *** nadya has joined #openstack-infra | 14:24 | |
| *** jamesdenton has quit IRC | 14:24 | |
| *** psilvad has quit IRC | 14:25 | |
| *** EricGonczer_ has joined #openstack-infra | 14:25 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/project-config: WIP: always clone quickstart extras with quickstart https://review.openstack.org/401282 | 14:26 |
| *** pcrews has quit IRC | 14:26 | |
| *** psachin has quit IRC | 14:26 | |
| *** pcrews has joined #openstack-infra | 14:28 | |
| *** thorst has quit IRC | 14:31 | |
| *** nmagnezi has quit IRC | 14:31 | |
| *** tesseract has joined #openstack-infra | 14:32 | |
| *** tesseract is now known as Guest48653 | 14:33 | |
| *** ihrachys has joined #openstack-infra | 14:33 | |
| openstackgerrit | Merged openstack-infra/zuul: Skip individual test_requirements tests https://review.openstack.org/401018 | 14:35 |
| *** rbuzatu_ has joined #openstack-infra | 14:38 | |
| openstackgerrit | Tung Doan proposed openstack-infra/project-config: Tacker changes to xenial node https://review.openstack.org/400780 | 14:39 |
| *** Guest48653 has quit IRC | 14:39 | |
| *** tesseract- has joined #openstack-infra | 14:39 | |
| fungi | dulek: i think the difference you spotted is right on the money: http://paste.openstack.org/show/590216/ | 14:39 |
| fungi | note the list brackets | 14:40 |
| dulek | fungi: Aww… | 14:40 |
| fungi | i'm testing now to confirm | 14:41 |
| *** yamamoto has quit IRC | 14:42 | |
| *** rbuzatu has quit IRC | 14:42 | |
| *** ssbarnea_ has joined #openstack-infra | 14:43 | |
| *** rbuzatu_ has quit IRC | 14:43 | |
| *** yamamoto has joined #openstack-infra | 14:44 | |
| *** yamamoto has quit IRC | 14:45 | |
| *** mmoster_ has quit IRC | 14:46 | |
| *** ssbarnea has quit IRC | 14:46 | |
| *** derekh_afk is now known as derekh | 14:47 | |
| ihrachys | any ideas why grenade fails on trusty on ssh verification? https://bugs.launchpad.net/grenade/+bug/1643485 | 14:49 |
| openstack | Launchpad bug 1643485 in grenade "Stable Newton grenade multinode jobs fail with "Failed to connect to the host via ssh: Host key verification failed."" [Critical,New] | 14:49 |
| dulek | fungi: I have the commit changing that to single value notation ready. | 14:49 |
| ihrachys | it breaks newton neutron gate and earlier, and I suspect even more, all trusty jobs | 14:49 |
| ihrachys | *grenade | 14:49 |
| dulek | ihrachys: It's failing only on osic/rax nodes. | 14:49 |
| fungi | dulek: yeah, i'm so far failing to actually get that alteration to take effect | 14:49 |
| *** tesseract- has quit IRC | 14:49 | |
| dulek | fungi: So you suppose it could be something else? | 14:50 |
| ihrachys | dulek: ok. is there any fix/people working on it/ideas why the failure? | 14:50 |
| dulek | ihrachys: I've pinged #osic-ops on the issue, no movement from their side yet. | 14:50 |
| *** flepied1 has joined #openstack-infra | 14:50 | |
| dulek | ihrachys: You can nag them too. ;) | 14:51 |
| cloudnull | dulek: mind dropping something in #osic-clouds assuming you suspect its something wrong with the cloud | 14:51 |
| fungi | dulek: i'm so far still trying to rule out that something has caused the zuul launchers to not actually read configuration updates, or is preventing them from registering new jobs into gearman | 14:51 |
| *** jpena|lunch is now known as jpena | 14:51 | |
| *** hurgleburgler has joined #openstack-infra | 14:51 | |
| fungi | but it's also possible that my manual attempt to update this on one of the zuul launchers is based on poor assumptions on my part | 14:51 |
| cloudnull | if you think its a gate issue there's little we can do from an infra side (osic infra that is). | 14:51 |
| dulek | cloudnull: To be honest I'm not sure, but key error on Ansible side suggests it is cloud's fault. | 14:52 |
| *** andymaier has joined #openstack-infra | 14:52 | |
| cloudnull | we're talking about the missing host key right ? | 14:52 |
| *** rbuzatu has joined #openstack-infra | 14:53 | |
| fungi | dulek: what key error? in a job log? | 14:53 |
| *** jaosorior has quit IRC | 14:53 | |
| *** flepied has quit IRC | 14:53 | |
| dulek | fungi: That's another thread - trusty jobs failing on OSIC and RAX nodes. | 14:54 |
| fungi | unless it's a multi-node job, jobs shouldn't be sshing over the network to other nodes | 14:54 |
| *** jaosorior has joined #openstack-infra | 14:54 | |
| *** anilvenkata has joined #openstack-infra | 14:54 | |
| dulek | fungi: It is happening for multinode jobs. | 14:54 |
| anilvenkata | DinaBelova, Hi | 14:54 |
| *** yamamoto has joined #openstack-infra | 14:54 | |
| fungi | dulek: oh, only multi-node? and it's at the step where the primary node tries to ssh to the subnode? | 14:54 |
| dulek | fungi: That's why we've pushed to switch the Cinder multinode grenade to xenial - to get over it failing on OSIC's trusty. | 14:54 |
| dulek | fungi: I'm not really sure, but here's the logstash for this one: http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22Failed%20to%20connect%20to%20the%20host%20via%20ssh:%20Host%20key%20verification%20failed%5C%22&from=864000s | 14:55 |
| *** nherciu_ has quit IRC | 14:55 | |
| fungi | well, there's nothing within the cloud provider that should make ssh between nodes of one distro release fail and between another work | 14:55 |
| *** jkilpatr_ has joined #openstack-infra | 14:55 | |
| cloudnull | ++ ^ | 14:56 |
| fungi | so i have a very strong feeling any provider-specific behavior differences you' | 14:56 |
| fungi | re seeing on that behavior will come down to nodepool having managed to update images in some providers and not others, and something on newer images is broken | 14:56 |
| fungi | when did it start? | 14:56 |
| *** jkilpatr has quit IRC | 14:56 | |
| *** Goneri has joined #openstack-infra | 14:56 | |
| dulek | fungi: Cool, thanks for clarifying. :) My apologies cloudnull! | 14:57 |
| dulek | fungi: logstash shows 21th. | 14:57 |
| fungi | also, seriously hoping another infra-root happens along and can take over. i'm actually supposed to be cooking and cleaning before my inlaws arrive in a few hours | 14:57 |
| fungi | haven't even had my coffee yet | 14:58 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/project-config: WIP: Run oooq ha and nonha jobs for quickstart https://review.openstack.org/401298 | 14:58 |
| pabelanger | fungi: o/ | 14:58 |
| pabelanger | don't mind helping | 14:58 |
| fungi | oh. pabelanger! | 14:58 |
| pabelanger | let me catch up on backscroll | 14:58 |
| openstackgerrit | Carlos Camacho proposed openstack-infra/tripleo-ci: Add Panko to scenario001 https://review.openstack.org/401299 | 14:59 |
| cloudnull | fungi: wait for the inlaws to arrive, then make them clean :) | 14:59 |
| fungi | pabelanger: two current issues. for some reason the change to switch cinder's grenade multinode jobs to xenial didn't register a node-specific entry in gearman so the gate is backed up waiting for nodes to run those jobs for several hours. also the impetus for them switching is because ssh between subnodes on trusty seems to have been broken for a couple days | 15:00 |
| cloudnull | it can be forced family fun | 15:00 |
| *** john-davidge has quit IRC | 15:00 | |
| *** Julien-zte has quit IRC | 15:00 | |
| fungi | okay! problem 1 is solved | 15:00 |
| *** john-davidge has joined #openstack-infra | 15:00 | |
| pabelanger | fungi: ack | 15:01 |
| fungi | dulek: my ministrations proved effective, i just didn't wait long enough for the launcher to finish reloading the modified job config | 15:01 |
| fungi | dulek: go ahead and push up your proposed change for that and i'll approve it | 15:01 |
| *** abregman_ has joined #openstack-infra | 15:01 | |
| *** Julien-zte has joined #openstack-infra | 15:01 | |
| fungi | pabelanger: if you can look into the trusty multinode problem with ssh between subnodes, that would be a huge help | 15:02 |
| *** Julien-zte has quit IRC | 15:02 | |
| pabelanger | fungi: sure, let me see if i can hold some nodes | 15:02 |
| *** oanson has joined #openstack-infra | 15:02 | |
| *** Julien-zte has joined #openstack-infra | 15:02 | |
| *** Julien-zte has quit IRC | 15:03 | |
| pabelanger | Oh | 15:03 |
| pabelanger | neat, I cannot ssh into it either | 15:03 |
| pabelanger | Unable to negotiate with 162.242.232.36 port 22: no matching host key type found. Their offer: ssh-dss | 15:03 |
| *** Julien-zte has joined #openstack-infra | 15:03 | |
| *** Julien-zte has quit IRC | 15:03 | |
| pabelanger | looks like a side affect of our recent change to runtime-ssh-host-keys | 15:03 |
| fungi | pabelanger: i suspect the fix for the dib release has broken something on trusty, but that's just a hunch | 15:04 |
| fungi | yeah, that | 15:04 |
| *** Julien-zte has joined #openstack-infra | 15:04 | |
| *** Julien-zte has quit IRC | 15:04 | |
| fungi | it apparently only started breaking on the 21st, which is roughly when the fix for host key regeneration went in | 15:04 |
| *** claudiub|2 has joined #openstack-infra | 15:04 | |
| *** Julien-zte has joined #openstack-infra | 15:05 | |
| pabelanger | yup | 15:05 |
| fungi | or at least when we started building and uploading new images for it | 15:05 |
| *** abregman|mtg has quit IRC | 15:05 | |
| *** thorst has joined #openstack-infra | 15:05 | |
| *** Julien-zte has quit IRC | 15:05 | |
| *** john-davidge has quit IRC | 15:05 | |
| pabelanger | ssh -oHostKeyAlgorithms=+ssh-dss 162.242.232.36 is my fix | 15:05 |
| pabelanger | but let me see which keys are generated | 15:05 |
| *** panda is now known as panda|brb | 15:05 | |
| *** Julien-zte has joined #openstack-infra | 15:06 | |
| *** Julien-zte has quit IRC | 15:06 | |
| *** Julien-zte has joined #openstack-infra | 15:06 | |
| openstackgerrit | Michal Dulko proposed openstack-infra/project-config: Fix Cinder multinode grenade job https://review.openstack.org/401304 | 15:06 |
| *** mmandel has joined #openstack-infra | 15:07 | |
| *** Julien-zte has quit IRC | 15:07 | |
| *** claudiub has quit IRC | 15:07 | |
| dulek | fungi, pabelanger: ^^ | 15:07 |
| *** dizquierdo has quit IRC | 15:08 | |
| *** mpaolino has joined #openstack-infra | 15:08 | |
| *** ihrachys has quit IRC | 15:08 | |
| *** jaosorior has quit IRC | 15:09 | |
| pabelanger | fungi: Yup, we are missing ssh-rsa keys | 15:10 |
| *** jaosorior has joined #openstack-infra | 15:10 | |
| pabelanger | host keys* | 15:10 |
| openstackgerrit | Stéphane Albert proposed openstack-infra/project-config: Add specs repository for CloudKitty https://review.openstack.org/401305 | 15:10 |
| *** abregman_ has quit IRC | 15:11 | |
| *** abregman has joined #openstack-infra | 15:11 | |
| Jeffrey4l | anyone can explain more about cross project testing? http://docs.openstack.org/infra/zuul/gating.html#cross-project-testing i am confused with the second table. | 15:12 |
| *** rossella__ has quit IRC | 15:12 | |
| *** thorst has quit IRC | 15:12 | |
| *** ihrachys has joined #openstack-infra | 15:12 | |
| pabelanger | fungi: Hmm, I think we had pipefail issues on trusty too | 15:12 |
| *** senk has joined #openstack-infra | 15:12 | |
| pabelanger | logic there to generate rsa keys, but didn't happen | 15:12 |
| anilvenkata | I have posted a question regarding osprofiler https://answers.launchpad.net/osprofiler/+question/404334 | 15:12 |
| anilvenkata | can someone help? | 15:13 |
| *** wolverineav has quit IRC | 15:13 | |
| openstackgerrit | Tung Doan proposed openstack-infra/project-config: Tacker-Xenial https://review.openstack.org/401306 | 15:13 |
| pabelanger | I'll delete those images now | 15:13 |
| *** senk_ has quit IRC | 15:14 | |
| *** rossella_s has joined #openstack-infra | 15:15 | |
| *** senk_ has joined #openstack-infra | 15:15 | |
| pabelanger | clarkb: so, ya, we did see pipefail issues with non systemd thing | 15:15 |
| pabelanger | only noticed it now | 15:15 |
| *** pcrews has quit IRC | 15:16 | |
| *** thorst has joined #openstack-infra | 15:16 | |
| fungi | anilvenkata: you probably want to ask in #openstack-neutron or #openstack-oslo | 15:16 |
| openstackgerrit | Merged openstack-infra/project-config: Fix Cinder multinode grenade job https://review.openstack.org/401304 | 15:17 |
| *** inc0 has joined #openstack-infra | 15:17 | |
| fungi | anilvenkata: the community infrastructure sysadmin team doesn't really do anything with osprofiler as far as i'm aware | 15:17 |
| fungi | it's more of a library used by openstack projects | 15:17 |
| *** panda|brb is now known as panda | 15:17 | |
| anilvenkata | fungi, yes, it was a mistake. thanks Jeremy | 15:18 |
| *** senk has quit IRC | 15:18 | |
| *** sandanar_ has quit IRC | 15:19 | |
| *** zz_dimtruck is now known as dimtruck | 15:19 | |
| pabelanger | dulek: fungi: AJaeger: won't be an issue in zuulv3, but we should consider improving our JJB validation tools to error on a single item list. re: 401304 | 15:20 |
| pabelanger | could poke at that on the train today | 15:20 |
| *** thorst has quit IRC | 15:20 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: WIP - Deploy TripleO with Puppet 4 https://review.openstack.org/371209 | 15:22 |
| openstackgerrit | Stéphane Albert proposed openstack-infra/project-config: Add specs repository for CloudKitty https://review.openstack.org/401305 | 15:22 |
| *** hongbin has joined #openstack-infra | 15:22 | |
| *** wolverineav has joined #openstack-infra | 15:23 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: WIP - Deploy TripleO with Puppet 4 https://review.openstack.org/371209 | 15:23 |
| *** tosky has quit IRC | 15:24 | |
| pabelanger | fungi: dulek: broken ubuntu-trusty images should be removed from all clouds. We should have enough coverage in infra-cloud, osic-cloud1-disk and ovh today until replacements uploaded | 15:24 |
| fungi | pabelanger: oh, so were those older images that hadn't been replaced? | 15:25 |
| pabelanger | never | 15:25 |
| pabelanger | uploaded yesterday | 15:25 |
| pabelanger | anything 2+ days was broken | 15:25 |
| *** chlong has joined #openstack-infra | 15:25 | |
| fungi | got it | 15:25 |
| *** tosky has joined #openstack-infra | 15:25 | |
| fungi | thanks! | 15:25 |
| pabelanger | happy to help :) | 15:26 |
| pabelanger | next up is fedora-24 images, we've been having some issues with them recently | 15:27 |
| *** martinkopec has quit IRC | 15:27 | |
| *** drifterza_ has joined #openstack-infra | 15:27 | |
| *** wolverineav has quit IRC | 15:27 | |
| *** jamesdenton has joined #openstack-infra | 15:29 | |
| *** drifterza has quit IRC | 15:30 | |
| pabelanger | okay, fedora-24 should be cleaned up now too | 15:33 |
| *** wolverineav has joined #openstack-infra | 15:33 | |
| *** EricGonc_ has joined #openstack-infra | 15:34 | |
| *** chlong has quit IRC | 15:34 | |
| *** ihrachys has quit IRC | 15:35 | |
| *** pcrews has joined #openstack-infra | 15:35 | |
| anilvenkata | clarkb, ping | 15:35 |
| *** EricGonczer_ has quit IRC | 15:36 | |
| *** charlledaniel has left #openstack-infra | 15:37 | |
| *** wolverineav has quit IRC | 15:38 | |
| vgridnev | does someone know where is the source of truth for this file: https://github.com/openstack/python-neutronclient/blob/master/tools/tox_install.sh ? | 15:40 |
| fungi | vgridnev: no idea what you mean with that question, but maybe you're looking for https://review.openstack.org/317909 ? | 15:42 |
| fungi | that's the point of origin of that file as far back as i can track it in the repo anyway | 15:42 |
| EmilienM | inc0, fungi: it's sad to see a duplicate effort here https://review.openstack.org/#/c/399171/ | 15:43 |
| EmilienM | inc0, fungi: it looks like someone already created this module https://github.com/cristifalcas/puppet-docker_distribution | 15:43 |
| pabelanger | cloudnull: we are seeing a large amount of nodes in the delete state for osic-cloud1, do you mind looking into it? | 15:43 |
| inc0 | EmilienM, not really, this module is for centos only | 15:43 |
| pabelanger | cloudnull: look to be over 4 hours old | 15:44 |
| EmilienM | and? we could make it work on ubuntu/debian | 15:44 |
| EmilienM | why creating a new module from scratch? | 15:44 |
| inc0 | yeah, but it's really simple to run docker registry | 15:44 |
| vgridnev | fungi, actually I guess that this file was copied from some place, so I'm looking for that place | 15:44 |
| inc0 | I could write ansible for it in 10min | 15:44 |
| *** berendt has quit IRC | 15:44 | |
| fungi | vgridnev: in that case, you'd need to ask the author | 15:44 |
| inc0 | I just struggle because I don't know first thing about puppet | 15:44 |
| EmilienM | inc0: it's doesn't explain why we duplicate a module | 15:44 |
| EmilienM | we should rather add missing bits and consume it | 15:45 |
| inc0 | EmilienM, fork->ubuntuize can be harder | 15:45 |
| EmilienM | I'm not sure Infra wants to manage yet another module | 15:45 |
| inc0 | someone has to manage it anyway | 15:45 |
| EmilienM | inc0: why that? you could send a PR to the repo | 15:45 |
| inc0 | this doesn;t have community | 15:45 |
| inc0 | it's single person effort | 15:46 |
| inc0 | and we're banking Kolla's gate on it | 15:46 |
| *** rbuzatu has quit IRC | 15:46 | |
| fungi | EmilienM: inc0: yeah, i should have asked if there was prior research into an existing module to do this (though really we just need to puppet install of a package and conffile_ | 15:46 |
| EmilienM | fungi: for now, who knows how it will evolve | 15:46 |
| inc0 | https://review.openstack.org/#/c/401003/ <- these are conf files and everythong | 15:46 |
| inc0 | we need for now | 15:47 |
| inc0 | and I don't expect needing anything more in forseeable future | 15:47 |
| inc0 | what I need from this PS is puppetize it and make secrets managable | 15:47 |
| fungi | EmilienM: yep, i agree. we didn't want to embed the package/file resource into our system-config repo, but should have looked to see if there was already one out there | 15:47 |
| inc0 | again, centos only | 15:47 |
| fungi | even if it's overkill and needs patching to work for our use case | 15:48 |
| EmilienM | that's not a good pretex | 15:48 |
| inc0 | with 3 commits total | 15:48 |
| *** panda is now known as panda|brb | 15:48 | |
| pabelanger | I still say you could update your jobs today, to use an image downloaded from tarballs.o.o, stand up a local docker-registry and not be blocked by this effort to add a docker-registry to infra | 15:48 |
| pabelanger | if kolla jobs are blocked today | 15:48 |
| EmilienM | inc0: Puppet OpenStack modules support both debian/redhat families, creating a new project just because a distro is not supported is not a valid pretext | 15:49 |
| inc0 | pabelanger, we don't have any place to host these images | 15:49 |
| fungi | EmilienM: inc0: i don't have time to discuss it today, but we should look into combining efforts on cristifalcas/puppet-docker_distribution and retiring the openstack-infra one | 15:49 |
| *** chlong has joined #openstack-infra | 15:49 | |
| pabelanger | inc0: tarballs.o.o | 15:49 |
| pabelanger | that is where you host them | 15:49 |
| inc0 | pabelanger, we don't host *images* there | 15:50 |
| inc0 | built | 15:50 |
| pabelanger | but download them each time the job starts, import into local docker-registry | 15:50 |
| pabelanger | inc0: right, but you can | 15:50 |
| pabelanger | inc0: add a publisher to your post pipeline job | 15:50 |
| inc0 | but we need to build -> tarball and host | 15:50 |
| pabelanger | yes, other projects do this today | 15:50 |
| inc0 | which will multiply amount of data transfered/consumer couple times | 15:50 |
| pabelanger | eg: tripleo, trove, etc | 15:50 |
| inc0 | 2 gigs will turn into 100s probably | 15:50 |
| EmilienM | fungi: I agree. | 15:50 |
| fungi | inc0: technically we _do_ host some images there, for example https://tarballs.openstack.org/trove/images/ and https://tarballs.openstack.org/ironic-python-agent/coreos/ | 15:51 |
| pabelanger | inc0: I don't see how that will change once infra has a docker-registry | 15:51 |
| pabelanger | it will be the same about of data | 15:51 |
| * fungi needs to go afk for a while and get non-openstack things done | 15:52 | |
| inc0 | pabelanger, docker registry understands deduplication of layers | 15:52 |
| *** EricGonc_ has quit IRC | 15:52 | |
| inc0 | it reuses shared stuff | 15:52 |
| pabelanger | I feel, we are rushing into running a docker-registry in infra, because of the false statement that kolla jobs are blocked. Where is there a way we can unblock them today and maybe slow down on standing up the registry until everybody understands the steps needed | 15:53 |
| *** nadya has quit IRC | 15:53 | |
| inc0 | pabelanger, trust me, we would use something else if we could | 15:53 |
| *** wolverineav has joined #openstack-infra | 15:53 | |
| pabelanger | inc0: yes, I am saying run that locally for now, on the vm in the gate. | 15:53 |
| inc0 | docker registry is infrastructure meant to serve purpose we need | 15:53 |
| *** EricGonczer_ has joined #openstack-infra | 15:53 | |
| *** nadya has joined #openstack-infra | 15:54 | |
| inc0 | it's more than just http server for files | 15:54 |
| pabelanger | I am not disputing that, I am suggesting your kolla jobs launch it each time for now | 15:54 |
| pabelanger | rather then rushing to stand one up in infra | 15:54 |
| pabelanger | job launches, download image from tarballs.o.o, start docker-registry, image said image, have your jobs use http://localhost to access docker-registry | 15:55 |
| pabelanger | import said image* | 15:55 |
| inc0 | pabelanger, issue is that we want to have standing registry | 15:55 |
| openstackgerrit | Stéphane Albert proposed openstack-infra/project-config: Add specs repository for CloudKitty https://review.openstack.org/401305 | 15:55 |
| *** andymaier has quit IRC | 15:56 | |
| inc0 | to have access to built images in different gates | 15:56 |
| inc0 | not build it locally at all | 15:56 |
| *** nadya has quit IRC | 15:56 | |
| inc0 | that will allow us to have voting gates | 15:56 |
| *** nadya_ has joined #openstack-infra | 15:56 | |
| pabelanger | how will the way I am suggesting not give voting gates? | 15:56 |
| cloudnull | pabelanger: looking into it now | 15:56 |
| pabelanger | you still only build once | 15:56 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-openstackci: Add a nodepool_builder manifest https://review.openstack.org/400998 | 15:56 |
| pabelanger | cloudnull: thanks | 15:57 |
| *** xarses has joined #openstack-infra | 15:57 | |
| *** drifterza__ has joined #openstack-infra | 15:57 | |
| inc0 | pabelanger, ok, another question | 15:57 |
| *** drifterza__ is now known as drifterza | 15:57 | |
| *** nadya_ has quit IRC | 15:57 | |
| inc0 | can we have afs share? | 15:57 |
| inc0 | that we mount to gates on given path? | 15:57 |
| *** nadya has joined #openstack-infra | 15:58 | |
| *** abregman is now known as abregman|mtg | 15:58 | |
| *** wolverineav has quit IRC | 15:58 | |
| pabelanger | technically yes, we don't have jobs that do it today. | 15:58 |
| *** panda|brb is now known as panda | 15:58 | |
| pabelanger | I know there has been some speed concerns from jeblair in the past | 15:58 |
| inc0 | brb weekly meeting | 15:58 |
| jeblair | inc0: we have avoided putting afs itself on the test nodes, because it's kind of heavy-weight (it changes a lot about the test nodes) -- plus -- since the test nodes are ephemeral, you don't get the caching benefit | 15:58 |
| jeblair | inc0: however.... | 15:59 |
| openstackgerrit | Merged openstack-infra/puppet-openstackci: Use nodepool::builder for nodepool https://review.openstack.org/400993 | 15:59 |
| inc0 | jeblair, pabelanger we could use shared fs and bind local registry to it each time | 15:59 |
| *** makowals has joined #openstack-infra | 15:59 | |
| inc0 | ro for most of gates, rw when we want to build new stuff | 15:59 |
| *** isaacb has quit IRC | 16:00 | |
| jeblair | inc0: for either the case of the docker registry server, or pabelanger's suggestion of hosting image files and using a local docker server, we *can* put those in afs and the worker nodes can transfer the images from a nearby mirror | 16:00 |
| inc0 | but I feel that's going to be harder than jsut hosting single registry somewher | 16:00 |
| inc0 | e | 16:00 |
| *** senk_ has quit IRC | 16:01 | |
| *** Sukhdev_ has joined #openstack-infra | 16:01 | |
| *** drifterza_ has quit IRC | 16:01 | |
| *** jascott1 has joined #openstack-infra | 16:01 | |
| pabelanger | I actually think the download image from tarballs.o.o / local registry is the faster way. Which is why I am suggesting it to fix your voting gates, until we properly setup the docker-registry infra on our side. | 16:01 |
| pabelanger | we already have the post publishers in place | 16:02 |
| *** nadya has quit IRC | 16:02 | |
| jeblair | inc0: yeah, i think i see what you're asking -- and i think i would recommend against running afs on the worker nodes themselves because there will be no cache, and the files will be transferred from a remote server (possibly DFW->CDG) over UDP; it won't be very fast. we've found the best approach is to have long-running afs clients in each region serving things to workers in that region. that way those clients act as a cache. | 16:02 |
| pabelanger | you just need to add wget container and start registry to your jobs | 16:02 |
| *** makowals_ has quit IRC | 16:02 | |
| inc0 | guys, I'll be back here in 1 hr | 16:03 |
| inc0 | thanks! | 16:03 |
| jeblair | inc0: (your idea *would* work -- the speed is the only issue) | 16:03 |
| *** ad_rien_ has quit IRC | 16:03 | |
| inc0 | jeblair, I'd still have central registry somewhere tbh | 16:03 |
| *** strigazi is now known as strigazi_AFK | 16:04 | |
| *** anilvenkata has quit IRC | 16:05 | |
| jeblair | pabelanger: i do like your idea, btw. i think it will solve the problem quickly with minimal infrastructure. | 16:05 |
| jeblair | pabelanger: i think i will land 400998... do you want to continue work in a followup change? | 16:06 |
| pabelanger | jeblair: yes, quick fix, while we churn on our infra needs. | 16:06 |
| pabelanger | jeblair: yes, +2 | 16:06 |
| *** EricGonc_ has joined #openstack-infra | 16:07 | |
| *** rbuzatu has joined #openstack-infra | 16:07 | |
| *** EricGonczer_ has quit IRC | 16:08 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci: WIP TLS everywhere job https://review.openstack.org/391738 | 16:09 |
| *** armax has joined #openstack-infra | 16:09 | |
| *** mpaolino has quit IRC | 16:09 | |
| openstackgerrit | Merged openstack-infra/puppet-openstackci: Add a nodepool_builder manifest https://review.openstack.org/400998 | 16:11 |
| pabelanger | \o/ | 16:11 |
| *** gouthamr has quit IRC | 16:12 | |
| *** wolverineav has joined #openstack-infra | 16:13 | |
| *** gouthamr has joined #openstack-infra | 16:13 | |
| *** pcrews has quit IRC | 16:15 | |
| *** wolverineav has quit IRC | 16:18 | |
| *** Apoorva has joined #openstack-infra | 16:19 | |
| *** makowals has quit IRC | 16:23 | |
| *** wolverineav has joined #openstack-infra | 16:24 | |
| *** fguillot has quit IRC | 16:24 | |
| *** Sukhdev_ has quit IRC | 16:24 | |
| *** jgriffith_away is now known as jgriffith | 16:25 | |
| *** fguillot has joined #openstack-infra | 16:26 | |
| *** andymaier has joined #openstack-infra | 16:27 | |
| *** dkehn__ has joined #openstack-infra | 16:28 | |
| *** mmandel has quit IRC | 16:28 | |
| *** wolverineav has quit IRC | 16:28 | |
| openstackgerrit | Stéphane Albert proposed openstack-infra/project-config: Add specs repository for CloudKitty https://review.openstack.org/401305 | 16:30 |
| *** wolverineav has joined #openstack-infra | 16:30 | |
| *** daemontool has joined #openstack-infra | 16:32 | |
| *** Apoorva has quit IRC | 16:32 | |
| *** dprince has quit IRC | 16:33 | |
| *** ramishra has quit IRC | 16:34 | |
| *** wolverineav has quit IRC | 16:35 | |
| *** lbeliveau has quit IRC | 16:36 | |
| *** senk has joined #openstack-infra | 16:37 | |
| *** weshay is now known as weshay_lunch | 16:37 | |
| *** thorst has joined #openstack-infra | 16:38 | |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-openstackci: Add mysql passwords to nodepool builder https://review.openstack.org/401344 | 16:38 |
| jeblair | pabelanger: ^ | 16:38 |
| *** thorst has quit IRC | 16:39 | |
| *** pcrews has joined #openstack-infra | 16:40 | |
| *** allanice001 has joined #openstack-infra | 16:40 | |
| pabelanger | jeblair: +2 | 16:41 |
| *** ccamacho has quit IRC | 16:41 | |
| *** annegentle has quit IRC | 16:41 | |
| *** yamahata has joined #openstack-infra | 16:42 | |
| *** ramishra has joined #openstack-infra | 16:43 | |
| *** nadya has joined #openstack-infra | 16:43 | |
| *** yamamoto has quit IRC | 16:45 | |
| *** ad_rien_ has joined #openstack-infra | 16:47 | |
| *** wolverineav has joined #openstack-infra | 16:50 | |
| *** hashar has quit IRC | 16:50 | |
| cloudnull | pabelanger: all of the instances in "deleting" have been cleared out. | 16:51 |
| cloudnull | let me know if you see something like that again. | 16:52 |
| pabelanger | cloudnull: for sure, thanks again | 16:52 |
| *** mtanino has joined #openstack-infra | 16:52 | |
| *** dankolbrs has quit IRC | 16:54 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Add nodepool builder nodes https://review.openstack.org/400999 | 16:54 |
| cloudnull | looking into the root cause however it looks like gremlins. a couple of the conductor nodes we're getting connection resets to the DB and after a restart all looks to be happy again. so idk why it was pissed off. maybe residual issues from our ssl misshap | 16:54 |
| *** Kiall has joined #openstack-infra | 16:54 | |
| *** dankolbrs has joined #openstack-infra | 16:54 | |
| cloudnull | rather, compounded issues from our ssl mishap. | 16:54 |
| cloudnull | though that could be a red herring. | 16:55 |
| *** mmandel has joined #openstack-infra | 16:55 | |
| *** wolverineav has quit IRC | 16:55 | |
| *** drifterza has quit IRC | 16:55 | |
| cloudnull | idk. its almost eat till i pass out time so the brain is not firing on the regular 1.25 cylinders | 16:56 |
| *** andymaier has quit IRC | 16:56 | |
| *** ad_rien_ has quit IRC | 16:56 | |
| pabelanger | cloudnull: We started accumulating deletes about the same time as the SSL outage, according to grafana.o.o | 16:56 |
| pabelanger | so, that does seem to make sense | 16:56 |
| cloudnull | I'm keeping an eye on it for a bit to see if it comes back but otherwise it looks like we're back in business. | 16:57 |
| *** rhallisey_ has joined #openstack-infra | 16:57 | |
| *** dprince has joined #openstack-infra | 16:58 | |
| openstackgerrit | Merged openstack-infra/puppet-openstackci: Add mysql passwords to nodepool builder https://review.openstack.org/401344 | 16:58 |
| *** ccamacho has joined #openstack-infra | 16:59 | |
| clarkb | is ssl outage code for letsencrypt updates? | 16:59 |
| cloudnull | clarkb: yup | 17:00 |
| cloudnull | we thought we nailed down the schedule | 17:00 |
| cloudnull | we were wrong | 17:00 |
| clarkb | fungi dulek re that jjb confog thing that is probably a zuul launcher behavior | 17:00 |
| cloudnull | it | 17:00 |
| cloudnull | * it'd be easy if we were using something like haproxy which can run certbot | 17:01 |
| cloudnull | but we have an f5 and certbot is a no go there. (we dont want to run it there) | 17:01 |
| *** cody-somerville has joined #openstack-infra | 17:02 | |
| cloudnull | so we're stuck doing the API calls manually and it would seem that the expire date was forgotten again | 17:02 |
| fungi | similarly, the thread on debian-devel about how the acme protocol and clients like certbot are evolving too fast in non-backward-compatible ways to be suitable for stable distros gives me pause | 17:02 |
| *** xarses has quit IRC | 17:02 | |
| pabelanger | ansible recently added a letsencrypt task to core | 17:03 |
| pabelanger | this week | 17:03 |
| *** xarses has joined #openstack-infra | 17:03 | |
| cloudnull | oh, that's cool | 17:03 |
| pabelanger | https://docs.ansible.com/ansible/letsencrypt_module.html | 17:03 |
| *** SergK has quit IRC | 17:03 | |
| * fungi files all this away in his little black book of why he'll give letsencrypt a few years to mature and then check in on it later | 17:03 | |
| *** SergK has joined #openstack-infra | 17:04 | |
| pabelanger | plan to try it with some ansible things shortly | 17:04 |
| cloudnull | I have it running on a few sites I manage. it works well, and its free. | 17:04 |
| cloudnull | but running it in an env that requires external gear is a nightmare. | 17:04 |
| * pabelanger nods | 17:04 | |
| *** chandankumar has quit IRC | 17:05 | |
| cloudnull | but its a free nightmare. so we'll continue causing ourselves headache ever few months, because free. | 17:05 |
| cloudnull | unless irtermite gets us a real multi-domain cert ;) | 17:06 |
| *** makowals has joined #openstack-infra | 17:06 | |
| * cloudnull holds my breath | 17:06 | |
| openstackgerrit | Mikhail S Medvedev proposed openstack-infra/irc-meetings: Remove third-party-ci working group meeting https://review.openstack.org/401352 | 17:06 |
| *** thorst has joined #openstack-infra | 17:06 | |
| mmedvede | krtaylor: lennyb: fyi ^ | 17:07 |
| *** thorst_ has joined #openstack-infra | 17:07 | |
| *** sdake_ is now known as sdake | 17:09 | |
| notmyname | we've got some questions in swift about using the upper constraints in tox.ini. There's an inline comment on https://review.openstack.org/#/c/354291/ that is beyond my knowledge to answer | 17:09 |
| clarkb | notmyname: looking | 17:10 |
| notmyname | clarkb: thanks :-) | 17:10 |
| *** ggnel_t has quit IRC | 17:10 | |
| *** thorst has quit IRC | 17:11 | |
| *** senk has quit IRC | 17:11 | |
| *** thorst_ has quit IRC | 17:11 | |
| openstackgerrit | Erlon R. Cruz proposed openstack-infra/project-config: Adds new experimental Cinder migration job https://review.openstack.org/381737 | 17:11 |
| *** annegentle has joined #openstack-infra | 17:12 | |
| *** rhallisey_ has quit IRC | 17:12 | |
| *** rhallisey_ has joined #openstack-infra | 17:12 | |
| *** panda is now known as panda|bbl | 17:13 | |
| *** flepied1 has quit IRC | 17:14 | |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-openstackci: Add missing required args to nodepool_builder https://review.openstack.org/401357 | 17:14 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Add nodepool builder nodes https://review.openstack.org/400999 | 17:14 |
| inc0 | pabelanger, jeblair I'm back, sooo technically we could tar-gz directory with all the images | 17:14 |
| inc0 | and on gate setup download-untar-run registry | 17:15 |
| *** jaosorior has quit IRC | 17:15 | |
| inc0 | and on successful merge in kolla - upload tar to tarballs.o.o | 17:15 |
| clarkb | notmyname: posted | 17:16 |
| jlibosva | clarkb: hi | 17:17 |
| notmyname | clarkb: thanks again | 17:17 |
| *** yamahata has quit IRC | 17:17 | |
| clarkb | jlibosva: hi | 17:17 |
| jlibosva | clarkb: we're experiencing some likely an infra issue that blocks some patches that we need to backport to Newton in order to switch to Xenial | 17:18 |
| *** rbuzatu has quit IRC | 17:18 | |
| jlibosva | clarkb: do you have any good contacts that can help with https://bugs.launchpad.net/grenade/+bug/1643485 ? | 17:18 |
| openstack | Launchpad bug 1643485 in grenade "Stable Newton grenade multinode jobs fail with "Failed to connect to the host via ssh: Host key verification failed."" [Critical,New] | 17:18 |
| fungi | jlibosva: i believe pabelanger just fixed it an hour or so ago | 17:18 |
| fungi | though he can confirm | 17:18 |
| jlibosva | really? those are best news today | 17:18 |
| fungi | sounded like we were stuck on old trusty images in a bunch of places | 17:19 |
| *** tjones has joined #openstack-infra | 17:19 | |
| jlibosva | fungi: does it mean we're safe to recheck now? | 17:19 |
| clarkb | notmyname: and one more comment explaining virtauelvn specific things | 17:19 |
| jlibosva | fungi: do you know if there is a bug that is solved so I can close the one I sent as dup? | 17:19 |
| clarkb | jlibosva: yes I think its safe to recheck now | 17:19 |
| fungi | jlibosva: sounded like it to me, but i'd give pabelanger an opportunity to confirm first. he seems to still be around | 17:19 |
| clarkb | pabelanger had said all the bad images were removed | 17:20 |
| pabelanger | clarkb: jlibosva: fungi: yes, I have removed the bad ubuntu-trusty images from our clouds. So, we should be good. However, if there is still failures, I may have missed a cloud. | 17:20 |
| *** e0ne has quit IRC | 17:20 | |
| pabelanger | we should start uploads shortly for images from today | 17:20 |
| fungi | jlibosva: as i understand it, the bug was in an older diskimage-builder release, fixed a few days ago but not updated in all our providers | 17:20 |
| *** knangia has joined #openstack-infra | 17:20 | |
| jlibosva | fungi: clarkb pabelanger cool, thanks guys for fixing it :) | 17:21 |
| pabelanger | yes, that is correct | 17:21 |
| fungi | i don't know what the bug filed against dib was, if tehre was one | 17:21 |
| fungi | to clarify... i know the gist of the problem it had, but don't know if anyone filed a bug report about it formally | 17:22 |
| clarkb | I didn't file a bug | 17:22 |
| pabelanger | no, we didn't file a bug (that I know of) | 17:22 |
| Jeffrey4l | fungi, could u help me on setting up kolla and kolla-ansible cross project testing? i wrote something in ml http://lists.openstack.org/pipermail/openstack-dev/2016-November/107847.html ( sorry for the bad format ) | 17:22 |
| openstackgerrit | Mathieu Rohon proposed openstack-infra/project-config: Make net-bgpvpn jobs run on trusty for mitaka and older https://review.openstack.org/397111 | 17:22 |
| pabelanger | we should try to do that in the future | 17:23 |
| jlibosva | ok, so I'll close that one as expired or invalid | 17:23 |
| *** jlibosva has quit IRC | 17:23 | |
| pabelanger | I'll make a better effort to do that | 17:23 |
| clarkb | my brain had stopped working by that point as evidenced by my failures to debug | 17:24 |
| pabelanger | inc0: I think the first step would be to publish your images, from post, to tarballs.o.o. I can help with that | 17:24 |
| *** jordanP has quit IRC | 17:25 | |
| inc0 | pabelanger, publishing all of them as tar will be super hard | 17:25 |
| inc0 | as tars will squash images and turn it into 100s of gigabytes | 17:25 |
| *** Apoorva has joined #openstack-infra | 17:25 | |
| pabelanger | inc0: what are they now? | 17:25 |
| *** mmandel has quit IRC | 17:26 | |
| fungi | Jeffrey4l: possibly next week, but i'm not really around much for the rest of this week. it's a major holiday here | 17:26 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Add nodepool builder nodes https://review.openstack.org/400999 | 17:27 |
| fungi | Jeffrey4l: the short answe is that if the job checks out all the repos it needs using zuul-cloner or devstack-gate then cross-repo testing should "just work" | 17:27 |
| Jeffrey4l | fungi, ok. or is it possible to give some example link for this? which project is using such feature? | 17:27 |
| Jeffrey4l | kolla and kolla-ansible using zuul-cloner right now. ( but i am not sure the cross project testing works ) | 17:28 |
| inc0 | pabelanger, less than 10gigs | 17:28 |
| *** jamesde__ has joined #openstack-infra | 17:28 | |
| *** abregman|mtg has quit IRC | 17:28 | |
| *** hogepodge has joined #openstack-infra | 17:28 | |
| *** aviau has quit IRC | 17:28 | |
| fungi | Jeffrey4l: you should be able to tell if you add a depends-on header in a kolla-ansible change to a kolla change id and then look at the job logs where zuul-cloner says what refs it's checking out | 17:29 |
| *** aviau has joined #openstack-infra | 17:29 | |
| *** allanice001 has quit IRC | 17:29 | |
| jeblair | Jeffrey4l: do you have a link to a log from a cross-testing job? | 17:29 |
| Jeffrey4l | jeblair, fungi yep. let me find it. | 17:29 |
| fungi | Jeffrey4l: if the job in question testing kolla-ansible consumes kolla via git refs checked out by zuul-cloner then i would expect it to work fine | 17:30 |
| *** matrohon has quit IRC | 17:30 | |
| *** sambetts is now known as sambetts|afk | 17:30 | |
| *** jamesdenton has quit IRC | 17:30 | |
| fungi | if it uses some other mechanism to get kolla installed, then there is probably still work to be done | 17:30 |
| Jeffrey4l | OK. let me check it. | 17:31 |
| pabelanger | inc0: where are the images stored on disk in your build images? | 17:32 |
| *** xarses has quit IRC | 17:32 | |
| *** xarses has joined #openstack-infra | 17:32 | |
| inc0 | pabelanger, /var/lib/docker in general, but that depends on storage driver | 17:32 |
| inc0 | but in registry you can specify directory where to keep them | 17:33 |
| *** lucasagomes is now known as lucas-hungry | 17:33 | |
| *** lucas-hungry is now known as lucas-afk | 17:33 | |
| inc0 | and that's storage-driver independeny | 17:33 |
| *** hogepodge has quit IRC | 17:33 | |
| *** rbuzatu has joined #openstack-infra | 17:34 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Add nodepool builder nodes https://review.openstack.org/400999 | 17:35 |
| *** pdomin has joined #openstack-infra | 17:36 | |
| *** esikachev has quit IRC | 17:37 | |
| clarkb | fungi: do you happen to have a debian instance with facter installed available to you? if so what are operatingsystemrelease and lsbdistrelease facts there? | 17:38 |
| inc0 | also another thing - if we actually run local registry, it wont keep base OS image there, just deltas above | 17:38 |
| inc0 | again - huge disk savings | 17:38 |
| clarkb | inc0: where will it keep the base image? | 17:38 |
| *** rbuzatu has quit IRC | 17:38 | |
| clarkb | (seems like it should have a copy of that too) | 17:38 |
| inc0 | clarkb, dockerhub | 17:38 |
| *** NBziouech has quit IRC | 17:39 | |
| inc0 | dockerhub is fine | 17:39 |
| clarkb | that seems broken if the intent is to cache things for reliability | 17:39 |
| clarkb | (eg if dockerhub is fine for the base image why isn't dockerhub fine for your deltas) | 17:39 |
| fungi | clarkb: installed facter temporarily on my workstation... operatingsystemrelease=stretch/sid lsbdistrelease=unstable | 17:40 |
| inc0 | clarkb, ubuntu base image isn't changing that often, uploading to dockerhub of all our images takes hours | 17:40 |
| fungi | want details from a stable (jessie) machine? | 17:40 |
| inc0 | imagine doing that every build | 17:40 |
| clarkb | fungi: ty | 17:41 |
| jeblair | inc0: *can* it keep a copy of the base image? | 17:41 |
| clarkb | fungi: no trying to figure out if I can use >= comparison on all debuntu or just ubuntu distros | 17:41 |
| clarkb | fungi: looks like just ubuntu | 17:41 |
| inc0 | jeblair, ofc, just don't see much value in this | 17:41 |
| inc0 | but it sure can | 17:41 |
| fungi | clarkb: on jessie i have operatingsystemrelease=8.6 lsbdistrelease=8.6 | 17:42 |
| fungi | so not very consistent between stable and testing/unstable | 17:42 |
| clarkb | fungi: huh, I will just restrict to ubuntu for now | 17:42 |
| clarkb | since thats simple | 17:42 |
| *** weshay_lunch is now known as weshay | 17:42 | |
| jeblair | inc0: generally we like to avoid going out over the internet in jobs if possible, so when we start running a registry in each of our regions, those jobs will be able to get everything they need from a nearby server | 17:42 |
| *** claudiub|2 has quit IRC | 17:42 | |
| *** shardy has quit IRC | 17:43 | |
| *** mkoderer has quit IRC | 17:43 | |
| *** hogepodge has joined #openstack-infra | 17:43 | |
| jeblair | pabelanger: it looks like https://review.openstack.org/400999 and https://review.openstack.org/401357 are ready | 17:44 |
| inc0 | jeblair, ok, we can host it in registry | 17:44 |
| inc0 | but let's get registry first;) | 17:45 |
| *** mhickey has quit IRC | 17:45 | |
| *** yamamoto has joined #openstack-infra | 17:45 | |
| *** daemontool has quit IRC | 17:46 | |
| *** matrohon has joined #openstack-infra | 17:46 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Deploy network envs appropriate for the job https://review.openstack.org/347503 | 17:47 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Test with bonded public interface https://review.openstack.org/385562 | 17:47 |
| openstackgerrit | Clark Boylan proposed openstack-infra/puppet-zanata: Hack to support service on xenial https://review.openstack.org/401363 | 17:49 |
| clarkb | fungi: pabelanger pleia2 lets see if the puppet noop apply job likes ^ | 17:49 |
| *** chlong has quit IRC | 17:50 | |
| *** degorenko is now known as _degorenko|afk | 17:50 | |
| openstackgerrit | Miles Gould proposed openstack/diskimage-builder: Allow installing proliant tools from an OS package https://review.openstack.org/344214 | 17:52 |
| *** derekh has quit IRC | 17:52 | |
| *** yamamoto has quit IRC | 17:52 | |
| *** csomerville has joined #openstack-infra | 17:54 | |
| *** jamesde__ has quit IRC | 17:54 | |
| *** sdake has quit IRC | 17:54 | |
| Jeffrey4l | fungi, jeblair seems it works as expect. magic ;) | 17:54 |
| *** dave-mccowan has joined #openstack-infra | 17:56 | |
| *** cody-somerville has quit IRC | 17:57 | |
| *** dave-mccowan has quit IRC | 17:57 | |
| *** ralonsoh has quit IRC | 18:00 | |
| *** amoralej is now known as amoralej|off | 18:02 | |
| *** hashar has joined #openstack-infra | 18:03 | |
| mgagne | Is there a way to permanently update the path to clouds.yaml in Nodepool? You can set OS_CLIENT_CONFIG_FILE in upstart or systemd but it won't be used by CLI. | 18:03 |
| *** hashar is now known as hasharAway | 18:04 | |
| *** nadya has quit IRC | 18:04 | |
| inc0 | pabelanger, jeblair so out of all ideas we had so far, tar-gz of registry per base distro and uploading it to tarballs.o.o is best | 18:04 |
| inc0 | besides ofc of having dedicated registry | 18:04 |
| *** nadya has joined #openstack-infra | 18:04 | |
| inc0 | pabelanger, let say we'd like to tar-gz dir and upload it to tarballs.o.o every time change to kolla successfully merges to repo | 18:05 |
| inc0 | how would we do this? | 18:05 |
| *** senk has joined #openstack-infra | 18:05 | |
| mordred | inc0: I apologize that I have been AFK more than normal over the last week - did we hit a blocker on having a registry? or are you just looking for solutions in the meantime? | 18:05 |
| inc0 | mordred, blocker - no | 18:05 |
| inc0 | it's super straightforward | 18:06 |
| inc0 | mordred, all it takes (including auth) is this https://review.openstack.org/#/c/401003/ | 18:06 |
| pabelanger | inc0: you'd create a publisher, like we do today for tarballs. jenkins/jobs/macros.yaml in project-config | 18:06 |
| inc0 | (after a fashion) | 18:06 |
| pabelanger | target would be specific to kolla | 18:06 |
| *** mdrabe has quit IRC | 18:06 | |
| mordred | inc0: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/macros.yaml#n1115 is an example of the one we did for debian packages | 18:07 |
| pabelanger | and the new publisher would be attached to a job-template that runs in the post pipeline | 18:07 |
| pabelanger | yup, deb-pkgs is a good example too | 18:07 |
| mordred | inc0: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/deb-build-pkg.yaml#n34 is using that publisher macros | 18:08 |
| *** nadya has quit IRC | 18:08 | |
| inc0 | mordred, we are examining how to achieve registry functionality without changing infra | 18:09 |
| *** drifterza has joined #openstack-infra | 18:09 | |
| *** tosky has quit IRC | 18:09 | |
| *** chandankumar has joined #openstack-infra | 18:11 | |
| inc0 | seems straightforward:) | 18:12 |
| inc0 | publisher part | 18:12 |
| *** senk has quit IRC | 18:13 | |
| *** senk has joined #openstack-infra | 18:14 | |
| *** jpena is now known as jpena|off | 18:15 | |
| mgagne | mordred: is there a way to override the path to clouds.yaml in Nodepool? I find the auto-detection used by os-client-config to be suboptimal for nodepool context. It assumes that you will be running CLI as nodepool user (so it finds config in $HOME) or it assumes your config is global in /etc/openstack which is not something I'm willing to do. | 18:15 |
| pabelanger | fedora-24 doesn't appears to be setting up networking correctly | 18:16 |
| pabelanger | ianw: ^ you know if the issue? | 18:17 |
| *** senk has quit IRC | 18:17 | |
| pabelanger | mgagne: OS_CLIENT_CONFIG_FILE is how I choose a specific clouds.yaml file | 18:18 |
| mgagne | pabelanger: right but you can understand how it's suboptimal | 18:19 |
| *** sreeram_v has joined #openstack-infra | 18:20 | |
| *** xarses has quit IRC | 18:20 | |
| pabelanger | mgagne: agreed, today I use the nodepool user to run all my commands. | 18:20 |
| pabelanger | we store ours in nodepool/.config/openstack today | 18:21 |
| *** xarses has joined #openstack-infra | 18:22 | |
| *** dtantsur is now known as dtantsur|afk | 18:24 | |
| *** sreeram_v has quit IRC | 18:24 | |
| *** tqtran has joined #openstack-infra | 18:28 | |
| mordred | mgagne: it's a good point - I think we can likely add something to the nodepool command to look for it somewhere else ... | 18:28 |
| mgagne | mordred: would there be a way to put that in nodepool.yaml so CLI and daemon can benefit from it? | 18:29 |
| mordred | mgagne: currently exporting OS_CLIENT_CONFIG_FILE is the best bet - but I totally see what you're saying | 18:29 |
| *** chandankumar has quit IRC | 18:29 | |
| mordred | mgagne: yah - I think so | 18:29 |
| *** wolverineav has joined #openstack-infra | 18:29 | |
| mgagne | I don't mind exporting OS_CLIENT_CONFIG_FILE in /etc/environment but that's a huge hack to me | 18:30 |
| pabelanger | I'm starting to think we need some sort of nodepool.conf file, where we can store things like this. Today, we are using CLI args (eg: --upload-workers) but the list is starting to grow. And I'm not sure nodepool.yaml makes the most sense for some of these things, cc clarkb | 18:30 |
| jeblair | i don't see why this one couldn't go in nodepool.yaml | 18:30 |
| *** annegentle has quit IRC | 18:31 | |
| mgagne | pabelanger: true. but I also feel like there are too many config files | 18:31 |
| jeblair | nodepool.yaml *is* the config file for nodepool :) | 18:31 |
| mgagne | secure.conf,nodepool.yaml and clouds.yaml | 18:31 |
| mgagne | and now nodepool.conf | 18:31 |
| pabelanger | jeblair: clarkb has some objections with --upload-worker / build-worker settings in the past, hence why we only did CLI args | 18:31 |
| openstackgerrit | Miles Gould proposed openstack/diskimage-builder: Allow installing proliant tools from an OS package https://review.openstack.org/344214 | 18:31 |
| jeblair | pabelanger: i think those are okay as cli args | 18:31 |
| jeblair | (or could go in a defaults file) | 18:32 |
| mordred | jeblair: yah - I agree - I think an optional clouds_yaml_location (or something) in nodepool.yaml that points to a clouds.yaml would make perfect sense | 18:32 |
| pabelanger | jeblair: Yup, that's what I do today | 18:32 |
| *** electrofelix has quit IRC | 18:33 | |
| jeblair | (the clouds.yaml config is needed to understand the nodepool config, so it makes sense to be able to 'include by reference' as it were) | 18:33 |
| jeblair | pabelanger: can you +3 https://review.openstack.org/401357 and https://review.openstack.org/400999 ? | 18:34 |
| *** wolverin_ has joined #openstack-infra | 18:34 | |
| pabelanger | looking | 18:34 |
| mgagne | clouds-config, clouds-config-file, clouds-file, os-client-config are names I'm thinking about | 18:34 |
| *** ihrachys has joined #openstack-infra | 18:35 | |
| *** markvoelker has quit IRC | 18:35 | |
| clarkb | I would use clouds-config-file I think | 18:36 |
| clarkb | pabelanger: my concern with putting the thread config in nodepool.yaml was that nodepool.yaml should be portable, however thread selection likely isn't based on available bandwidth, io, cpu etc | 18:36 |
| clarkb | pabelanger: so I didn't want to tie a performance optimization toggle to a portable config file | 18:37 |
| pabelanger | clarkb: ack | 18:37 |
| *** wolverineav has quit IRC | 18:38 | |
| openstackgerrit | Merged openstack-infra/puppet-openstackci: Add missing required args to nodepool_builder https://review.openstack.org/401357 | 18:41 |
| *** flwang1 has joined #openstack-infra | 18:41 | |
| *** jgriffith is now known as jgriffith_away | 18:42 | |
| openstackgerrit | Miles Gould proposed openstack/diskimage-builder: Allow installing proliant tools from an OS package https://review.openstack.org/344214 | 18:42 |
| clarkb | greppign for wildfly-systemd-daemon-reload in http://logs.openstack.org/63/401363/1/check/gate-infra-puppet-apply-ubuntu-xenial/3f45747/puppetapplytest63.final.out seems to show that working as expected | 18:42 |
| clarkb | pabelanger: fungi pleia2 maybe we can get quick review on 401363 and we can try to build the translate-dev replacement on xenial again | 18:42 |
| openstackgerrit | Merged openstack-infra/system-config: Add nodepool builder nodes https://review.openstack.org/400999 | 18:43 |
| openstackgerrit | Andrey Kurilin proposed openstack-infra/project-config: [rally] improvements of devstack-rally-gate jobs https://review.openstack.org/401389 | 18:48 |
| andreykurilin | AJaeger: hi! I finally made a patch as you suggested(with right {node} params) - https://review.openstack.org/401389 | 18:49 |
| *** dave-mccowan has joined #openstack-infra | 18:49 | |
| *** rhallisey_ has quit IRC | 18:49 | |
| jeblair | pabelanger: i think we may want to add authn/z to our nodepool zk config soon -- since we're about to have a remote host connecting to it | 18:49 |
| jeblair | pabelanger: i think for the moment, i'll just manually open an iptables port so we can proceed with testing | 18:50 |
| *** newmember has quit IRC | 18:50 | |
| inc0 | jeblair, pabelanger mordred sooo...how do we go with decision? do we make infra registry or hack around tarballs and publishers? | 18:51 |
| inc0 | for what is worth, I'd rather run registry in infra, less hacky | 18:52 |
| *** xarses has quit IRC | 18:53 | |
| AJaeger | amrith: do you have time to discuss https://review.openstack.org/398689 ? | 18:54 |
| *** wolverin_ has quit IRC | 18:55 | |
| *** yamahata has joined #openstack-infra | 18:55 | |
| *** wolverineav has joined #openstack-infra | 18:55 | |
| jeblair | inc0: i'm not involved enough to advocate for a decision one way or the other | 18:55 |
| clarkb | jeblair: after a quick zk docs read on how to set up ssl I think thats probably for the best currently. None of the data is actually private so we don't need encryption until we auth | 18:56 |
| openstackgerrit | Andrey Kurilin proposed openstack-infra/project-config: [rally] improvements of devstack-rally-gate jobs https://review.openstack.org/401389 | 18:56 |
| *** xarses has joined #openstack-infra | 18:56 | |
| clarkb | (looks like you have to change from NIO to netty and configure a bunch of java ssl store stuff that I bet we get wrong the first time or two if trying to jam it in today) | 18:56 |
| jeblair | clarkb: oh nice. then yes. :) | 18:56 |
| inc0 | jeblair, fair enough:) thanks for help anyway | 18:57 |
| jeblair | inc0: np. btw, did you see my comments on your change? | 18:57 |
| inc0 | fungi, ^ couple lines above | 18:57 |
| AJaeger | andreykurilin: great! | 18:57 |
| inc0 | yeah, if we decide to run registry in infra I'll need to find someone who will puppetize this anyway | 18:57 |
| clarkb | jeblair: https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide | 18:58 |
| inc0 | as it's by no means ready to merge | 18:58 |
| clarkb | what is a .jks? | 18:58 |
| inc0 | we need some mechanism for secrets and well, make it puppet-working | 18:58 |
| jeblair | clarkb: key store? | 18:58 |
| pabelanger | clarkb: +2 | 18:58 |
| clarkb | jeblair: ya I think its some special java key thing | 18:58 |
| jeblair | clarkb, pabelanger: 4G or 8G instance? | 18:59 |
| *** dave-mcc_ has joined #openstack-infra | 18:59 | |
| pabelanger | clarkb: jeblair: is there a different is upload speeds? | 18:59 |
| clarkb | jeblair: sicne we can't use tmpfs anyways, I don't think we need a ton of memory | 18:59 |
| clarkb | oh but ya there is a difference in network bw | 18:59 |
| pabelanger | I think 4g is enough, but if 8gb is the faster network we should do that | 19:00 |
| *** wolverineav has quit IRC | 19:00 | |
| clarkb | my local build VM is a 1cpu 2GB ram machine and it does ok. Most of the bottleneck is in disk iop | 19:00 |
| jeblair | huh, i don't see network stuff in 'openstack flavor list' | 19:00 |
| clarkb | where you can see memory help dib is if your image is small enough to build in a tmpfs (ours are not) | 19:01 |
| *** cathy_ has joined #openstack-infra | 19:01 | |
| openstackgerrit | Andrey Kurilin proposed openstack-infra/project-config: [rally] improvements of devstack-rally-gate jobs https://review.openstack.org/401389 | 19:01 |
| jeblair | btw, i just noticed this in flavor-list: | 19:01 |
| jeblair | | container1-128M | 128 MB Container | 128 | 2 | 0 | 12 | N/A | | 19:01 |
| *** ad_rien_ has joined #openstack-infra | 19:01 | |
| *** dave-mccowan has quit IRC | 19:02 | |
| pabelanger | :) | 19:02 |
| jeblair | ah, rxtx factor is in --long | 19:02 |
| cathy_ | hi, could anyone help us get a channel "networking-sfc" registered for networking-sfc project discussion? | 19:02 |
| jeblair | 800 vs 1600 | 19:03 |
| AJaeger | cathy_: let me get you a link... | 19:03 |
| openstackgerrit | Merged openstack-infra/project-config: Run {pipeline}-tempest-dsvm-networking-calico on Xenial https://review.openstack.org/400975 | 19:03 |
| AJaeger | cathy_: http://docs.openstack.org/infra/system-config/irc.html | 19:03 |
| openstackgerrit | Merged openstack-infra/project-config: Enable Nimble tempest plugin test https://review.openstack.org/400539 | 19:04 |
| cathy_ | AJaeger: thanks. will take a look | 19:04 |
| jeblair | clarkb, pabelanger: okay, how do we decipher that value again? given rxtx==1600, what's our max up/down throughput? | 19:05 |
| AJaeger | pabelanger: there's a grafana change that I'm not sure whether it's correct - could you put it on your low-prio review queue, please? https://review.openstack.org/400654 | 19:05 |
| jeblair | https://www.rackspace.com/cloud/public-pricing/#cloud-servers is not helping on account of meteor showers or something | 19:07 |
| clarkb | jeblair: I think its half that for public internets | 19:07 |
| clarkb | jeblair: in megabits per second | 19:07 |
| jeblair | clarkb: we seem to be topping out at 400mbps on our 8g server which has rxtx=1600 | 19:07 |
| jeblair | clarkb: so we're only using half? | 19:08 |
| *** ad_rien_ has quit IRC | 19:08 | |
| clarkb | it might be half 1600 = 800 then half 800 for up and down? | 19:09 |
| pabelanger | https://developer.rackspace.com/docs/user-guides/infrastructure/cloud-config/compute/cloud-servers-product-concepts/create-server/#rxtx-factor says 40% | 19:10 |
| pabelanger | is 640Mb/s | 19:10 |
| jeblair | so if we went with 4g, we would be limited to 320, which is less than what we are pushing today | 19:11 |
| pabelanger | Ya, we can try it | 19:11 |
| pabelanger | I have no objections | 19:11 |
| jeblair | well, we are most concerned with upload speed, right? are we sure we want to do that? | 19:11 |
| *** nherciu has joined #openstack-infra | 19:12 | |
| jeblair | i'm leaning toward 8g so we can keep upload parity | 19:12 |
| clarkb | ya I think if we want to also host the images publicly then we will want more bw | 19:12 |
| clarkb | ++ to 8G | 19:12 |
| pabelanger | sure, I was saying we could try 4GB, get stats, if the upload speed is not enough, rebuild as 8GB. I'm open to both. | 19:13 |
| jeblair | xenial or trusty? | 19:13 |
| pabelanger | xenial would be nice | 19:14 |
| clarkb | we are trusty today, both should work (though puppeting xenial is my new nemesis due to systemd) | 19:14 |
| *** jgriffith_away is now known as jgriffith | 19:14 | |
| *** sourav has joined #openstack-infra | 19:14 | |
| jeblair | i accidentally launched it as trusty. i may just leave it that way for now. :) | 19:14 |
| clarkb | jeblair: I did that too yesterday | 19:15 |
| clarkb | the launch node readme doesn't tell you to set an image for servers | 19:16 |
| clarkb | and default is trusty | 19:16 |
| openstackgerrit | Stéphane Albert proposed openstack-infra/project-config: Add specs repository for CloudKitty https://review.openstack.org/401305 | 19:16 |
| openstackgerrit | sebastian marcet proposed openstack-infra/puppet-openstackid: Puppet script upgrade to LV 5.X https://review.openstack.org/399253 | 19:16 |
| jeblair | yep; i guess we should change the default | 19:16 |
| *** lin_yang has joined #openstack-infra | 19:17 | |
| *** asettle has quit IRC | 19:19 | |
| *** sourav has quit IRC | 19:21 | |
| *** esikachev has joined #openstack-infra | 19:21 | |
| jeblair | what does $group in puppet do? | 19:22 |
| clarkb | jeblair: its what tells puppet what its hiera group is | 19:22 |
| clarkb | jeblair: on the ansible side we set the group for ansible grouping, and in puppet we set $group to map between them ensuring the correct hiera file are copied onto the host | 19:22 |
| jeblair | clarkb: i think the ansible group.txt ensures the file is copied.... do we also need to set it in puppet to use the file after it's copied? | 19:23 |
| clarkb | jeblair: yes, so that puppet knows where to look for the file | 19:23 |
| jeblair | got it | 19:23 |
| jeblair | <sigh> | 19:23 |
| clarkb | jeblair: if you look in our hiera config file it uses the group var as part of the path lookup | 19:23 |
| jeblair | we should be able to remove that | 19:24 |
| openstackgerrit | Isaku Yamahata proposed openstack-infra/project-config: networking-odl: graph for unit test in gate queue https://review.openstack.org/401398 | 19:24 |
| *** dtardivel has quit IRC | 19:24 | |
| *** jamesdenton has joined #openstack-infra | 19:25 | |
| *** mmandel has joined #openstack-infra | 19:26 | |
| *** smarcet has joined #openstack-infra | 19:27 | |
| *** ad_rien_ has joined #openstack-infra | 19:27 | |
| pabelanger | excitement for nb01.o.o mounts | 19:28 |
| *** makowals has quit IRC | 19:29 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Add nodepool hiera group https://review.openstack.org/401399 | 19:29 |
| openstackgerrit | Merged openstack-infra/storyboard: Fix install instructions for MySQL https://review.openstack.org/397341 | 19:30 |
| *** mmandel has quit IRC | 19:30 | |
| jeblair | clarkb, pabelanger: https://review.openstack.org/401399 | 19:30 |
| smarcet | fungi:yolanda: afternoon , when u have a chance please review https://review.openstack.org/#/c/399253/4, upgrades openstackid puppet to use laravel 5.x and mantains backguard compatiblity with laravel 4.x ( production version) i would like to test it first only on dev, any chance to pause the puppet agent for openstackid production in order to test it ? | 19:30 |
| *** abregman has joined #openstack-infra | 19:31 | |
| *** jamesdenton has quit IRC | 19:31 | |
| clarkb | jeblair: +2 with a comment on where to pu the group hiera yaml file | 19:32 |
| pabelanger | +3 | 19:32 |
| jeblair | clarkb: yeah, i just copied the fqdn to that file; will remove the fqdn once this lands | 19:32 |
| clarkb | jeblair: ++ thats what I did for translate-dev | 19:32 |
| pabelanger | will be dismounting train shortly, if pings to me go unanswered for a bit | 19:33 |
| inc0 | pabelanger, fungi mordred so ad tarball.o.o we just had discussion in kolla | 19:33 |
| inc0 | and if we could rsync instead of pack tarball | 19:34 |
| inc0 | we'd save *a lot* of network traffic | 19:34 |
| *** flwang1 has quit IRC | 19:34 | |
| inc0 | or...well registry;) | 19:34 |
| *** pmalik has left #openstack-infra | 19:35 | |
| *** smarcet has quit IRC | 19:35 | |
| openstackgerrit | Merged openstack-infra/puppet-zanata: Hack to support service on xenial https://review.openstack.org/401363 | 19:36 |
| *** abregman is now known as abregman|afk | 19:38 | |
| amrith | AJaeger, PING | 19:38 |
| AJaeger | Hi amrith ! | 19:39 |
| amrith | JUST SAW YOUR EMAIL (SORRY, CAPS LOCK IS STUCK) | 19:39 |
| amrith | ok, different PC, works better | 19:39 |
| amrith | will you be around for a couple of minutes? | 19:39 |
| AJaeger | amrith: looking at https://review.openstack.org/#/c/398689/ : I would expect in that change the following: | 19:39 |
| AJaeger | amrith: http://docs.openstack.org/infra/manual/drivers.html#remove-project-from-infrastructure-systems and I would expect a governance change http://docs.openstack.org/infra/manual/drivers.html#remove-repository-from-the-governance-repository | 19:40 |
| AJaeger | which parts of the project-config change are not clear? | 19:40 |
| *** hasharAway is now known as hashar | 19:40 | |
| AJaeger | I see only a change for gerritbot/channels.yaml but that section lists several other files you need to touch | 19:41 |
| AJaeger | IMHO that's the step you need to do now to *finish* retiring the repo | 19:41 |
| AJaeger | amrith: still with me on your caps-lock less PC? ;) | 19:42 |
| amrith | yes | 19:42 |
| amrith | one second | 19:42 |
| AJaeger | sure... | 19:42 |
| openstackgerrit | Merged openstack-infra/system-config: Add nodepool hiera group https://review.openstack.org/401399 | 19:42 |
| openstackgerrit | Sujitha proposed openstack-infra/devstack-gate: Do not merge: Patch to test multi api setup https://review.openstack.org/394659 | 19:42 |
| amrith | AJaeger, I have done http://docs.openstack.org/infra/manual/drivers.html#end-project-gating and http://docs.openstack.org/infra/manual/drivers.html#remove-project-content | 19:43 |
| amrith | I think the commit you see is https://review.openstack.org/#/c/398689/4 which is http://docs.openstack.org/infra/manual/drivers.html#end-project-gating | 19:43 |
| amrith | Once this merges, and https://review.openstack.org/#/c/399697/ has merged | 19:44 |
| amrith | I will push another change with the http://docs.openstack.org/infra/manual/drivers.html#remove-project-from-infrastructure-systems and beyond | 19:44 |
| AJaeger | amrith: that document explains two steps - the first step to enable 399697 to merge and thene the final retirement. | 19:44 |
| AJaeger | You can merge those two steps in one. | 19:44 |
| *** smarcet has joined #openstack-infra | 19:44 | |
| AJaeger | amrith: 399697 has merged. So, enhance your change with the description from http://docs.openstack.org/infra/manual/drivers.html#remove-project-from-infrastructure-systems | 19:45 |
| *** jamesdenton has joined #openstack-infra | 19:45 | |
| clarkb | ok puppetmaster should update puppet-zanata module shortly, will try booting a new translate-dev01 once that is in place | 19:45 |
| clarkb | then I need to go start doing food prep | 19:46 |
| AJaeger | amrith: no need for two changes... If you really insist, go for it - but your description in the commit message of 398689 makes it really final. And that's why I'll stay with my -1. If you merge them, then the commit message is fine. I only see that single change with a very final commit message - and just seeing that, I'll stay with my -1. Does that explain it? | 19:47 |
| *** panda|bbl is now known as panda | 19:49 | |
| amrith | OK, I'll push one change with the steps of project config | 19:49 |
| AJaeger | thanks, amrith | 19:49 |
| *** dave-mcc_ has quit IRC | 19:49 | |
| *** betherly_ has joined #openstack-infra | 19:50 | |
| *** lbeliveau has joined #openstack-infra | 19:51 | |
| openstackgerrit | Vitaly Gridnev proposed openstack-infra/project-config: [sahara] finish transitioning scenario jobs to xenial https://review.openstack.org/400814 | 19:56 |
| *** ad_rien_ has quit IRC | 19:56 | |
| *** nadya has joined #openstack-infra | 19:56 | |
| *** smarcet has quit IRC | 19:57 | |
| *** wolverineav has joined #openstack-infra | 20:00 | |
| *** cathy_ has quit IRC | 20:02 | |
| *** jpich has quit IRC | 20:03 | |
| *** smarcet has joined #openstack-infra | 20:05 | |
| *** rods has quit IRC | 20:05 | |
| *** wolverin_ has joined #openstack-infra | 20:05 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Update Angular from 1.3.13 to 1.5.8 https://review.openstack.org/384532 | 20:07 |
| *** wolverineav has quit IRC | 20:07 | |
| *** rods has joined #openstack-infra | 20:07 | |
| *** ihrachys has quit IRC | 20:07 | |
| mgagne | Is there a way to get a resolution on this review? It adds the ability for jeepyb to manage All-Projects: https://review.openstack.org/#/c/248130/ | 20:08 |
| *** lseki has quit IRC | 20:08 | |
| openstackgerrit | amrith proposed openstack-infra/project-config: Get rid of the trove-image-builder project https://review.openstack.org/398689 | 20:08 |
| clarkb | hrm puppet still broken, I think I may have to call it a week | 20:08 |
| amrith | AJaeger, I think this is the one-step-everything in project-config. | 20:09 |
| clarkb | wildfly wins this one | 20:09 |
| *** jamesdenton has quit IRC | 20:09 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Make subheading more succint https://review.openstack.org/399195 | 20:09 |
| *** oanson has quit IRC | 20:09 | |
| amrith | and to all in the US, a happy long holiday weekend. | 20:10 |
| *** ijw has joined #openstack-infra | 20:11 | |
| *** drifterza has quit IRC | 20:11 | |
| *** jamesdenton has joined #openstack-infra | 20:13 | |
| *** tjones has left #openstack-infra | 20:13 | |
| *** ihrachys has joined #openstack-infra | 20:13 | |
| *** wolverineav has joined #openstack-infra | 20:13 | |
| *** ihrachys has quit IRC | 20:14 | |
| *** wolverineav has quit IRC | 20:14 | |
| *** ihrachys has joined #openstack-infra | 20:14 | |
| *** wolverineav has joined #openstack-infra | 20:15 | |
| *** wolverineav has quit IRC | 20:16 | |
| *** inc0 has quit IRC | 20:16 | |
| clarkb | doesn't look like the daemon-reload exec ran at all, wonder if the initial module install failed for some reason | 20:16 |
| clarkb | will have to pick up on monday | 20:16 |
| *** ldnunes has quit IRC | 20:16 | |
| *** wolverin_ has quit IRC | 20:17 | |
| *** abregman|afk has quit IRC | 20:21 | |
| *** allanice001 has joined #openstack-infra | 20:21 | |
| *** e0ne has joined #openstack-infra | 20:22 | |
| openstackgerrit | Carlos Camacho proposed openstack-infra/tripleo-ci: Add Panko to scenario001 https://review.openstack.org/401299 | 20:22 |
| *** wolverineav has joined #openstack-infra | 20:23 | |
| *** wolverineav has quit IRC | 20:27 | |
| *** wolverineav has joined #openstack-infra | 20:27 | |
| *** wolverin_ has joined #openstack-infra | 20:29 | |
| *** wolverineav has quit IRC | 20:29 | |
| *** wolverin_ has quit IRC | 20:30 | |
| *** wolverineav has joined #openstack-infra | 20:31 | |
| *** wolverineav has quit IRC | 20:33 | |
| pabelanger | back home now | 20:33 |
| *** nicolasbock has quit IRC | 20:33 | |
| *** e0ne has quit IRC | 20:34 | |
| *** wolverineav has joined #openstack-infra | 20:35 | |
| pabelanger | Jeffrey4l: inc0: we do use rsync today for publisher, even thought it is listed as scp for JJB. So, you'll get some network savings. | 20:35 |
| *** vaidy has quit IRC | 20:36 | |
| *** Sukhdev has joined #openstack-infra | 20:36 | |
| *** isviridov_away has quit IRC | 20:36 | |
| openstackgerrit | Thomas Herve proposed openstack-infra/project-config: Add a amqp1 job for Heat https://review.openstack.org/400799 | 20:37 |
| *** wolverin_ has joined #openstack-infra | 20:37 | |
| *** flwang1 has joined #openstack-infra | 20:39 | |
| *** wolveri__ has joined #openstack-infra | 20:40 | |
| *** wolverineav has quit IRC | 20:40 | |
| *** betherly_ is now known as betherly | 20:41 | |
| *** wolverin_ has quit IRC | 20:42 | |
| *** armax has quit IRC | 20:43 | |
| *** wolveri__ has quit IRC | 20:44 | |
| *** armax has joined #openstack-infra | 20:45 | |
| *** armax has quit IRC | 20:45 | |
| *** csomerville has quit IRC | 20:45 | |
| *** dprince has quit IRC | 20:46 | |
| *** newmember has joined #openstack-infra | 20:47 | |
| *** gouthamr has quit IRC | 20:50 | |
| *** nadya has quit IRC | 20:50 | |
| *** baoli has quit IRC | 20:51 | |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-nodepool: Conditionally set mysql max connections https://review.openstack.org/401417 | 20:51 |
| *** baoli has joined #openstack-infra | 20:51 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Fix nodepool builder branch name https://review.openstack.org/401419 | 20:52 |
| jeblair | pabelanger: ^ can you look at those two? | 20:52 |
| *** flaper87 has quit IRC | 20:52 | |
| pabelanger | sure | 20:53 |
| *** cody-somerville has joined #openstack-infra | 20:53 | |
| *** cody-somerville has quit IRC | 20:53 | |
| *** cody-somerville has joined #openstack-infra | 20:53 | |
| jeblair | pabelanger: we pull all the data from the node to the launcher regardless (even though we use rsync, the target is always empty for that stage of the process) | 20:53 |
| *** sdake has joined #openstack-infra | 20:54 | |
| pabelanger | jeblair: +2 | 20:54 |
| pabelanger | jeblair: that is true | 20:54 |
| jeblair | This message is to inform you that our monitoring systems have detected a problem with the server which hosts your Cloud Block Storage device 'static.openstack.org/main05' at 20:41 UTC. We are currently investigating the issue and will update you as soon as we have additional information regarding the alert. Please do not access or modify 'static.openstack.org/main05' during this process. | 20:55 |
| jeblair | infra-root: ^ | 20:55 |
| jeblair | (from 14 minutes ago) | 20:56 |
| *** baoli has quit IRC | 20:56 | |
| jeblair | #status log received at 20:41:09 UTC: This message is to inform you that our monitoring systems have detected a problem with the server which hosts your Cloud Block Storage device 'static.openstack.org/main05' at 20:41 UTC. We are currently investigating the issue and will update you as soon as we have additional information regarding the alert. Please do not access or modify 'static.openstack.org/main05' during this process. | 20:57 |
| openstackstatus | jeblair: finished logging | 20:57 |
| jeblair | indeed, static is throwing errors about /dev/xvdg | 20:59 |
| *** esikachev has quit IRC | 21:00 | |
| jeblair | and it's been remounted read-only | 21:00 |
| jeblair | and zuul is reporting post_failures | 21:00 |
| *** wolverineav has joined #openstack-infra | 21:01 | |
| *** Guest2 has joined #openstack-infra | 21:02 | |
| *** Guest2 has quit IRC | 21:02 | |
| jeblair | status alert Due to a problem with the cinder volume backing the log server, jobs are failing with POST_FAILURE. Please avoid issuing 'recheck' commands until the issue is resolved. | 21:02 |
| jeblair | how does that look? | 21:03 |
| *** wolverineav has quit IRC | 21:04 | |
| *** wolverineav has joined #openstack-infra | 21:05 | |
| *** Guest2 has joined #openstack-infra | 21:05 | |
| jeblair | #status alert Due to a problem with the cinder volume backing the log server, jobs are failing with POST_FAILURE. Please avoid issuing 'recheck' commands until the issue is resolved. | 21:05 |
| openstackstatus | jeblair: sending alert | 21:05 |
| *** wolverineav has quit IRC | 21:06 | |
| *** ihrachys has quit IRC | 21:06 | |
| -openstackstatus- NOTICE: Due to a problem with the cinder volume backing the log server, jobs are failing with POST_FAILURE. Please avoid issuing 'recheck' commands until the issue is resolved. | 21:07 | |
| *** ChanServ changes topic to "Due to a problem with the cinder volume backing the log server, jobs are failing with POST_FAILURE. Please avoid issuing 'recheck' commands until the issue is resolved." | 21:07 | |
| *** wolverineav has joined #openstack-infra | 21:08 | |
| *** wolverin_ has joined #openstack-infra | 21:09 | |
| *** wolveri__ has joined #openstack-infra | 21:10 | |
| openstackstatus | jeblair: finished sending alert | 21:11 |
| *** ad_rien_ has joined #openstack-infra | 21:11 | |
| *** wolverineav has quit IRC | 21:13 | |
| *** inc0 has joined #openstack-infra | 21:13 | |
| *** wolverin_ has quit IRC | 21:14 | |
| inc0 | fungi, around? | 21:14 |
| *** wolveri__ has quit IRC | 21:15 | |
| *** wolverineav has joined #openstack-infra | 21:15 | |
| *** rbuzatu has joined #openstack-infra | 21:16 | |
| *** Apoorva has quit IRC | 21:16 | |
| *** jamesdenton has quit IRC | 21:18 | |
| *** wolverineav has quit IRC | 21:20 | |
| *** rbuzatu has quit IRC | 21:20 | |
| mnaser | would anyone know where i can find sample uses of this? https://github.com/openstack-infra/ansible-role-puppet | 21:20 |
| mnaser | more around how to make it find the hiera data in the right places | 21:21 |
| rcarrillocruz | mnaser: you should check the system-config repo | 21:21 |
| rcarrillocruz | there you'll find the hiera used | 21:21 |
| jeblair | mnaser: http://git.openstack.org/cgit/openstack-infra/system-config/tree/playbooks | 21:21 |
| *** wolverineav has joined #openstack-infra | 21:22 | |
| *** jkilpatr_ has quit IRC | 21:22 | |
| *** jamielennox|away is now known as jamielennox | 21:22 | |
| mnaser | okay great, thank you! we'll try to adopt this in a way of using ansible while maintaining all of our puppet stuff (and getting rid of our master as well) | 21:22 |
| jeblair | mnaser: re hiera specifically -- http://git.openstack.org/cgit/openstack-infra/system-config/tree/playbooks/group_vars/all.yaml | 21:22 |
| mnaser | hm, that looks like it copies all hieradata.. does that mean every VM has all of the hieradata of everything in the infra? | 21:23 |
| mnaser | i was hoping of a way so that only the role specific ones get shipped | 21:23 |
| *** fguillot has quit IRC | 21:24 | |
| jeblair | mnaser: no, our setup copies common hiera, hiera matching the fqdn, and uses ansible groups to copy group hiera | 21:24 |
| mnaser | oh yeah, i was just looking at it and that sounds ideal | 21:24 |
| *** csomerville has joined #openstack-infra | 21:24 | |
| jeblair | yeah, what you described is exactly what we wanted -- masterless puppet, but still scoped secrets | 21:24 |
| *** wolverineav has quit IRC | 21:25 | |
| *** wolverineav has joined #openstack-infra | 21:25 | |
| *** csomerville has quit IRC | 21:25 | |
| *** csomerville has joined #openstack-infra | 21:25 | |
| jeblair | mnaser: here's the default path it looks in: https://github.com/openstack-infra/ansible-role-puppet/blob/master/defaults/main.yml | 21:26 |
| jeblair | mnaser: so you'll want to set those vars | 21:26 |
| *** wolverin_ has joined #openstack-infra | 21:26 | |
| jeblair | mnaser: and here's the code that gets the actual files: https://github.com/openstack-infra/ansible-role-puppet/blob/master/library/puppet_get_hiera_file_list#L42 | 21:27 |
| *** armax has joined #openstack-infra | 21:27 | |
| *** cody-somerville has quit IRC | 21:28 | |
| jeblair | mnaser: so it will fetch $puppet_hiera_datadir/common.yaml, $puppet_hiera_datadir/fqdn/$fqdn.yaml, $puppet_hiera_datadir/group/$group.yaml | 21:28 |
| jhesketh | Morning | 21:28 |
| mnaser | jeblair: probably like to follow along what you do for as much as we can. if i can bother you for a few more questions .. how are you handling external modules (ex: Puppetfile-like stuff?) .. and I assume you have another repo to store all the hiera secrets | 21:28 |
| jeblair | jhesketh: morning. cinder failure on static. see topic. | 21:28 |
| jhesketh | yeah, just saw that | 21:28 |
| jhesketh | jeblair: are you poking at it? | 21:29 |
| jeblair | jhesketh: still waiting for resolution from rax | 21:29 |
| jhesketh | hmm okay | 21:30 |
| mnaser | (and as silly as that sounds, system-config is the repo for your .. system-config, for sakes of name convention, what's the name of choice for your hiera one?) | 21:30 |
| *** wolverineav has quit IRC | 21:30 | |
| jeblair | mnaser: the secrets are local-only on our "puppetmaster" (the host we run this ansible from) | 21:30 |
| *** cody-somerville has joined #openstack-infra | 21:30 | |
| *** cody-somerville has joined #openstack-infra | 21:30 | |
| *** allanice001 has quit IRC | 21:30 | |
| jeblair | mnaser: and we have a script to install all the puppet modules (which the role will copy over) | 21:30 |
| mnaser | i see, so some sort of bastion host where you kick ansible off-of | 21:30 |
| jeblair | yep | 21:30 |
| *** rossella_s has quit IRC | 21:31 | |
| mnaser | makes sense! i guess we can just write a small script to install r10k and reuse our existing puppetfile in that case | 21:31 |
| *** wolverin_ has quit IRC | 21:31 | |
| *** rossella_s has joined #openstack-infra | 21:32 | |
| jeblair | mnaser: this is the entrypoint for the whole thing: http://git.openstack.org/cgit/openstack-infra/system-config/tree/run_all.sh | 21:32 |
| *** csomerville has quit IRC | 21:32 | |
| mnaser | and is that manually kicked off at the moment or is it cobbled up to CI so once things merge it just calls it (in an ideal world we could reach that..) | 21:33 |
| jeblair | mnaser: which runs this playbook http://git.openstack.org/cgit/openstack-infra/system-config/tree/playbooks/update_puppet.yaml which runs this script to install modules: http://git.openstack.org/cgit/openstack-infra/system-config/tree/install_modules.sh | 21:34 |
| *** ijw has quit IRC | 21:34 | |
| jeblair | mnaser: it's a cronjob (puppet for our 'puppetmaster' configures the cronjob :) | 21:34 |
| mnaser | gotcha | 21:34 |
| jeblair | mnaser: in zuulv3, we will probably have zuul run it | 21:34 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/devstack-gate: Add role for fix_etc_hosts https://review.openstack.org/383052 | 21:34 |
| *** Jeffrey4l has quit IRC | 21:35 | |
| jeblair | jhesketh: at least, i haven't gotten a followup email from rax, and static.o.o is still spewing read io errors | 21:35 |
| jeblair | so i assume it's still broken | 21:35 |
| *** Jeffrey4l has joined #openstack-infra | 21:35 | |
| *** xarses has quit IRC | 21:36 | |
| *** xarses has joined #openstack-infra | 21:37 | |
| *** ijw has joined #openstack-infra | 21:37 | |
| *** jkilpatr_ has joined #openstack-infra | 21:38 | |
| *** rbuzatu has joined #openstack-infra | 21:40 | |
| *** esikachev has joined #openstack-infra | 21:40 | |
| *** pradk has quit IRC | 21:43 | |
| jhesketh | jeblair: that's a fair assumption | 21:43 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/devstack-gate: Add gather_host_info playbook and role https://review.openstack.org/401454 | 21:47 |
| *** thorst_ has joined #openstack-infra | 21:47 | |
| *** acoles has joined #openstack-infra | 21:49 | |
| acoles | clarkb: thanks for your replies on https://review.openstack.org/#/c/354291/ | 21:50 |
| *** yolanda has quit IRC | 21:50 | |
| *** kgiusti has left #openstack-infra | 21:51 | |
| *** wolverineav has joined #openstack-infra | 21:54 | |
| *** jcoufal has quit IRC | 21:55 | |
| *** aeng_afk has quit IRC | 21:58 | |
| *** aeng has joined #openstack-infra | 21:58 | |
| *** esikachev has quit IRC | 21:58 | |
| *** aeng has quit IRC | 21:58 | |
| *** wolverineav has quit IRC | 21:59 | |
| *** xarses has quit IRC | 21:59 | |
| *** xarses has joined #openstack-infra | 22:00 | |
| *** wolverineav has joined #openstack-infra | 22:01 | |
| *** boden has quit IRC | 22:02 | |
| *** Guest2 has quit IRC | 22:03 | |
| *** tpsilva has quit IRC | 22:05 | |
| *** wolverineav has quit IRC | 22:05 | |
| *** wolverineav has joined #openstack-infra | 22:06 | |
| ianw | sorry, running late today ... let me know if i can help | 22:08 |
| *** e0ne has joined #openstack-infra | 22:09 | |
| jeblair | still no okay email, and still seeing read errors. the rax status page has been green the whole time. | 22:09 |
| *** hashar has quit IRC | 22:10 | |
| *** acoles is now known as acoles_ | 22:11 | |
| *** ccamacho has quit IRC | 22:11 | |
| *** wolverineav has quit IRC | 22:11 | |
| ianw | "Please do not access or modify 'static.openstack.org/main05' during this process." i'm guessing we have about zero chance of unmounting it | 22:12 |
| jeblair | ianw: we probably could, but the kernel helpfully remounted it ro for us a while ago | 22:13 |
| jeblair | so we're certainly not modifying it | 22:13 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/devstack-gate: WIP Add setup_host playbook https://review.openstack.org/391519 | 22:13 |
| *** rbuzatu_ has joined #openstack-infra | 22:14 | |
| ianw | oh, read errors, cool | 22:14 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/devstack-gate: WIP Add setup_host playbook https://review.openstack.org/391519 | 22:15 |
| *** rbuzatu has quit IRC | 22:17 | |
| *** adriant has joined #openstack-infra | 22:17 | |
| *** xarses has quit IRC | 22:18 | |
| *** slagle has quit IRC | 22:18 | |
| jeblair | oh hey syslog got quieter | 22:19 |
| *** chlong has joined #openstack-infra | 22:19 | |
| *** thorst_ has quit IRC | 22:20 | |
| jeblair | jhesketh, ianw: lack of read errors suggests maybe rax fixed the cinder host? | 22:21 |
| *** saibarspeis has joined #openstack-infra | 22:21 | |
| *** sreeram_v has joined #openstack-infra | 22:21 | |
| *** saibarspeis has quit IRC | 22:21 | |
| openstackgerrit | amrith proposed openstack-infra/infra-manual: update the drivers.rst for project retirement steps https://review.openstack.org/401459 | 22:21 |
| *** aeng has joined #openstack-infra | 22:22 | |
| openstackgerrit | amrith proposed openstack-infra/project-config: Get rid of the trove-image-builder project https://review.openstack.org/398689 | 22:22 |
| openstackgerrit | amrith proposed openstack-infra/infra-manual: update the drivers.rst for project retirement steps https://review.openstack.org/401459 | 22:23 |
| *** sreeram_v has quit IRC | 22:26 | |
| jeblair | jhesketh, ianw: no okay email yet... should we wait a bit more for that, or assume we can proceed with fsck, etc? | 22:27 |
| ianw | jeblair: yeah, my simultaneous "for i in /srv/static/logs/*; do find $i & ; done" didn't raise any errors after a minute or so, that seems like it would be pretty stressful | 22:27 |
| ianw | was all xvdg | 22:28 |
| *** chlong has quit IRC | 22:30 | |
| jhesketh | jeblair: if all the volumes are there etc, and with ianw's checking, I'd assume fsck is okay, but I'm not 100% sure | 22:31 |
| jeblair | jhesketh, ianw: it looks like docs-draft and tarballs are the only filesystems that went read-only | 22:31 |
| jeblair | shall i unmount them, fsck, and remound? | 22:32 |
| ianw | jeblair: that might make sense, the errors were on xvdg? say for a random sector | 22:32 |
| ianw | 201396888 * 512 / (4096 * 1024) | 22:32 |
| ianw | = 24584 | 22:32 |
| ianw | Physical extent 0 to 51199: | 22:32 |
| ianw | Logical volume/dev/main/tarballs | 22:32 |
| ianw | Logical extents0 to 51199 | 22:32 |
| ianw | which puts it in tarballs | 22:33 |
| ianw | anyway, yeah, i say try the fsck | 22:33 |
| jeblair | ++ | 22:33 |
| jeblair | i'll start with docs-draft | 22:33 |
| *** matrohon has quit IRC | 22:33 | |
| *** gildub has joined #openstack-infra | 22:33 | |
| jeblair | amrith: i left some comments on your changes | 22:34 |
| *** Julien-zte has joined #openstack-infra | 22:34 | |
| amrith | jeblair, thx | 22:34 |
| amrith | let me look | 22:34 |
| amrith | jeblair, with that section in place, the ci job was failing; I had to remove it to make it pass. | 22:35 |
| amrith | hence the doc update as well. | 22:35 |
| amrith | so maybe both are wrong? | 22:35 |
| *** panda is now known as panda|zZ | 22:35 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Create experimental kolla-publish-image-jobs https://review.openstack.org/401462 | 22:36 |
| pabelanger | inc0: ^ is the experimental job | 22:36 |
| inc0 | many thanks! | 22:36 |
| pabelanger | inc0: I've set it up to only use the build target | 22:36 |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Use xenial as default node for gate-tempest-dsvm-neutron-src-neutron-lib https://review.openstack.org/397401 | 22:37 |
| pabelanger | which, I think it correct | 22:37 |
| jeblair | amrith: huh. i think that check is wrong. we should consult with AJaeger on this. | 22:37 |
| jeblair | http://logs.openstack.org/89/398689/5/check/gate-project-config-layout/2ee33d7/console.html#_2016-11-23_20_15_26_819965 <-- that one | 22:37 |
| *** wolverineav has joined #openstack-infra | 22:38 | |
| amrith | jeblair, please see http://logs.openstack.org/89/398689/5/check/gate-project-config-layout/2ee33d7/ | 22:38 |
| amrith | ah, yes ... that one | 22:38 |
| amrith | I was going based on that failure. | 22:38 |
| *** e0ne has quit IRC | 22:39 | |
| amrith | so, would you please also put a -0.5 on https://review.openstack.org/401459. I think half of that change is still valid :) | 22:39 |
| jeblair | amrith: it may have been written with adding a project in mind, but without consideration of retiring one (which happens much less often) | 22:39 |
| inc0 | pabelanger, if I read it correctly, all build jobs will publish images to tarballs? | 22:39 |
| amrith | thanks jeblair will ping AJaeger about it tomorrow; he doesn't get thanksgiving (nor do i) | 22:40 |
| *** nherciu has quit IRC | 22:40 | |
| *** cody-somerville has quit IRC | 22:40 | |
| jeblair | amrith: k. i'll write my comments in the change too for better context | 22:40 |
| amrith | ok, thanks! | 22:41 |
| ianw | pabelanger: i know you've spent a bit of time with tracing subprocesses for ansible, wouldn't mind if you took a look at https://review.openstack.org/#/c/401048/ | 22:42 |
| ianw | i realised though, we're still building centos6 images. which means it has to be python2.6 compatible :/ | 22:42 |
| *** wolverineav has quit IRC | 22:42 | |
| fungi | inc0: not really around, sorry. entertaining family and cooking a lot | 22:42 |
| ianw | but, we also build xenial images that only have python3 | 22:43 |
| inc0 | fungi, np, we moved on thanks to pabelanger :) | 22:43 |
| fungi | pabelanger: is awesome | 22:43 |
| inc0 | sorry to bother you before holiday | 22:43 |
| fungi | no apologies needed. i'm just sorry i can't be around more at the moment | 22:43 |
| inc0 | we all need some rnr I think | 22:44 |
| *** gildub_ has joined #openstack-infra | 22:44 | |
| inc0 | speaking of which, I think I'll start my holiday too;) talk to you all next week, and have a delicious turkey | 22:45 |
| jeblair | #status log This message is to inform you that your Cloud Block Storage device static.openstack.org/main05 has been returned to service. | 22:45 |
| openstackstatus | jeblair: finished logging | 22:45 |
| jeblair | http://paste.openstack.org/show/590266/ | 22:45 |
| *** smarcet has quit IRC | 22:46 | |
| *** isviridov_away has joined #openstack-infra | 22:46 | |
| jeblair | remounted docs-draft | 22:46 |
| jeblair | fsck tarballs now | 22:46 |
| *** vaidy has joined #openstack-infra | 22:46 | |
| pabelanger | inc0: yup, all build jobs. Is that right? | 22:46 |
| jeblair | http://paste.openstack.org/show/590267/ | 22:46 |
| jeblair | ianw, jhesketh: i think i can send status ok now. agree? | 22:47 |
| inc0 | pabelanger, yeah, just thinking how to find out which one has to build tarballs | 22:47 |
| *** gildub has quit IRC | 22:47 | |
| inc0 | but I guess I'll just do it everywhere for now | 22:47 |
| ianw | jeblair: yep. give thanks to the anonymous rackspace operator who had to get up, wander into the server room and pull a drive out :) | 22:47 |
| pabelanger | inc0: maybe we should start with one, to make sure others don't overwrite the contents of another job | 22:48 |
| inc0 | pabelanger, we can keep it like that | 22:48 |
| jeblair | ianw: we should send them pie | 22:49 |
| inc0 | we won't use it anyway before it works | 22:49 |
| inc0 | and when it works, we need it for all;) | 22:49 |
| pabelanger | ianw: looking | 22:49 |
| jeblair | #status ok The affected filesystems on the log server are repaired. Please leave 'recheck' comments on any changes which failed with POST_FAILURE. | 22:49 |
| openstackstatus | jeblair: sending ok | 22:49 |
| inc0 | anyway, talk to you guys later, happy thanksgiving | 22:50 |
| ianw | pabelanger: so centos-minimal failures to build seem to be that systemd isn't brought in any more. i think that something changed in dependencies that package-installs was bringing it in before, but not any more | 22:50 |
| *** wolverineav has joined #openstack-infra | 22:50 | |
| jhesketh | jeblair: yep, sounds good to me (sorry catching up) | 22:50 |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Switch gate-neutronclient-test-dsvm-functional-adv-svcs to xenial https://review.openstack.org/401472 | 22:51 |
| pabelanger | ianw: okay | 22:51 |
| *** inc0 has quit IRC | 22:51 | |
| ianw | of course, it took about a day of yak shaving to realise that :) | 22:51 |
| *** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/" | 22:52 | |
| -openstackstatus- NOTICE: The affected filesystems on the log server are repaired. Please leave 'recheck' comments on any changes which failed with POST_FAILURE. | 22:52 | |
| *** ad_rien_ has quit IRC | 22:53 | |
| pabelanger | ianw: did something in DIB change or a RPM dependency? If DIB, surprised testing didn't catch it | 22:54 |
| ianw | pabelanger: i think an rpm dependency, and i think that we don't really test dib-init-system. and since we don't boot the images we build ... | 22:55 |
| *** wolverineav has quit IRC | 22:55 | |
| ianw | at least in the "functional" tests | 22:55 |
| fungi | #status log Configuration management updates are temporarily disabled for openstackid.org in preparation for validating change 399253. | 22:55 |
| openstackstatus | jeblair: finished sending ok | 22:56 |
| openstackstatus | fungi: finished logging | 22:56 |
| openstackgerrit | Merged openstack-infra/puppet-nodepool: Conditionally set mysql max connections https://review.openstack.org/401417 | 22:56 |
| pabelanger | ianw: Ya, I'd like to maybe enhance https://review.openstack.org/#/c/399865/ for diskimage-builder jobs. Not only build and launch them with nodepool, but SSH into them and do some basic audit of the image | 22:56 |
| pabelanger | to help catch breakages before nodepool-builder does | 22:56 |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Switch gate-neutronclient-test-dsvm-functional-adv-svcs to xenial https://review.openstack.org/401472 | 22:57 |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Switch gate-neutronclient-test-dsvm-functional to xenial https://review.openstack.org/401473 | 22:57 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/devstack-gate: WIP Add setup_host playbook https://review.openstack.org/391519 | 22:57 |
| ianw | pabelanger: yeah, cool. we do run the nodepool dsvm for dib changes, but not building a centos image there. might be good to add that for variety | 22:57 |
| pabelanger | ianw: yes, agree. we need to do some rework to expand the images for DIB jobs, but not nodepool patches. Might hack on that next week | 22:58 |
| *** dimtruck is now known as zz_dimtruck | 22:59 | |
| *** xyang1 has quit IRC | 22:59 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: yum-minimal: add systemd to initial install https://review.openstack.org/401504 | 23:00 |
| *** wolverineav has joined #openstack-infra | 23:00 | |
| *** rbuzatu_ has quit IRC | 23:00 | |
| *** rbuzatu has joined #openstack-infra | 23:02 | |
| *** annegentle has joined #openstack-infra | 23:03 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Catch errors in DIB_INIT_SYSTEM export https://review.openstack.org/400542 | 23:04 |
| *** wolverineav has quit IRC | 23:05 | |
| *** rlandy is now known as rlandy|bbl | 23:06 | |
| openstackgerrit | Merged openstack-infra/system-config: Fix nodepool builder branch name https://review.openstack.org/401419 | 23:08 |
| *** wolverineav has joined #openstack-infra | 23:08 | |
| *** mtanino has quit IRC | 23:12 | |
| *** wolverineav has quit IRC | 23:12 | |
| *** wolverineav has joined #openstack-infra | 23:18 | |
| markmcclain | getting a strange error on a job POST_FAILURE | 23:19 |
| markmcclain | the log file of the job says the task was a success | 23:19 |
| *** sreeram_v has joined #openstack-infra | 23:19 | |
| markmcclain | where's the best place to start digging into this? | 23:20 |
| markmcclain | pabelanger: ^^^ | 23:21 |
| *** wolverineav has quit IRC | 23:22 | |
| markmcclain | ok... poking around the job's ansible logs looks like rsync is failing | 23:24 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/devstack-gate: Add gather_host_info playbook and role https://review.openstack.org/401454 | 23:24 |
| *** sreeram_v has quit IRC | 23:24 | |
| markmcclain | nevermind... didn't have previous status message in my scrollback | 23:24 |
| jeblair | markmcclain: if it happened between 20:40 ... yeah :) | 23:25 |
| *** wolverineav has joined #openstack-infra | 23:28 | |
| openstackgerrit | Ryan Beisner proposed openstack-infra/project-config: Update charm job definitions https://review.openstack.org/399299 | 23:29 |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Switch neutronclient functional jobs to xenial https://review.openstack.org/401791 | 23:29 |
| *** dkehn__ has quit IRC | 23:29 | |
| openstackgerrit | Ryan Beisner proposed openstack-infra/project-config: Update charm job definitions https://review.openstack.org/399299 | 23:30 |
| *** wolverineav has quit IRC | 23:32 | |
| fungi | markmcclain: there's also https://wiki.openstack.org/wiki/Infrastructure_Status where we archive them all (reverse chronological order, so most recent at the top) | 23:34 |
| markmcclain | fungi: learned something new... I'll bookmark it | 23:37 |
| *** sdake has quit IRC | 23:37 | |
| *** wolverineav has joined #openstack-infra | 23:38 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/devstack-gate: Add setup_host playbook https://review.openstack.org/391519 | 23:38 |
| *** Julien-zte has quit IRC | 23:38 | |
| *** gildub_ has quit IRC | 23:39 | |
| rcarrillocruz | jeblair: heya, following the d-g ansible refactor chat we had, does ^ look better now? | 23:39 |
| rcarrillocruz | mordred: ^ | 23:39 |
| rcarrillocruz | the idea is to pull out stuff from setup_host function into playbooks/roles, then invoke them from setup_host.yaml playbook | 23:40 |
| *** sdake has joined #openstack-infra | 23:40 | |
| *** sdake has quit IRC | 23:40 | |
| *** sdake has joined #openstack-infra | 23:40 | |
| *** mriedem has quit IRC | 23:41 | |
| *** wolverineav has quit IRC | 23:42 | |
| markmcclain | fungi, jeblair: so just got a POST_FAILURE (23:31:53) | 23:42 |
| *** dingyichen has joined #openstack-infra | 23:43 | |
| *** shashank_hegde has joined #openstack-infra | 23:45 | |
| *** tlian2 has joined #openstack-infra | 23:46 | |
| ianw | hmm, yeah, me too for a job | 23:47 |
| ianw | checking static again... | 23:47 |
| *** wolverineav has joined #openstack-infra | 23:48 | |
| *** tlian has quit IRC | 23:48 | |
| ianw | yeah, looks like same disk is unhappy again | 23:49 |
| *** jamesdenton has joined #openstack-infra | 23:50 | |
| ianw | yeah, and everything is r/o | 23:50 |
| ianw | #status alert Due to a problem with the cinder volume backing the log server, jobs are failing with POST_FAILURE. Please avoid issuing 'recheck' commands until the issue is resolved. | 23:50 |
| *** tlian2 has quit IRC | 23:51 | |
| ianw | ahh, i probably can't do that, not sure i ever added perms | 23:51 |
| *** jamesdenton has quit IRC | 23:52 | |
| *** wolverineav has quit IRC | 23:53 | |
| *** wolverineav has joined #openstack-infra | 23:53 | |
| openstackgerrit | Ryan Beisner proposed openstack-infra/project-config: Update charm job definitions https://review.openstack.org/399299 | 23:54 |
| *** wolverineav has quit IRC | 23:54 | |
| *** wolverineav has joined #openstack-infra | 23:55 | |
| *** wolverineav has quit IRC | 23:56 | |
| ianw | ok, at 9:47am there was another outage alert, and 10:40 AM AEDT a restore alert (i.e. 10 minutes ago) | 23:56 |
| *** newmember has quit IRC | 23:57 | |
| markmcclain | ianw: thanks for checking | 23:59 |
| *** tlian has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!