crinkle | the posix provider "prevents the use of globbing and shell built-ins" --> seems like it shouldn't work | 00:00 |
---|---|---|
crinkle | but it worked for me | 00:00 |
*** tbarron has joined #openstack-infra | 00:00 | |
*** tsekiyam_ has quit IRC | 00:00 | |
*** hichihara has joined #openstack-infra | 00:01 | |
*** jasondotstar has quit IRC | 00:01 | |
*** PpPyg has joined #openstack-infra | 00:01 | |
*** boris-42 has joined #openstack-infra | 00:02 | |
*** camunoz_mtg has quit IRC | 00:02 | |
*** sergmelikyan has quit IRC | 00:02 | |
*** shashank_hegde has quit IRC | 00:04 | |
*** sergmelikyan has joined #openstack-infra | 00:04 | |
*** pcrews has quit IRC | 00:05 | |
*** madhu_ak has quit IRC | 00:05 | |
*** jamesmcarthur has quit IRC | 00:06 | |
*** DericHorn-HP_ has joined #openstack-infra | 00:07 | |
*** nimrodsun_ has joined #openstack-infra | 00:07 | |
*** yfried_ has joined #openstack-infra | 00:07 | |
*** yfried_ is now known as yfried|afk | 00:07 | |
*** dimtruck is now known as zz_dimtruck | 00:08 | |
*** pahuang__ has quit IRC | 00:09 | |
*** mwagner_lap has joined #openstack-infra | 00:09 | |
*** achanda has quit IRC | 00:16 | |
*** jasondotstar has joined #openstack-infra | 00:16 | |
*** jasondotstar has quit IRC | 00:16 | |
*** davideagnello has quit IRC | 00:16 | |
*** jasondotstar has joined #openstack-infra | 00:16 | |
*** DericHorn-HP has quit IRC | 00:17 | |
*** ildikov has quit IRC | 00:17 | |
*** yfried has quit IRC | 00:17 | |
*** linkedinyou has quit IRC | 00:17 | |
*** nimrodsun has quit IRC | 00:17 | |
*** DericHorn-HP_ is now known as DericHorn-HP | 00:17 | |
akraminsky_ | afazekas: im having some issues with sshing into my subnodes using their private IP's. Can you explain why you use the private ip's instead of public in devstack-gate-vm.sh line 499? :) | 00:18 |
*** nimrodsun_ is now known as nimrodsun | 00:18 | |
*** shardy_ has joined #openstack-infra | 00:18 | |
akraminsky_ | afazekas: https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate.sh | 00:18 |
*** shardy has quit IRC | 00:19 | |
*** shardy_ has quit IRC | 00:23 | |
*** shardy has joined #openstack-infra | 00:23 | |
*** ashleighfarnham has quit IRC | 00:24 | |
*** ildikov has joined #openstack-infra | 00:25 | |
*** pahuang has joined #openstack-infra | 00:26 | |
*** tbarron has quit IRC | 00:27 | |
*** esmute_ has quit IRC | 00:29 | |
*** esmute__ has joined #openstack-infra | 00:29 | |
*** tbarron has joined #openstack-infra | 00:30 | |
*** patrickeast has quit IRC | 00:30 | |
*** geoffarnold has quit IRC | 00:32 | |
*** tbarron has quit IRC | 00:33 | |
*** sdake has quit IRC | 00:34 | |
*** pvaneck has quit IRC | 00:36 | |
*** sergmelikyan has quit IRC | 00:37 | |
*** sdake has joined #openstack-infra | 00:37 | |
*** Swami has quit IRC | 00:39 | |
*** tbarron has joined #openstack-infra | 00:40 | |
*** cody-somerville has quit IRC | 00:42 | |
*** salv-orlando has quit IRC | 00:43 | |
*** shashank_hegde has joined #openstack-infra | 00:43 | |
*** esmute__ has quit IRC | 00:45 | |
*** sergmelikyan has joined #openstack-infra | 00:46 | |
*** rlandy has quit IRC | 00:46 | |
*** rlandy has joined #openstack-infra | 00:47 | |
*** sergmelikyan has quit IRC | 00:47 | |
*** DericHorn-HP has quit IRC | 00:54 | |
*** sergmelikyan has joined #openstack-infra | 00:55 | |
*** jasondotstar has quit IRC | 00:56 | |
*** sergmelikyan has quit IRC | 00:56 | |
openstackgerrit | liusheng proposed openstack/requirements: Bump gabbit to version 1.1.0 https://review.openstack.org/201953 | 00:58 |
openstackgerrit | liusheng proposed openstack/requirements: Bump gabbi to version 1.1.0 https://review.openstack.org/201953 | 00:59 |
*** ZZelle_ has quit IRC | 00:59 | |
*** serg_melikyan has joined #openstack-infra | 01:00 | |
*** erlon has quit IRC | 01:00 | |
*** shashank_hegde has quit IRC | 01:05 | |
*** kutija has quit IRC | 01:07 | |
*** ddieterl_ has joined #openstack-infra | 01:08 | |
*** ddieterly has quit IRC | 01:08 | |
*** amitgandhinz has joined #openstack-infra | 01:09 | |
*** Krinkle has joined #openstack-infra | 01:11 | |
*** shashank_hegde has joined #openstack-infra | 01:11 | |
*** shashank_hegde has quit IRC | 01:11 | |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-cgit: Add Gemfile and puppet 4 checks https://review.openstack.org/203870 | 01:12 |
*** esmute_ has joined #openstack-infra | 01:12 | |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-diskimage_builder: Add Gemfile and puppet 4 checks https://review.openstack.org/202661 | 01:12 |
openstackgerrit | takehirokaneko proposed openstack-infra/project-config: Add coverage test to check pipeline of stackforge/rack https://review.openstack.org/203976 | 01:13 |
*** DericHorn-HP has joined #openstack-infra | 01:13 | |
*** amickus has quit IRC | 01:14 | |
*** serg_melikyan has quit IRC | 01:15 | |
*** puranamr has joined #openstack-infra | 01:16 | |
*** camunoz has joined #openstack-infra | 01:17 | |
*** sergmelikyan has joined #openstack-infra | 01:18 | |
openstackgerrit | IWAMOTO Toshihiro proposed openstack/requirements: Add ryu https://review.openstack.org/154354 | 01:20 |
*** sergmelikyan has quit IRC | 01:23 | |
*** esmute_ has quit IRC | 01:24 | |
*** alexsyip has quit IRC | 01:26 | |
*** asselin_ has joined #openstack-infra | 01:27 | |
*** bdemers has joined #openstack-infra | 01:33 | |
*** esmute_ has joined #openstack-infra | 01:33 | |
*** shashank_hegde has joined #openstack-infra | 01:35 | |
*** rlandy has quit IRC | 01:37 | |
*** thorst has joined #openstack-infra | 01:38 | |
*** flepied has joined #openstack-infra | 01:38 | |
*** rlandy has joined #openstack-infra | 01:38 | |
*** changbl has joined #openstack-infra | 01:43 | |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-diskimage_builder: Add Gemfile and puppet 4 checks https://review.openstack.org/202661 | 01:43 |
*** salv-orlando has joined #openstack-infra | 01:43 | |
*** pcrews has joined #openstack-infra | 01:44 | |
*** sergmelikyan has joined #openstack-infra | 01:44 | |
*** armax has quit IRC | 01:44 | |
*** yamamoto has quit IRC | 01:45 | |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-cgit: Add Gemfile and puppet 4 checks https://review.openstack.org/203870 | 01:45 |
*** jamesmcarthur has joined #openstack-infra | 01:46 | |
*** salv-orlando has quit IRC | 01:46 | |
*** amitgandhinz has quit IRC | 01:46 | |
*** sergmelikyan has quit IRC | 01:46 | |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-drupal: Add Gemfile and puppet 4 checks https://review.openstack.org/203881 | 01:46 |
*** pal has joined #openstack-infra | 01:47 | |
*** puranamr has quit IRC | 01:47 | |
StevenK | pleia2: If you're still around, https://review.openstack.org/203927 needs a review, pretty please. | 01:47 |
StevenK | nibalizer, crinkle: Did either of you poke at the manifest log file I uploaded, by the way? | 01:48 |
nibalizer | StevenK: ah no, relink? | 01:48 |
StevenK | nibalizer: http://wedontsleep.org/~steven/manifest.log.gz | 01:49 |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-elastic_recheck: Add Gemfile and puppet 4 checks https://review.openstack.org/203884 | 01:50 |
*** bdemers has quit IRC | 01:50 | |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-elasticsearch: Add Gemfile and puppet 4 checks https://review.openstack.org/203886 | 01:50 |
nibalizer | StevenK: what puppet version? | 01:50 |
StevenK | nibalizer: Whatever the latest from Trusty is | 01:51 |
StevenK | I think | 01:51 |
StevenK | Let me boot the VM | 01:51 |
nibalizer | ok | 01:51 |
*** DericHorn-HP has quit IRC | 01:52 | |
openstackgerrit | Timo Tijhof proposed openstack-infra/zuul: Cloner: Implement cache-no-hardlinks argument https://review.openstack.org/117626 | 01:54 |
*** bradjones has quit IRC | 01:54 | |
*** sergmelikyan has joined #openstack-infra | 01:54 | |
*** bdemers has joined #openstack-infra | 01:55 | |
*** sergmelikyan has quit IRC | 01:55 | |
*** sergmelikyan has joined #openstack-infra | 01:56 | |
*** bradjones has joined #openstack-infra | 01:56 | |
*** bradjones has quit IRC | 01:56 | |
*** bradjones has joined #openstack-infra | 01:56 | |
*** sergmelikyan has quit IRC | 01:57 | |
StevenK | nibalizer: Sorry about that. It's 3.8.1 | 01:59 |
nibalizer | no worries | 02:00 |
nibalizer | just as long as its not 4 lol | 02:00 |
*** DericHorn-HP has joined #openstack-infra | 02:00 | |
nibalizer | oh you know what that looks like to me | 02:01 |
nibalizer | is a weird puppet way of saying the superior directory doesn't exist | 02:01 |
nibalizer | does /opt/wildfly/standalone/deployments/ exist? | 02:01 |
StevenK | drwxr-xr-x 2 wildfly wildfly 4.0K Jul 21 04:16 deployments | 02:01 |
StevenK | Yup | 02:01 |
nibalizer | hrm | 02:01 |
*** sergmelikyan has joined #openstack-infra | 02:02 | |
*** julim has joined #openstack-infra | 02:03 | |
* nibalizer strokes nonexistent goatee | 02:03 | |
*** rlandy has quit IRC | 02:04 | |
*** annegent_ has quit IRC | 02:04 | |
crinkle | I don't understand why it's referring to a .lock file, is this a different patchset than master? | 02:04 |
nibalizer | yes this question ^ | 02:05 |
nibalizer | it looks like it should be looking for zanata-war-3.6.2.war | 02:05 |
*** annegentle has joined #openstack-infra | 02:06 | |
crinkle | I thought it was looking for ROOT.war http://git.openstack.org/cgit/openstack-infra/puppet-zanata/tree/manifests/init.pp#n85 | 02:06 |
StevenK | That's what it's trying to create | 02:07 |
StevenK | -rw-r--r-- 1 wildfly wildfly 97M Apr 2 07:04 /home/wildfly/zanata-war-3.6.2.war | 02:07 |
StevenK | crinkle: And no | 02:07 |
*** sergmelikyan has quit IRC | 02:07 | |
StevenK | root@zanata-server:/etc/puppet/modules/zanata# git log --pretty=format:%h -n 1 | 02:08 |
StevenK | 52d2ebc | 02:08 |
*** esmute_ has quit IRC | 02:08 | |
*** yamamoto has joined #openstack-infra | 02:09 | |
nibalizer | StevenK: okay the beers are calling to me | 02:09 |
*** annegentle has quit IRC | 02:10 | |
nibalizer | sorry I didn't remember to debug this before my day ended | 02:10 |
nibalizer | fungi: pleia2 yall osconpartying? ping me if you want to meetup | 02:10 |
crinkle | StevenK: and adding file:// to the value in the source parameter fixes it? | 02:11 |
StevenK | crinkle: That makes it work, yes | 02:12 |
crinkle | that's really weird | 02:12 |
nibalizer | i just want to say one last time that this behaviour is pretty o.O | 02:12 |
nibalizer | like wat | 02:12 |
nibalizer | maybe we should just open a jira ticket for puppet like 'wat' | 02:12 |
*** esmute_ has joined #openstack-infra | 02:13 | |
*** DericHorn-HP has quit IRC | 02:14 | |
*** puranamr has joined #openstack-infra | 02:18 | |
*** puranamr has quit IRC | 02:18 | |
*** ivar-laz_ has quit IRC | 02:18 | |
*** ivar-lazzaro has joined #openstack-infra | 02:19 | |
*** thorst has quit IRC | 02:20 | |
*** flepied has quit IRC | 02:24 | |
*** linkedinyou has joined #openstack-infra | 02:25 | |
*** dims_ has joined #openstack-infra | 02:25 | |
*** koolhead17 has joined #openstack-infra | 02:30 | |
*** unicell has quit IRC | 02:31 | |
*** timrc_ has joined #openstack-infra | 02:32 | |
*** pcrews has quit IRC | 02:33 | |
*** achanda has joined #openstack-infra | 02:33 | |
*** bharath has joined #openstack-infra | 02:35 | |
*** pahuang has quit IRC | 02:35 | |
*** rossella_s has quit IRC | 02:37 | |
*** rossella_s has joined #openstack-infra | 02:37 | |
*** glauco has joined #openstack-infra | 02:39 | |
*** bharath has quit IRC | 02:39 | |
*** gyee has quit IRC | 02:39 | |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-accessbot: Remove puppet-lint-numericvariable gem https://review.openstack.org/204343 | 02:42 |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-ansible: Remove puppet-lint-numericvariable gem https://review.openstack.org/204344 | 02:42 |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-apparmor: Remove puppet-lint-numericvariable gem https://review.openstack.org/204345 | 02:42 |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-apps_site: Remove puppet-lint-numericvariable gem https://review.openstack.org/204346 | 02:42 |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-askbot: Remove puppet-lint-numericvariable gem https://review.openstack.org/204347 | 02:42 |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-asterisk: Remove puppet-lint-numericvariable gem https://review.openstack.org/204348 | 02:42 |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-bandersnatch: Remove puppet-lint-numericvariable gem https://review.openstack.org/204349 | 02:42 |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-bugdaystats: Remove puppet-lint-numericvariable gem https://review.openstack.org/204350 | 02:42 |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-bup: Remove puppet-lint-numericvariable gem https://review.openstack.org/204351 | 02:42 |
*** pal has quit IRC | 02:43 | |
*** pcrews has joined #openstack-infra | 02:45 | |
*** pahuang has joined #openstack-infra | 02:47 | |
*** mkj_wk_ is now known as mkj_wk | 02:53 | |
*** julim has quit IRC | 02:53 | |
openstackgerrit | Timothy R. Chavez proposed openstack-infra/puppet-cgit: Add better defaults for clone-prefix https://review.openstack.org/204245 | 02:54 |
*** yamahata has quit IRC | 02:54 | |
*** unicell has joined #openstack-infra | 02:54 | |
*** Krinkle is now known as Krinkle_ | 02:55 | |
*** amitgandhinz has joined #openstack-infra | 02:57 | |
*** DericHorn-HP has joined #openstack-infra | 02:57 | |
*** DericHorn-HP has quit IRC | 02:58 | |
*** unicell1 has joined #openstack-infra | 02:58 | |
*** sabeen has joined #openstack-infra | 02:59 | |
*** unicell has quit IRC | 02:59 | |
*** amitgandhinz has quit IRC | 03:01 | |
*** auxian has joined #openstack-infra | 03:01 | |
*** DericHorn-HP has joined #openstack-infra | 03:02 | |
*** jerryz has quit IRC | 03:07 | |
*** dingyichen has joined #openstack-infra | 03:07 | |
openstackgerrit | Vadivel Poonathan proposed openstack-infra/project-config: Add ALE Omniswitch Networking project to Openstack https://review.openstack.org/203477 | 03:08 |
*** linkedinyou|2 has joined #openstack-infra | 03:09 | |
*** kiran-r has joined #openstack-infra | 03:10 | |
*** linkedinyou|2 has quit IRC | 03:11 | |
*** linkedinyou has quit IRC | 03:12 | |
*** heyongli has quit IRC | 03:12 | |
*** DericHorn-HP has quit IRC | 03:13 | |
*** bdemers has quit IRC | 03:14 | |
*** heyongli has joined #openstack-infra | 03:14 | |
*** bdemers has joined #openstack-infra | 03:14 | |
*** ivar-laz_ has joined #openstack-infra | 03:15 | |
*** dims_ has quit IRC | 03:15 | |
*** bdemers has quit IRC | 03:18 | |
*** ivar-lazzaro has quit IRC | 03:18 | |
*** bdemers has joined #openstack-infra | 03:21 | |
*** koolhead17 has quit IRC | 03:21 | |
*** salv-orlando has joined #openstack-infra | 03:22 | |
*** tlian has quit IRC | 03:24 | |
*** bdemers has quit IRC | 03:26 | |
*** bdemers has joined #openstack-infra | 03:27 | |
*** salv-orlando has quit IRC | 03:27 | |
*** kiran-r has quit IRC | 03:28 | |
*** pal has joined #openstack-infra | 03:29 | |
*** linkedinyou has joined #openstack-infra | 03:31 | |
*** glauco has quit IRC | 03:31 | |
*** thorst has joined #openstack-infra | 03:31 | |
*** crc32 has joined #openstack-infra | 03:32 | |
*** pahuang has quit IRC | 03:33 | |
*** bhunter71 has quit IRC | 03:33 | |
*** heyongli has quit IRC | 03:35 | |
*** sherif has joined #openstack-infra | 03:35 | |
*** DericHorn-HP has joined #openstack-infra | 03:36 | |
*** salv-orlando has joined #openstack-infra | 03:37 | |
*** glauco_ has joined #openstack-infra | 03:37 | |
*** salv-orlando has quit IRC | 03:37 | |
*** heyongli has joined #openstack-infra | 03:40 | |
*** bdemers has quit IRC | 03:41 | |
*** bdemers has joined #openstack-infra | 03:41 | |
openstackgerrit | litong01 proposed openstack-infra/project-config: Add new monitor repository for OpenStack monitoring service https://review.openstack.org/203259 | 03:43 |
*** jamesmcarthur has quit IRC | 03:45 | |
*** thorst has quit IRC | 03:45 | |
*** gus_ is now known as gus | 03:45 | |
*** glauco_ has quit IRC | 03:47 | |
*** dims_ has joined #openstack-infra | 03:49 | |
*** pahuang has joined #openstack-infra | 03:50 | |
*** achanda has quit IRC | 03:50 | |
*** bdemers has quit IRC | 03:55 | |
*** dims_ has quit IRC | 03:56 | |
*** bdemers has joined #openstack-infra | 03:56 | |
*** gildub has quit IRC | 03:57 | |
*** coolsvap|away is now known as coolsvap | 04:01 | |
*** rfolco has quit IRC | 04:03 | |
*** jamesmcarthur has joined #openstack-infra | 04:07 | |
*** sherif has quit IRC | 04:13 | |
*** Kennan2 has joined #openstack-infra | 04:14 | |
*** Kennan has quit IRC | 04:15 | |
openstackgerrit | Merged openstack-infra/gearman-plugin: Fix deadlock from a WORK_FAIL event https://review.openstack.org/192429 | 04:16 |
*** bdemers has quit IRC | 04:17 | |
*** bdemers has joined #openstack-infra | 04:18 | |
*** gildub has joined #openstack-infra | 04:19 | |
*** linkedinyou has quit IRC | 04:22 | |
openstackgerrit | Merged openstack/requirements: Add keystoneauth to projects.txt https://review.openstack.org/203256 | 04:22 |
*** VivCheri has joined #openstack-infra | 04:22 | |
*** VivCheri has quit IRC | 04:24 | |
*** camunoz is now known as camunoz_mtg | 04:24 | |
lifeless | mordred: file a bug on pbr, I'll look at it and see if I can eithe rdo it trivially, or mentor someone on it | 04:24 |
*** puranamr has joined #openstack-infra | 04:24 | |
openstackgerrit | Merged openstack/requirements: Bumping Cinder client to 1.3.1 https://review.openstack.org/199348 | 04:26 |
openstackgerrit | Merged openstack-infra/project-config: Remove gate-tempest-dsvm-docker from nova's check queue https://review.openstack.org/204264 | 04:27 |
*** tlbr_ has quit IRC | 04:31 | |
*** tlbr has joined #openstack-infra | 04:32 | |
*** glauco_ has joined #openstack-infra | 04:32 | |
*** boris-42 has quit IRC | 04:32 | |
*** bdemers has quit IRC | 04:34 | |
*** bdemers has joined #openstack-infra | 04:35 | |
*** DericHorn-HP has quit IRC | 04:35 | |
*** glauco_ has quit IRC | 04:36 | |
*** puranamr has quit IRC | 04:37 | |
openstackgerrit | Merged openstack/requirements: mock 1.2 has the 1.1.4 regression fixed and is 2.6 compatible https://review.openstack.org/203231 | 04:37 |
*** achanda has joined #openstack-infra | 04:37 | |
*** puranamr has joined #openstack-infra | 04:37 | |
*** bdemers has quit IRC | 04:42 | |
*** bdemers has joined #openstack-infra | 04:42 | |
openstackgerrit | Merged openstack/diskimage-builder: debian: install DHCP client and ifconfig packages https://review.openstack.org/200030 | 04:45 |
*** harlowja_at_home has joined #openstack-infra | 04:47 | |
openstackgerrit | Merged openstack/requirements: Bump lower/upper constraint of futurist to newly released 0.1.2 https://review.openstack.org/201709 | 04:47 |
*** larainema has quit IRC | 04:48 | |
*** larainema has joined #openstack-infra | 04:48 | |
*** tsg has quit IRC | 04:52 | |
openstackgerrit | Merged openstack/requirements: Adding angular-gettext to global requirements https://review.openstack.org/199349 | 04:53 |
openstackgerrit | OpenStack Proposal Bot proposed openstack-dev/hacking: Updated from global requirements https://review.openstack.org/186829 | 04:54 |
openstackgerrit | OpenStack Proposal Bot proposed openstack-dev/pbr: Updated from global requirements https://review.openstack.org/204386 | 04:54 |
openstackgerrit | OpenStack Proposal Bot proposed openstack/diskimage-builder: Updated from global requirements https://review.openstack.org/201852 | 04:54 |
*** bdemers has quit IRC | 04:55 | |
*** bdemers has joined #openstack-infra | 04:55 | |
*** maishsk has quit IRC | 04:57 | |
*** dalgaaf has quit IRC | 04:58 | |
*** jamesmcarthur has quit IRC | 05:03 | |
*** kutija has joined #openstack-infra | 05:03 | |
*** mrmartin has joined #openstack-infra | 05:05 | |
*** shashank_hegde has quit IRC | 05:06 | |
fungi | nibalizer: oh, sorry, clarkb drug us out looking for a brewery which wasn't overrun by a tool concert | 05:07 |
fungi | we skipped from ex novo over to lompoc, then called it an early evening | 05:08 |
nibalizer | hah. sounds like clarkb | 05:08 |
fungi | well, he has babies now | 05:08 |
nibalizer | ill be conferencing properly tomorrow so we will likely see each lther | 05:08 |
fungi | those apparently are lots of work | 05:08 |
fungi | yep, i'll be there all day. see you there! | 05:08 |
openstackgerrit | Merged openstack-infra/project-config: Switch default pattern in create-zanata-xml.py https://review.openstack.org/203927 | 05:10 |
*** puranamr has quit IRC | 05:10 | |
*** esmute_ has quit IRC | 05:13 | |
*** esmute_ has joined #openstack-infra | 05:16 | |
crinkle | StevenK: I think nibalizer was right about the parent directory not being created | 05:16 |
crinkle | StevenK: it gets created by an exec that untars a tarball in the wildfly module https://github.com/biemond/biemond-wildfly/blob/v0.2.4/manifests/install.pp#L16-L23 | 05:16 |
crinkle | StevenK: adding a require relationship on that resource fixed it for me http://paste.openstack.org/show/398518/ | 05:17 |
*** deepakcs has joined #openstack-infra | 05:17 | |
openstackgerrit | David Cheperdak proposed openstack-infra/project-config: Add Stackforge project Fusion. https://review.openstack.org/204434 | 05:19 |
*** ivar-laz_ has quit IRC | 05:21 | |
*** yamamoto has quit IRC | 05:21 | |
*** ivar-lazzaro has joined #openstack-infra | 05:21 | |
*** yamamoto has joined #openstack-infra | 05:23 | |
clarkb | Tool and oscon at tge same time whats going on | 05:24 |
fungi | so much crazy | 05:25 |
*** ivar-lazzaro has quit IRC | 05:26 | |
*** glauco has joined #openstack-infra | 05:26 | |
*** harlowja_at_home has quit IRC | 05:26 | |
afazekas | akraminsky_: The nodes expected to be able to see each other on the private net with less trouble. The private_ip is not necessarily directly reachable for external user. | 05:30 |
*** davidjc has joined #openstack-infra | 05:30 | |
*** abregman has joined #openstack-infra | 05:30 | |
StevenK | crinkle: Ah! An ordering problem, neat | 05:31 |
StevenK | crinkle: Thanks for digging into it :-) | 05:31 |
*** glauco has quit IRC | 05:31 | |
crinkle | StevenK: no problem :) | 05:31 |
StevenK | Right, and that's why file:// works -- it isn't that it works per-se, but it's a second run, and the directory exists. | 05:32 |
StevenK | So blah | 05:32 |
crinkle | right | 05:32 |
*** salv-orlando has joined #openstack-infra | 05:33 | |
StevenK | crinkle: Hmm, that's another place that the wildfly version gets hardcoded. | 05:35 |
*** salv-orlando has quit IRC | 05:38 | |
crinkle | StevenK: hmm i think you could use $zanata_file in place of the file name in that exec title | 05:38 |
StevenK | crinkle: That's the zanata WAR, not the wildfly tarball | 05:39 |
StevenK | But I may just copy that pattern | 05:39 |
crinkle | oh right | 05:39 |
crinkle | er no, i think it is the tarball in that exec | 05:40 |
*** mrmartin has quit IRC | 05:40 | |
StevenK | Yup, which comes from zanata_wildfly_install_url | 05:41 |
crinkle | oh I was looking at the wrong variable | 05:41 |
crinkle | you're right | 05:42 |
*** shashank_hegde has joined #openstack-infra | 05:43 | |
StevenK | crinkle: I have a diff locally, I'm just spawning an unsullied VM so we can see that the module behaves correctly for first ever run | 05:43 |
*** davidjc has quit IRC | 05:45 | |
*** heyongli has quit IRC | 05:49 | |
*** heyongli has joined #openstack-infra | 05:50 | |
*** afazekas has quit IRC | 05:55 | |
*** BharatK has joined #openstack-infra | 05:56 | |
*** _nadya_ has joined #openstack-infra | 05:56 | |
*** yfried|afk has quit IRC | 06:05 | |
*** dchen has joined #openstack-infra | 06:07 | |
*** amotoki has joined #openstack-infra | 06:09 | |
*** dingyichen has quit IRC | 06:09 | |
*** timrc_ has quit IRC | 06:10 | |
*** puranamr has joined #openstack-infra | 06:11 | |
*** camunoz_mtg has quit IRC | 06:11 | |
*** jtomasek has joined #openstack-infra | 06:12 | |
*** pahuang has quit IRC | 06:12 | |
*** andrey-mp has joined #openstack-infra | 06:13 | |
*** maishsk has joined #openstack-infra | 06:13 | |
*** puranamr has quit IRC | 06:15 | |
*** dims_ has joined #openstack-infra | 06:17 | |
StevenK | 2015-07-22 06:17:01 +0000 Puppet (err): Could not find dependency Exec[tar ${wildfly_file} in /var/tmp] for File. Blah. | 06:18 |
*** AJaeger has joined #openstack-infra | 06:18 | |
*** ParsectiX has joined #openstack-infra | 06:19 | |
AJaeger | any root still (already) around? jhesketh, fungi, clarkb, jeblair, pleia2, SergeyLukjanov: Looks like zuul is not removing jobs from the queue at all. Jobs are finished in the check queue but stay there according to status.openstack.org/zuul | 06:20 |
AJaeger | (same with gate queue) | 06:21 |
*** dims_ has quit IRC | 06:22 | |
*** pahuang has joined #openstack-infra | 06:23 | |
*** heyongli has quit IRC | 06:24 | |
*** dingyichen has joined #openstack-infra | 06:24 | |
*** camunoz_mtg has joined #openstack-infra | 06:24 | |
*** scheuran has joined #openstack-infra | 06:24 | |
*** dtantsur|afk is now known as dtantsur | 06:25 | |
*** kiran-r has joined #openstack-infra | 06:26 | |
*** dingyichen has quit IRC | 06:27 | |
*** jyuso1 has quit IRC | 06:29 | |
*** jyuso has joined #openstack-infra | 06:29 | |
*** boris-42 has joined #openstack-infra | 06:29 | |
*** caphrim007 has quit IRC | 06:30 | |
*** sdake has quit IRC | 06:31 | |
*** shardy_ has joined #openstack-infra | 06:31 | |
*** shardy has quit IRC | 06:32 | |
*** koolhead17 has joined #openstack-infra | 06:32 | |
openstackgerrit | Marc Koderer proposed openstack-infra/project-config: Add experimental job to test Manila Tempest plugin https://review.openstack.org/204454 | 06:34 |
*** heyongli has joined #openstack-infra | 06:34 | |
*** kushal has joined #openstack-infra | 06:35 | |
*** shardy_ has quit IRC | 06:36 | |
*** tangchen_ has joined #openstack-infra | 06:36 | |
tangchen_ | Hi, all. :) | 06:37 |
tangchen_ | anyone here ? | 06:37 |
*** shardy has joined #openstack-infra | 06:37 | |
*** rossella_s has quit IRC | 06:37 | |
tangchen_ | I'd like to ask something about 3rd party CI. | 06:37 |
*** rossella_s has joined #openstack-infra | 06:38 | |
*** afazekas_ has joined #openstack-infra | 06:39 | |
jhesketh | AJaeger: hmm, I'll take a look | 06:39 |
AJaeger | jhesketh: interesting that the first job in the check queue (nova, 187857,14) does not expand at all | 06:41 |
AJaeger | jhesketh: thanks! | 06:41 |
AJaeger | tangchen_: seems nobody with that knowledge is awake | 06:41 |
*** mrunge has joined #openstack-infra | 06:42 | |
*** mfalatic has quit IRC | 06:42 | |
openstackgerrit | Vadivel Poonathan proposed openstack-infra/project-config: Add ALE Omniswitch Networking project to Openstack https://review.openstack.org/203477 | 06:44 |
*** vlaza has joined #openstack-infra | 06:45 | |
*** mattt has left #openstack-infra | 06:45 | |
jhesketh | tangchen_: I might be able to help later after I've looked at the gate | 06:46 |
*** dims_ has joined #openstack-infra | 06:47 | |
openstackgerrit | Steve Kowalik proposed openstack-infra/puppet-zanata: Add second requirement for the ROOT.war deployment https://review.openstack.org/201943 | 06:48 |
tangchen_ | jhesketh, AJaeger: Hi, Thanks for reply. | 06:49 |
jhesketh | zuul's internal database seems to have gotten into a bad state | 06:50 |
jhesketh | I'm not sure if there's a fix that isn't a zuul restart | 06:50 |
jhesketh | but I'd like to figure out how it got there first | 06:50 |
tangchen_ | I'm setting up my first CI, and when I send a patch to gerrit, it cannot be merged. | 06:50 |
jhesketh | lots of http://paste.openstack.org/show/398833/ in the logs | 06:50 |
tangchen_ | It reports: "Unable to find change queue for change <Change 0x7fa7ef8b6250 204446,1> in project openstack-dev/sandbox" | 06:51 |
*** heyongli has quit IRC | 06:51 | |
*** deepakcs has quit IRC | 06:51 | |
tangchen_ | Do you guys have any idea of this ? | 06:51 |
*** andrey-mp has quit IRC | 06:53 | |
*** dims_ has quit IRC | 06:53 | |
*** heyongli has joined #openstack-infra | 06:53 | |
*** k4n0 has joined #openstack-infra | 06:55 | |
*** jyuso has quit IRC | 06:56 | |
openstackgerrit | David Cheperdak proposed openstack-infra/project-config: Add Stackforge project Fusion. https://review.openstack.org/204434 | 06:56 |
*** r-mibu has joined #openstack-infra | 06:57 | |
*** coolsvap is now known as coolsvap|away | 07:01 | |
*** _nadya_ has quit IRC | 07:04 | |
AJaeger | jhesketh: I have no idea about Zuul logs, sorry. But this looks indeed like some data structure is broken. | 07:07 |
*** bharath_ has joined #openstack-infra | 07:07 | |
jhesketh | AJaeger: it appears to be due to a race condition in the zuul reload | 07:07 |
jhesketh | AJaeger: I'm just prepping for a restart and re-queue | 07:07 |
AJaeger | jhesketh: ok | 07:08 |
*** kushal has quit IRC | 07:08 | |
*** sergmelikyan has joined #openstack-infra | 07:09 | |
*** heyongli has quit IRC | 07:10 | |
*** dims_ has joined #openstack-infra | 07:12 | |
*** esmute_ has quit IRC | 07:12 | |
*** ifarkas has joined #openstack-infra | 07:12 | |
*** deepakcs has joined #openstack-infra | 07:13 | |
*** glauco_ has joined #openstack-infra | 07:14 | |
*** dims_ has quit IRC | 07:16 | |
*** heyongli has joined #openstack-infra | 07:17 | |
jhesketh | restarting zuul | 07:18 |
*** glauco_ has quit IRC | 07:19 | |
AJaeger | jhesketh: what about sending a notification via IRC? | 07:19 |
jhesketh | good idea | 07:21 |
*** jerryz has joined #openstack-infra | 07:22 | |
jhesketh | #status notice zuul is undergoing an emergency restart. Jobs will be re-queued but some events may be lost. | 07:22 |
openstackstatus | jhesketh: sending notice | 07:22 |
*** e0ne has joined #openstack-infra | 07:22 | |
*** ildikov has quit IRC | 07:22 | |
-openstackstatus- NOTICE: zuul is undergoing an emergency restart. Jobs will be re-queued but some events may be lost. | 07:23 | |
*** cody-somerville has joined #openstack-infra | 07:23 | |
*** kushal has joined #openstack-infra | 07:24 | |
openstackstatus | jhesketh: finished sending notice | 07:24 |
*** salv-orlando has joined #openstack-infra | 07:26 | |
*** sshnaidm has joined #openstack-infra | 07:26 | |
AJaeger | thanks, jhesketh! Hope that everything is fine. | 07:26 |
AJaeger | Right now I wonder why stackforge/fuel-agent is on the top of the check queue - with only a noop change. | 07:27 |
jhesketh | I think the mergers are busy | 07:27 |
AJaeger | Let's hope ;) | 07:27 |
jhesketh | heh, yeah | 07:27 |
*** camunoz_mtg is now known as camunoz_gone | 07:28 | |
*** maishsk has quit IRC | 07:29 | |
*** achanda has quit IRC | 07:30 | |
*** fhubik has joined #openstack-infra | 07:31 | |
openstackgerrit | liusheng proposed openstack/requirements: Bump gabbi to version 1.1.0 https://review.openstack.org/201953 | 07:31 |
*** pal has quit IRC | 07:33 | |
*** heyongli has quit IRC | 07:33 | |
*** odyssey4me has quit IRC | 07:34 | |
*** odyssey4me has joined #openstack-infra | 07:37 | |
*** heyongli has joined #openstack-infra | 07:38 | |
*** koolhead17 has quit IRC | 07:38 | |
*** achanda has joined #openstack-infra | 07:39 | |
*** dizquierdo has joined #openstack-infra | 07:39 | |
*** jlanoux has joined #openstack-infra | 07:41 | |
openstackgerrit | Angus Lees proposed openstack-infra/project-config: Add new oslo.privsep project https://review.openstack.org/204477 | 07:43 |
*** dims_ has joined #openstack-infra | 07:44 | |
*** zeih has joined #openstack-infra | 07:44 | |
*** afazekas_ has quit IRC | 07:45 | |
AJaeger | jhesketh: doesn't look like any job is moving forward ;( - or am I to impatient? | 07:47 |
jhesketh | AJaeger: no | 07:48 |
jhesketh | clients are failing to reconnect to the gearman service | 07:48 |
jhesketh | eg mergers and jenkins | 07:48 |
*** dguerri` is now known as dguerri | 07:48 | |
jhesketh | I can telnet to the gearman service on localhost, but not remotely despite the firewall rules looking correct | 07:48 |
jhesketh | so I'm possibly missing something | 07:48 |
*** achanda has quit IRC | 07:49 | |
jhesketh | SergeyLukjanov, jeblair, fungi, clarkb, mordred, pleia2: anybody around per chance? | 07:49 |
openstackgerrit | Andrei V. Ostapenko proposed openstack-infra/project-config: Move from stackforge/magnetodb to openstack/magnetodb https://review.openstack.org/203524 | 07:51 |
openstackgerrit | Andrei V. Ostapenko proposed openstack-infra/project-config: Changes jobs description for MagnetoDB https://review.openstack.org/204478 | 07:51 |
kiran-r | There is a problem in jenkins | 07:54 |
kiran-r | ALl the tests are queued | 07:54 |
kiran-r | None are running | 07:54 |
*** dims_ has quit IRC | 07:54 | |
*** afazekas_ has joined #openstack-infra | 07:58 | |
flaper87 | AJaeger: is gertty working properly for you? (do you use it?) | 07:58 |
flaper87 | I'm having troubles to sync stuff | 07:58 |
jhesketh | kiran-r: yes there are some problems at the moment | 07:58 |
flaper87 | but I couldn't find a bug reporting issues | 07:58 |
flaper87 | jeblair: ^ | 07:58 |
AJaeger | flaper87: I'm not using gertty, still on my todo list. | 08:01 |
*** HeOS has quit IRC | 08:02 | |
flaper87 | AJaeger: gotcha | 08:03 |
*** jistr has joined #openstack-infra | 08:03 | |
*** gildub has quit IRC | 08:03 | |
*** scheuran has quit IRC | 08:04 | |
*** e0ne has quit IRC | 08:05 | |
AJaeger | flaper87: could you import glance translations again soonish, please? https://review.openstack.org/#/c/197206/ | 08:06 |
*** dizquierdo has quit IRC | 08:06 | |
flaper87 | AJaeger: Sir, yes sir. | 08:07 |
*** pal has joined #openstack-infra | 08:09 | |
*** dims_ has joined #openstack-infra | 08:12 | |
jhesketh | #status alert zuul jobs are currently stuck while problems with gearman are debugged | 08:12 |
openstackstatus | jhesketh: sending alert | 08:12 |
*** esmute_ has joined #openstack-infra | 08:13 | |
*** Hal has joined #openstack-infra | 08:14 | |
*** gaurangt has joined #openstack-infra | 08:14 | |
-openstackstatus- NOTICE: zuul jobs are currently stuck while problems with gearman are debugged | 08:14 | |
*** andrey-mp has joined #openstack-infra | 08:14 | |
*** ChanServ changes topic to "zuul jobs are currently stuck while problems with gearman are debugged" | 08:14 | |
*** Hal is now known as Guest83740 | 08:14 | |
*** dims_ has quit IRC | 08:16 | |
openstackstatus | jhesketh: finished sending alert | 08:16 |
*** esmute_ has quit IRC | 08:17 | |
*** derekh has joined #openstack-infra | 08:17 | |
*** scheuran has joined #openstack-infra | 08:19 | |
*** crc32 has quit IRC | 08:19 | |
*** mrmartin has joined #openstack-infra | 08:20 | |
*** heyongli has quit IRC | 08:21 | |
*** heyongli has joined #openstack-infra | 08:23 | |
*** maishsk has joined #openstack-infra | 08:23 | |
*** ihrachyshka has joined #openstack-infra | 08:23 | |
*** ihrachyshka_ has joined #openstack-infra | 08:23 | |
*** kiran-r has quit IRC | 08:24 | |
*** u_glide has joined #openstack-infra | 08:24 | |
*** heyongli has quit IRC | 08:27 | |
*** heyongli has joined #openstack-infra | 08:28 | |
*** ildikov has joined #openstack-infra | 08:29 | |
*** mpaolino has joined #openstack-infra | 08:29 | |
*** gaurangt1 has joined #openstack-infra | 08:30 | |
*** gaurangt has quit IRC | 08:31 | |
*** maishsk has quit IRC | 08:31 | |
*** boris-42 has quit IRC | 08:32 | |
*** maishsk has joined #openstack-infra | 08:32 | |
openstackgerrit | Atsushi SAKAI proposed openstack/diskimage-builder: typos on the document https://review.openstack.org/204491 | 08:32 |
*** jerryz has quit IRC | 08:33 | |
*** tmorin has joined #openstack-infra | 08:33 | |
*** bharath_ has quit IRC | 08:33 | |
tmorin | I have an issue where launchpad bugs are not autatically commented on by hudson | 08:34 |
tmorin | anyone could help ? | 08:34 |
tmorin | this happens for all changes of networking-bgpvpn project | 08:35 |
tmorin | example: https://review.openstack.org/#/c/199617/ (networking-bgpvpn project) | 08:35 |
tmorin | ping Ajaeger : would you have a look, or suggest someone who would ? | 08:35 |
*** dtantsur is now known as dtantsur|brb | 08:36 | |
*** dchen has quit IRC | 08:36 | |
openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Fix shade returning empty private_v4 https://review.openstack.org/204495 | 08:38 |
*** akasatkin has joined #openstack-infra | 08:38 | |
*** ZZelle has quit IRC | 08:39 | |
*** abregman is now known as abregman|afk | 08:40 | |
*** ZZelle has joined #openstack-infra | 08:40 | |
*** vinay_yadhav has joined #openstack-infra | 08:40 | |
*** zeih has quit IRC | 08:41 | |
*** dims_ has joined #openstack-infra | 08:41 | |
*** _nadya_ has joined #openstack-infra | 08:41 | |
*** zeih has joined #openstack-infra | 08:41 | |
openstackgerrit | Antoine Musso proposed openstack-infra/zuul: Document Gerrit labels being case sensitive https://review.openstack.org/204499 | 08:43 |
AJaeger | tmorin: your repo is networking-bgvpn, but launchpad uses bgvpn. | 08:45 |
*** dims_ has quit IRC | 08:45 | |
AJaeger | tmorin: see the greay box in http://docs.openstack.org/infra/manual/creators.html#add-the-repository-to-the-master-repositories-list | 08:46 |
tmorin | Ajaeger: yes, I had identified at the time the fact that the launchpad project had not the right name (not created by me), but failed to see what had to be done to point openstack-infra to the right LP project | 08:46 |
jhesketh | Okay tracked down the issue with zuul, going to see if I can restart and enqueue everything | 08:47 |
tmorin | Ajaeger: actually, I did this fix for networking-bagpipe-l2, but networking-bgpvpn was not done by me and had not the "group: .." statement | 08:48 |
tmorin | I will add it | 08:48 |
tmorin | thanks | 08:48 |
*** hashar has joined #openstack-infra | 08:48 | |
AJaeger | you're welcome, tmorin | 08:49 |
*** maishsk has quit IRC | 08:49 | |
openstackgerrit | Thomas Morin proposed openstack-infra/project-config: Point networking-bgpvpn to correct LP project https://review.openstack.org/204502 | 08:51 |
tmorin | AJaeger: done | 08:52 |
*** vinay_yadhav has quit IRC | 08:52 | |
tmorin | Ajaeger: for previous events on changes, I guess hudson will not add the comments and I have to update the bug by hand | 08:52 |
AJaeger | tmorin: thanks. Will review later... | 08:53 |
openstackgerrit | Joshua Hesketh proposed openstack-infra/puppet-zuul: Listen to all addresses by default https://review.openstack.org/204504 | 08:53 |
AJaeger | tmorin: Correct. Our CI (btw. we use Jenkins and Zuul and..., not hudson) will update bugs only at two times: | 08:53 |
AJaeger | When the first patch is send and when a patch merges | 08:53 |
AJaeger | So once your change is in, all patches that merge will become "Resolved" | 08:54 |
AJaeger | But Progress is only done for the very first time a change is submitted | 08:54 |
tmorin | Ajaeger: thanks for the info (mentioned hudson because saw the hudson-openstack username in LP comments) | 08:55 |
*** yamamoto has quit IRC | 08:57 | |
openstackgerrit | Joshua Hesketh proposed openstack-infra/system-config: Gearman should listen on 0.0.0.0 https://review.openstack.org/204505 | 08:57 |
*** hashar_ has joined #openstack-infra | 08:57 | |
*** maishsk has joined #openstack-infra | 08:58 | |
*** hashar has quit IRC | 08:58 | |
*** maishsk has quit IRC | 08:59 | |
*** shashank_hegde has quit IRC | 09:00 | |
*** zeih has quit IRC | 09:01 | |
AJaeger | jhesketh: thanks for all your debugging. I have to sign-off now... | 09:01 |
*** AJaeger has quit IRC | 09:01 | |
*** dguitarbite has quit IRC | 09:02 | |
*** glauco has joined #openstack-infra | 09:03 | |
*** zeih has joined #openstack-infra | 09:05 | |
*** yfried has joined #openstack-infra | 09:05 | |
*** e0ne has joined #openstack-infra | 09:07 | |
*** glauco has quit IRC | 09:07 | |
*** abregman|afk has quit IRC | 09:08 | |
*** scheuran has quit IRC | 09:08 | |
*** abregman has joined #openstack-infra | 09:08 | |
*** gus has quit IRC | 09:10 | |
*** bexelbie has joined #openstack-infra | 09:11 | |
*** maishsk has joined #openstack-infra | 09:12 | |
*** maishsk has quit IRC | 09:13 | |
*** s0nea has quit IRC | 09:14 | |
*** ihrachyshka has quit IRC | 09:15 | |
*** ihrachyshka_ is now known as ihrachyshka | 09:15 | |
*** s0nea has joined #openstack-infra | 09:15 | |
*** dims_ has joined #openstack-infra | 09:15 | |
*** maishsk has joined #openstack-infra | 09:17 | |
openstackgerrit | greghaynes proposed openstack/diskimage-builder: Document what our stable interfaces are https://review.openstack.org/198814 | 09:19 |
*** HeOS has joined #openstack-infra | 09:19 | |
*** abregman is now known as abregman|hulk_mo | 09:20 | |
*** dims_ has quit IRC | 09:20 | |
*** dims_ has joined #openstack-infra | 09:23 | |
*** scheuran has joined #openstack-infra | 09:23 | |
*** dguitarbite has joined #openstack-infra | 09:24 | |
mescanef | hi, please review changes in project-config : https://review.openstack.org/#/c/203582/ and https://review.openstack.org/#/c/204057/ | 09:24 |
*** hichihara has quit IRC | 09:24 | |
mescanef | :-) | 09:25 |
*** puranamr has joined #openstack-infra | 09:25 | |
*** dims_ has quit IRC | 09:27 | |
*** abregman|hulk_mo is now known as abregman|afk | 09:28 | |
openstackgerrit | greghaynes proposed openstack/diskimage-builder: Fail functests if refusing to run tests https://review.openstack.org/177002 | 09:28 |
*** skolekonov has joined #openstack-infra | 09:28 | |
*** gaurangt1 has quit IRC | 09:28 | |
*** coolsvap|away is now known as coolsvap | 09:28 | |
*** andrey-mp has quit IRC | 09:29 | |
*** e0ne has quit IRC | 09:29 | |
*** jasondotstar has joined #openstack-infra | 09:30 | |
*** puranamr has quit IRC | 09:30 | |
*** tmorin has quit IRC | 09:30 | |
*** fhubik is now known as fhubik_afk | 09:32 | |
openstackgerrit | greghaynes proposed openstack/diskimage-builder: Support building ACIs https://review.openstack.org/155484 | 09:32 |
*** jasondotstar has quit IRC | 09:35 | |
*** mattymo_ has joined #openstack-infra | 09:36 | |
*** dizquierdo has joined #openstack-infra | 09:37 | |
*** zeih has quit IRC | 09:43 | |
openstackgerrit | greghaynes proposed openstack-infra/glean: Dont fail is openssh is not installed https://review.openstack.org/204522 | 09:43 |
*** fhubik_afk is now known as fhubik | 09:45 | |
*** pblaho has joined #openstack-infra | 09:46 | |
*** pblaho has quit IRC | 09:47 | |
*** dims_ has joined #openstack-infra | 09:47 | |
*** pblaho has joined #openstack-infra | 09:47 | |
*** pblaho has quit IRC | 09:47 | |
*** Guest83740 has quit IRC | 09:48 | |
*** hashar_ is now known as hashar | 09:48 | |
*** bexelbie has quit IRC | 09:49 | |
*** andersonvom has quit IRC | 09:50 | |
openstackgerrit | Doron Chen proposed openstack-infra/project-config: adding a new stackforge/storlets project https://review.openstack.org/203418 | 09:51 |
*** ricky1 has joined #openstack-infra | 09:51 | |
ricky1 | o-k | 09:51 |
ricky1 | 'nuff said with the topic | 09:52 |
*** rediskin has joined #openstack-infra | 09:52 | |
*** pblaho has joined #openstack-infra | 09:55 | |
*** _nadya_ has quit IRC | 09:55 | |
*** pblaho has quit IRC | 09:56 | |
openstackgerrit | greghaynes proposed openstack-infra/project-config: Create jobs for a wheel mirror https://review.openstack.org/164927 | 09:56 |
*** gus has joined #openstack-infra | 09:57 | |
openstackgerrit | greghaynes proposed openstack-infra/system-config: Add wheel_mirror_slaves that use pypi-mirror https://review.openstack.org/165240 | 09:57 |
*** _kiran_ has joined #openstack-infra | 09:59 | |
_kiran_ | Isn't the jenkins fixed yet?? | 10:00 |
*** bexelbie has joined #openstack-infra | 10:02 | |
*** dtantsur|brb is now known as dtantsur | 10:07 | |
kozhukalov | ajaeger, pleia2, others could you please add me into the group fuel-upgrade-core, I am an author of this patch https://review.openstack.org/#/c/202541/3 | 10:09 |
*** hashar_ has joined #openstack-infra | 10:14 | |
*** kmartin has quit IRC | 10:16 | |
*** coolsvap is now known as coolsvap|away | 10:16 | |
*** hashar has quit IRC | 10:17 | |
*** e0ne has joined #openstack-infra | 10:17 | |
*** vlaza has quit IRC | 10:17 | |
*** dims_ has quit IRC | 10:18 | |
*** yfried is now known as yfried|afk | 10:18 | |
*** abregman|afk has quit IRC | 10:19 | |
*** DericHorn-HP has joined #openstack-infra | 10:20 | |
*** hashar has joined #openstack-infra | 10:21 | |
*** fhubik has quit IRC | 10:23 | |
*** fhubik has joined #openstack-infra | 10:23 | |
*** hashar_ has quit IRC | 10:24 | |
*** pelix has joined #openstack-infra | 10:26 | |
*** fhubik is now known as fhubik_afk | 10:26 | |
*** cody-somerville has quit IRC | 10:29 | |
*** ricky1 has quit IRC | 10:31 | |
*** jyuso1 has joined #openstack-infra | 10:31 | |
*** vlaza has joined #openstack-infra | 10:35 | |
*** dims_ has joined #openstack-infra | 10:36 | |
*** fhubik_afk is now known as fhubik | 10:37 | |
*** rossella_s has quit IRC | 10:37 | |
*** rossella_s has joined #openstack-infra | 10:37 | |
*** yfried|afk is now known as yfried | 10:38 | |
*** dulek has joined #openstack-infra | 10:40 | |
*** dims_ has quit IRC | 10:41 | |
*** dims_ has joined #openstack-infra | 10:41 | |
*** davideagnello has joined #openstack-infra | 10:43 | |
*** flepied has joined #openstack-infra | 10:46 | |
*** jasondotstar has joined #openstack-infra | 10:48 | |
openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/system-config: Add fuel-library channel https://review.openstack.org/200310 | 10:49 |
*** glauco_ has joined #openstack-infra | 10:50 | |
*** _kiran_ has quit IRC | 10:54 | |
*** cdent has joined #openstack-infra | 10:57 | |
*** glauco_ has quit IRC | 11:00 | |
*** tmorin has joined #openstack-infra | 11:01 | |
*** notnownikki has joined #openstack-infra | 11:02 | |
openstackgerrit | greghaynes proposed openstack/diskimage-builder: Fix test cleanup trap to cleanup tmpdir https://review.openstack.org/204550 | 11:02 |
*** davideagnello has quit IRC | 11:04 | |
*** gildub has joined #openstack-infra | 11:04 | |
*** tmorin has quit IRC | 11:05 | |
*** mwagner_lap has quit IRC | 11:10 | |
*** fhubik is now known as fhubik_afk | 11:12 | |
*** MarkAtwood has joined #openstack-infra | 11:14 | |
*** flepied has quit IRC | 11:16 | |
*** eharney has quit IRC | 11:18 | |
*** zeih has joined #openstack-infra | 11:18 | |
*** bradjones has quit IRC | 11:18 | |
*** glauco_ has joined #openstack-infra | 11:19 | |
*** MarkAtwood has quit IRC | 11:19 | |
*** flepied has joined #openstack-infra | 11:19 | |
*** bradjones has joined #openstack-infra | 11:20 | |
*** bradjones has quit IRC | 11:20 | |
*** bradjones has joined #openstack-infra | 11:20 | |
*** bexelbie has quit IRC | 11:20 | |
*** koolhead17 has joined #openstack-infra | 11:22 | |
*** fhubik_afk is now known as fhubik | 11:23 | |
*** pal has quit IRC | 11:24 | |
*** hashar has quit IRC | 11:25 | |
*** gildub has quit IRC | 11:27 | |
*** zeih has quit IRC | 11:28 | |
*** julim has joined #openstack-infra | 11:30 | |
*** DericHorn-HP has quit IRC | 11:30 | |
*** andrey-mp has joined #openstack-infra | 11:30 | |
*** k4n0 has quit IRC | 11:31 | |
*** yfried has quit IRC | 11:32 | |
odyssey4me | jhesketh aren't you meant to be on holiday? | 11:33 |
*** AJaeger has joined #openstack-infra | 11:33 | |
jhesketh | odyssey4me: that finished yesterday | 11:34 |
odyssey4me | jhesketh ah - I take it that gearman is still broken? we have one check executing, but another showing as NOT_REGISTERED | 11:35 |
jhesketh | so I found the problem with gearman, but now jobs aren't re-registering and I'm stuck again :-( | 11:36 |
*** shardy_ has joined #openstack-infra | 11:36 | |
odyssey4me | :( is there anything I can do to help? | 11:37 |
*** shardy has quit IRC | 11:37 | |
jhesketh | odyssey4me: I appreciate the offer, not entirely sure though sorry.. | 11:38 |
jhesketh | the missing jobs are registered with jenkins, and appear in gearman's status too | 11:39 |
* jhesketh is probably missing something obvious | 11:39 | |
*** mpaolino has quit IRC | 11:39 | |
*** yfried has joined #openstack-infra | 11:41 | |
*** shardy_ has quit IRC | 11:41 | |
*** shardy has joined #openstack-infra | 11:42 | |
*** hashar has joined #openstack-infra | 11:43 | |
*** gildub has joined #openstack-infra | 11:44 | |
*** mgershen has joined #openstack-infra | 11:44 | |
*** PpPyg has quit IRC | 11:44 | |
mgershen | I'm getting -1 form jenkins and a message of "NOT_REGISTERED", what should i do? https://review.openstack.org/#/c/201973/ | 11:45 |
*** zeih has joined #openstack-infra | 11:46 | |
AJaeger | mgershen: wait until everything is fixed, see topic | 11:46 |
AJaeger | mgershen: Once our CI infrastructure is fine, add a "recheck" comment | 11:46 |
mgershen | I'm a newbie, so I didn't see the connection. thank you :) | 11:46 |
*** shardy_ has joined #openstack-infra | 11:48 | |
*** shardy has quit IRC | 11:48 | |
*** eharney has joined #openstack-infra | 11:49 | |
*** eharney is now known as Guest7384 | 11:50 | |
*** abregman has joined #openstack-infra | 11:52 | |
*** david-lyle has joined #openstack-infra | 11:52 | |
*** abregman_ has joined #openstack-infra | 11:53 | |
*** shardy_ has quit IRC | 11:53 | |
*** pblaho has joined #openstack-infra | 11:53 | |
*** BharatK has quit IRC | 11:53 | |
*** pblaho has quit IRC | 11:54 | |
*** shardy has joined #openstack-infra | 11:54 | |
*** abregman_ has quit IRC | 11:54 | |
*** pblaho has joined #openstack-infra | 11:54 | |
*** abregman_ has joined #openstack-infra | 11:54 | |
*** abregman_ has quit IRC | 11:55 | |
*** abregman_ has joined #openstack-infra | 11:55 | |
*** erlon has joined #openstack-infra | 11:55 | |
*** ayoung has quit IRC | 11:55 | |
*** jaypipes has joined #openstack-infra | 11:55 | |
*** ldnunes has joined #openstack-infra | 11:57 | |
*** flepied has quit IRC | 11:58 | |
*** dprince has joined #openstack-infra | 11:59 | |
*** mpavone has joined #openstack-infra | 12:01 | |
*** mpavone has quit IRC | 12:01 | |
*** glauco_ has quit IRC | 12:03 | |
*** Guest7384 is now known as eharney | 12:03 | |
*** eharney has quit IRC | 12:03 | |
*** eharney has joined #openstack-infra | 12:03 | |
*** BharatK has joined #openstack-infra | 12:06 | |
*** bexelbie has joined #openstack-infra | 12:07 | |
*** zeih has quit IRC | 12:08 | |
*** tlian has joined #openstack-infra | 12:08 | |
*** mpaolino has joined #openstack-infra | 12:08 | |
*** koolhead17 has quit IRC | 12:09 | |
*** mrmartin has quit IRC | 12:10 | |
*** doude has quit IRC | 12:11 | |
*** doude has joined #openstack-infra | 12:11 | |
*** hashar_ has joined #openstack-infra | 12:11 | |
*** hashar has quit IRC | 12:13 | |
marekd | Hello, I have a question: When Jenkins runs gate jobs - does it always run on a fresh setup on newly created VM or something (VM?, devstack deployment) is reused? | 12:14 |
AJaeger | marekd: Each job is in a complete new VM | 12:15 |
AJaeger | (there are only a few exceptions) | 12:15 |
AJaeger | exceptions are jobs for creation of tarballs or uploading code to translation server | 12:16 |
*** BharatK has quit IRC | 12:16 | |
AJaeger | marekd: Does that help? | 12:16 |
marekd | AJaeger: ok thanks. I was asking wrt functional tests and whether we should really care about cleaning after ourselves. But since VM will be killed either way, I guess we can leave 'polluted' backends. | 12:16 |
marekd | AJaeger: yes, i think it helps, thanks! | 12:16 |
*** BharatK has joined #openstack-infra | 12:16 | |
*** andersonvom has joined #openstack-infra | 12:17 | |
AJaeger | marekd: consider that you as a developer might want to run a functional test. Clean up in for that case | 12:17 |
*** zeih has joined #openstack-infra | 12:18 | |
*** Hal has joined #openstack-infra | 12:18 | |
jhesketh | hmm, looks like it's just happening to jobs configured to run on bare-precise nodes | 12:19 |
jhesketh | but I can't see anything wrong with those nodes | 12:19 |
*** Hal is now known as Guest4225 | 12:19 | |
*** kgiusti has joined #openstack-infra | 12:19 | |
*** fhubik is now known as fhubik_afk | 12:20 | |
*** bswartz has quit IRC | 12:21 | |
*** shardy_ has joined #openstack-infra | 12:21 | |
*** shardy has quit IRC | 12:22 | |
samueldmq | good morning | 12:22 |
samueldmq | I have a change and Jenkins checks there are stated as 'NOT_REGISTERED' | 12:22 |
samueldmq | https://review.openstack.org/#/c/197980/ | 12:22 |
samueldmq | is there any known issue ? | 12:22 |
jhesketh | samueldmq: yes, CI is currently broken | 12:22 |
*** fhubik_afk is now known as fhubik | 12:22 | |
AJaeger | samueldmq: see topic | 12:23 |
*** Guest4225 has quit IRC | 12:23 | |
samueldmq | jhesketh: k thanks | 12:23 |
samueldmq | AJaeger: yes, sorry I haven't noticed | 12:24 |
*** dims_ has quit IRC | 12:24 | |
*** dguerri is now known as dguerri` | 12:25 | |
*** shardy_ has quit IRC | 12:26 | |
*** shardy has joined #openstack-infra | 12:27 | |
*** pblaho has quit IRC | 12:27 | |
*** mfalatic has joined #openstack-infra | 12:27 | |
*** pblaho has joined #openstack-infra | 12:28 | |
*** pblaho has quit IRC | 12:28 | |
openstackgerrit | Eric Harney proposed openstack-infra/project-config: Move Cinder LIO from experimental to non-voting https://review.openstack.org/204158 | 12:29 |
*** mpaolino has quit IRC | 12:29 | |
openstackgerrit | litong01 proposed openstack-infra/project-config: Add new monitor repository for OpenStack monitoring service https://review.openstack.org/203259 | 12:30 |
*** tongli has joined #openstack-infra | 12:30 | |
*** yamamoto has joined #openstack-infra | 12:31 | |
*** pblaho has joined #openstack-infra | 12:31 | |
*** pblaho has quit IRC | 12:31 | |
*** shardy has quit IRC | 12:31 | |
*** shardy has joined #openstack-infra | 12:32 | |
*** zeih has quit IRC | 12:32 | |
*** deepakcs has quit IRC | 12:32 | |
*** mwagner_lap has joined #openstack-infra | 12:33 | |
*** mpaolino has joined #openstack-infra | 12:34 | |
*** pblaho has joined #openstack-infra | 12:35 | |
*** gordc has joined #openstack-infra | 12:37 | |
*** pblaho has quit IRC | 12:38 | |
*** dkranz has joined #openstack-infra | 12:38 | |
*** bexelbie has quit IRC | 12:40 | |
*** heyongli has quit IRC | 12:40 | |
*** pblaho has joined #openstack-infra | 12:41 | |
*** heyongli has joined #openstack-infra | 12:42 | |
*** prad has joined #openstack-infra | 12:42 | |
openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Add 'Remove this story' button to detail page https://review.openstack.org/204587 | 12:43 |
*** rfolco has joined #openstack-infra | 12:43 | |
*** DericHorn-HP has joined #openstack-infra | 12:44 | |
*** pal has joined #openstack-infra | 12:48 | |
*** xyang1 has joined #openstack-infra | 12:48 | |
*** jsavak has joined #openstack-infra | 12:49 | |
*** thorst has joined #openstack-infra | 12:50 | |
*** bcourt has joined #openstack-infra | 12:51 | |
*** ParsectiX has quit IRC | 12:51 | |
*** dguerri` is now known as dguerri | 12:51 | |
*** zeih has joined #openstack-infra | 12:51 | |
*** pblaho has quit IRC | 12:52 | |
Daviey | Uhm, is something wrong with zuul? Seeing NOT_REGISTERED on Jenkins jobs on https://review.openstack.org/#/c/204492/ .. and the Zuul queue looks really short. | 12:53 |
mordred | Daviey: see channel topic | 12:54 |
*** pblaho has joined #openstack-infra | 12:54 | |
Daviey | mordred: Schoolboy error. Thanks | 12:54 |
*** bexelbie has joined #openstack-infra | 12:54 | |
mordred | Daviey: sho nuff! | 12:54 |
*** abregman_ has quit IRC | 12:55 | |
*** abregman_ has joined #openstack-infra | 12:55 | |
Daviey | Reading scrollback, i blush at this being question of the day. | 12:55 |
*** yamamoto has quit IRC | 12:55 | |
*** annegentle has joined #openstack-infra | 12:56 | |
*** unicell has joined #openstack-infra | 12:57 | |
*** iurygregory has joined #openstack-infra | 12:57 | |
jhesketh | mordred: oh heya, you're around | 12:57 |
jhesketh | mordred: have you got time to help look? | 12:57 |
*** unicell1 has quit IRC | 12:58 | |
mordred | jhesketh: I'm not really here (vacation) ... but I think those sorts of errors are usually due to something in jjb borking | 12:58 |
*** andrey-mp has quit IRC | 12:58 | |
mordred | jhesketh: I was just looking through project-config patches to see if anything landed recently | 12:58 |
jhesketh | mordred: okay, thanks for the hint | 12:58 |
mordred | jhesketh: any sense as to whetheror not there are bare-precise nodes in existence? | 12:58 |
*** radez_g0n3 is now known as radez | 12:59 | |
jhesketh | mordred: only the min, more aren't being spun up | 12:59 |
mordred | jhesketh: you may also want to try restarting a jenkins master and seeing if it'll re-register the jobs it has | 12:59 |
jhesketh | ie one | 12:59 |
jhesketh | mordred: already tried that | 12:59 |
*** yamamoto has joined #openstack-infra | 12:59 | |
openstackgerrit | Limor Stotland proposed openstack-infra/project-config: Add puppet module for Mistral https://review.openstack.org/202527 | 12:59 |
*** bknudson has joined #openstack-infra | 12:59 | |
mordred | jhesketh: bong | 12:59 |
*** ddieterl_ has quit IRC | 12:59 | |
mordred | jhesketh: and it's just bare-precise? or just for jobs taht have bare-precise mentioned anywhere? | 13:00 |
jhesketh | mordred: looking at the nodes that jobs are registered for on gearman it appears to only be bare-precise | 13:00 |
jhesketh | but it could be a red herring | 13:01 |
mordred | nod | 13:01 |
* greghaynes is curious if theres more context than "gearman is broken" | 13:01 | |
greghaynes | didnt see much in scrollback | 13:02 |
jhesketh | greghaynes: earlier gearman was broken when I restarted zuul and a recent change to gear made gearman bind to a single host | 13:02 |
greghaynes | ah | 13:02 |
jhesketh | greghaynes: my fixes for that are here https://review.openstack.org/#/c/204504/ https://review.openstack.org/#/c/204505/ | 13:02 |
greghaynes | hah | 13:02 |
*** eharney has quit IRC | 13:02 | |
jhesketh | in the recovery from the breakage not all jobs are registering with zuul | 13:02 |
mordred | jhesketh: I think I don't have much more ideas in my vacation head - hopefully fungi will awaken soon | 13:03 |
AJaeger | jhesketh: it started with Zuul having internal data structures broken, then gearman. Was their any cloud outage that hit us somehow? | 13:03 |
jhesketh | and I haven't gotten any further information to do a new status yet | 13:03 |
jhesketh | mordred: no trouble, thanks anyway! Go enjoy your time off :-) | 13:03 |
*** mgershen has quit IRC | 13:03 | |
greghaynes | jhesketh: ah, so those fixes got applied manually? | 13:03 |
*** MarkAtwood has joined #openstack-infra | 13:03 | |
jhesketh | AJaeger: not that I can see... I suspect that was due to a race in reloading zuul but my intention was to unwedge the gate first :-( | 13:03 |
jhesketh | greghaynes: yes | 13:04 |
openstackgerrit | Monty Taylor proposed openstack-infra/puppet-zuul: Listen to all addresses by default https://review.openstack.org/204504 | 13:04 |
AJaeger | jhesketh: I'll send an email to the mailing list for those not reading IRC topics ;) | 13:04 |
jhesketh | AJaeger: thanks, appreciated | 13:04 |
openstackgerrit | Monty Taylor proposed openstack-infra/puppet-zuul: Listen to all addresses by default https://review.openstack.org/204504 | 13:04 |
mordred | jhesketh: (those are just commit message fixes) | 13:04 |
jhesketh | mordred: okay thanks | 13:04 |
*** fhubik is now known as fhubik_afk | 13:04 | |
*** glauco_ has joined #openstack-infra | 13:05 | |
*** flepied has joined #openstack-infra | 13:05 | |
*** glauco_ has quit IRC | 13:05 | |
*** glauco_ has joined #openstack-infra | 13:06 | |
openstackgerrit | Imre Farkas proposed openstack-infra/project-config: Add openstack/python-dracclient project https://review.openstack.org/203991 | 13:06 |
*** dkranz has quit IRC | 13:06 | |
*** glauco_ has quit IRC | 13:06 | |
*** Hal has joined #openstack-infra | 13:07 | |
*** glauco_ has joined #openstack-infra | 13:07 | |
*** glauco_ has quit IRC | 13:07 | |
*** Hal is now known as Guest3787 | 13:07 | |
*** _nadya_ has joined #openstack-infra | 13:07 | |
*** romcheg has joined #openstack-infra | 13:07 | |
*** gildub has quit IRC | 13:08 | |
*** koolhead17 has joined #openstack-infra | 13:08 | |
*** MarkAtwood has quit IRC | 13:08 | |
*** glauco has joined #openstack-infra | 13:09 | |
*** skolekonov has quit IRC | 13:13 | |
*** rlandy has joined #openstack-infra | 13:13 | |
*** ParsectiX has joined #openstack-infra | 13:14 | |
*** eharney has joined #openstack-infra | 13:15 | |
*** eharney is now known as Guest57241 | 13:15 | |
*** funzo has joined #openstack-infra | 13:16 | |
*** baoli has joined #openstack-infra | 13:18 | |
*** dkranz has joined #openstack-infra | 13:19 | |
*** DericHorn-HP has quit IRC | 13:19 | |
*** baoli has quit IRC | 13:21 | |
*** Guest57241 is now known as eharney | 13:21 | |
*** eharney has quit IRC | 13:21 | |
*** eharney has joined #openstack-infra | 13:21 | |
*** amotoki has quit IRC | 13:25 | |
*** ParsectiX has quit IRC | 13:26 | |
*** david-lyle has quit IRC | 13:27 | |
*** puranamr has joined #openstack-infra | 13:27 | |
pleia2 | kozhukalov: I have added the account associated with vkozhukalov@mirantis.com to the fuel-upgrade-core group | 13:28 |
*** peristeri has joined #openstack-infra | 13:28 | |
*** mriedem has joined #openstack-infra | 13:29 | |
*** koolhead17 has quit IRC | 13:29 | |
*** yolanda has joined #openstack-infra | 13:29 | |
*** gildub has joined #openstack-infra | 13:29 | |
jhesketh | pleia2: oh hey, would you be able to help debug the gate per-chance? | 13:30 |
*** mjturek1 has joined #openstack-infra | 13:31 | |
*** puranamr has quit IRC | 13:32 | |
AJaeger | morning, pleia2 ! | 13:32 |
greghaynes | clarkb: Hey, so im playing with getting dib images booting with qemu in a test, curious wat you think about for networking | 13:33 |
annegentle | AJaeger: so it's CI down just for doc builds? | 13:33 |
annegentle | AJaeger: anything I can do to help? | 13:33 |
*** abregman_ has quit IRC | 13:34 | |
*** abregman has quit IRC | 13:34 | |
jhesketh | annegentle: it's down across the board sorry | 13:34 |
annegentle | jhesketh: bummer | 13:34 |
AJaeger | annegentle: down everywhere;) | 13:35 |
annegentle | I can't help much debugging that :) | 13:35 |
*** gildub has quit IRC | 13:35 | |
AJaeger | annegentle: wake up fungi or jeblair ;) | 13:35 |
annegentle | heh | 13:35 |
*** abregman has joined #openstack-infra | 13:35 | |
odyssey4me | jhesketh oddly, I just saw one of the previously 'NOT_REGISTERED' jobs run properly | 13:35 |
*** vlaza has quit IRC | 13:36 | |
jhesketh | odyssey4me: oh, do you have a link? | 13:36 |
odyssey4me | jhesketh https://review.openstack.org/204597 | 13:36 |
AJaeger | jhesketh: there are nova jobs now with registered jobs! | 13:36 |
odyssey4me | jhesketh https://jenkins07.openstack.org/job/os-ansible-deployment-check-lint/489/ was previously unregistered | 13:36 |
AJaeger | seems we have a self-healing cloud ;) | 13:37 |
jhesketh | yeah, looks like there are a few that have come up now | 13:37 |
AJaeger | jhesketh: check the bottom of the check queue on status.o.o/zuul | 13:37 |
annegentle | woo | 13:37 |
AJaeger | pleia2: what have you done? ;) | 13:37 |
AJaeger | or was it the friendly presence of annegentle ? ;) | 13:38 |
greghaynes | huh | 13:38 |
annegentle | heh | 13:38 |
annegentle | doubtful! | 13:38 |
greghaynes | thats... odd | 13:38 |
*** heyongli has quit IRC | 13:40 | |
*** jaypipes has quit IRC | 13:40 | |
samueldmq | so CI can repair itself .. looks like some advanced AI approach being used :) | 13:40 |
*** heyongli has joined #openstack-infra | 13:41 | |
*** markus_z has joined #openstack-infra | 13:41 | |
greghaynes | I wonder if there was just some network connectivity issues on the gearman server? | 13:41 |
annegentle | look out it'll be self aware on J-Day | 13:41 |
AJaeger | samueldmq: CI breaks itself, so it should heal itself as well ;) | 13:41 |
samueldmq | AJaeger: yeah that's pretty fair :) | 13:42 |
frickler | jhesketh: did you do anything at 13:05UTC? seems most jobs that are younger than that are working | 13:42 |
*** aysyd has joined #openstack-infra | 13:42 | |
*** ddieterly has joined #openstack-infra | 13:42 | |
jhesketh | frickler: hmm, I was poking at the precise nodes to see why they weren't registering | 13:43 |
jhesketh | I cycled a few of them through | 13:43 |
jhesketh | maybe that kicked things off | 13:43 |
jhesketh | but I couldn't say way | 13:43 |
jhesketh | *why | 13:43 |
openstackgerrit | will soula proposed openstack-infra/jenkins-job-builder: Add support for whitesource plugin https://review.openstack.org/204615 | 13:44 |
*** jcoufal has quit IRC | 13:45 | |
AJaeger | jhesketh, can you now requeue the jobs from this morning - or is that too late now? | 13:45 |
AJaeger | Or is it too early to put more load on? | 13:45 |
greghaynes | The queue is huge | 13:45 |
greghaynes | so... itll be a while | 13:45 |
*** BharatK has quit IRC | 13:45 | |
jhesketh | yeah so I have the previous queues snapshotted, but want to make sure it's actually fixed before I place them back in | 13:46 |
frickler | jhesketh: maybe the jenkins masters needed the trigger of a new slave in order to refresh the gearman connection | 13:46 |
AJaeger | jhesketh: understood | 13:46 |
jhesketh | and then the gate is going to take a while to catch up | 13:46 |
*** hashar_ has quit IRC | 13:47 | |
jhesketh | frickler: maybe, although I had tried that earlier, so not sure what I did differently | 13:47 |
*** ddieterly has quit IRC | 13:47 | |
*** kushal has quit IRC | 13:48 | |
*** dims_ has joined #openstack-infra | 13:48 | |
mriedem | are we ok to recheck? | 13:48 |
jhesketh | mriedem: probably | 13:48 |
*** tvanderwiel has joined #openstack-infra | 13:49 | |
*** ParsectiX has joined #openstack-infra | 13:50 | |
*** signed8bit has joined #openstack-infra | 13:52 | |
*** mrunge has quit IRC | 13:52 | |
*** vlaza has joined #openstack-infra | 13:52 | |
mriedem | ok cool | 13:55 |
*** zz_dimtruck is now known as dimtruck | 13:55 | |
*** pblaho has quit IRC | 13:56 | |
*** sabeen has quit IRC | 13:56 | |
*** fhubik_afk is now known as fhubik | 13:57 | |
*** evgenyl has left #openstack-infra | 13:57 | |
*** baoli has joined #openstack-infra | 13:57 | |
*** pblaho has joined #openstack-infra | 13:58 | |
*** bnemec has joined #openstack-infra | 13:58 | |
*** DericHorn-HP has joined #openstack-infra | 13:58 | |
*** mrmartin has joined #openstack-infra | 13:58 | |
andersonvom | hey folks, I'm trying to "run my own" (http://docs.openstack.org/infra/system-config/running-your-own.html) but I keep getting missing dependencies while installing puppet master (http://docs.openstack.org/infra/system-config/puppet.html#id2) | 13:59 |
*** sherif has joined #openstack-infra | 13:59 | |
andersonvom | is there any particular distro that is better suited to follow these instructions? | 13:59 |
openstackgerrit | Timothy R. Chavez proposed openstack-infra/puppet-cgit: Add better defaults for clone-prefix https://review.openstack.org/204245 | 14:00 |
*** dustins has joined #openstack-infra | 14:01 | |
AJaeger | jhesketh: do you want to answer my email message once you're happy? Please also update the topic | 14:02 |
*** claudiub has joined #openstack-infra | 14:03 | |
*** btully has joined #openstack-infra | 14:03 | |
pleia2 | jhesketh, AJaeger - sorry, stepped out to breakfast right after fulfilling kozhukalov's request, but glad to see it fixed itself! | 14:05 |
openstackgerrit | Paul Belanger proposed openstack-infra/puppet-exim: Add Gemfile and puppet 4 checks https://review.openstack.org/204243 | 14:05 |
jhesketh | AJaeger: will do, just waiting for the green to reach the top of the queue | 14:05 |
AJaeger | thanks, jhesketh ! | 14:06 |
*** bknudson has quit IRC | 14:06 | |
*** tsg has joined #openstack-infra | 14:07 | |
*** hashar has joined #openstack-infra | 14:07 | |
bauzas | heya, zuul still blocked or can we recheck ? | 14:07 |
*** bswartz has joined #openstack-infra | 14:07 | |
*** jaypipes has joined #openstack-infra | 14:07 | |
openstackgerrit | Merged openstack-infra/project-config: Rearrange the tempest jobs under right section https://review.openstack.org/204314 | 14:08 |
*** palendae has joined #openstack-infra | 14:08 | |
palendae | Hello - we have a review that's been +W since Friday, but doesn't appear to have shown up in Zuul's queue. It's https://review.openstack.org/#/c/202258/, and to my knowledge this change skips our defined gating | 14:09 |
*** amitgandhinz has joined #openstack-infra | 14:09 | |
openstackgerrit | Matthew Treinish proposed openstack-infra/irc-meetings: Change the alternate QA meeting time to 9:00 UTC https://review.openstack.org/204628 | 14:10 |
jhesketh | bauzas: should be okay to recheck | 14:10 |
openstackgerrit | Paul Belanger proposed openstack-infra/puppet-etherpad_lite: Add Gemfile and puppet 4 checks https://review.openstack.org/204242 | 14:10 |
bauzas | jhesketh: ack, cool, thanks for the update | 14:10 |
jhesketh | palendae: since Friday is a bit odd, probably okay to do a recheck though | 14:10 |
frickler | palendae: zuul was restarted earlier today, probably need to remove the +W and readd it | 14:10 |
jhesketh | palendae: it won't be in zuul's queue anymore anyway since it was restarted | 14:11 |
palendae | frickler, jhesketh: When was it restarted? I show a new +W added at 6:25AM EDT today | 14:11 |
pleia2 | jhesketh: care to update status? | 14:11 |
AJaeger | palendae: afterwards | 14:11 |
*** mestery has joined #openstack-infra | 14:11 | |
jhesketh | pleia2: yeah I'll put out another alert and then do an okay once green is at the top | 14:12 |
palendae | AJaeger: Ok, thanks | 14:12 |
pleia2 | jhesketh: great, thanks | 14:12 |
AJaeger | palendae: so, toggle the +A. And wait for a few hours | 14:12 |
palendae | AJaeger: Thanks, will do! | 14:12 |
*** jcoufal has joined #openstack-infra | 14:12 | |
jhesketh | #status alert CI is currently recovering from an outage overnight. It is safe to recheck results with NOT_REGISTERED errors. It may take some time for zuul to work through the backlog. | 14:12 |
openstackstatus | jhesketh: sending alert | 14:12 |
mtreinish | tonyb: want to review https://review.openstack.org/#/c/204628/ | 14:13 |
mtreinish | I assume you can do something on that patch | 14:13 |
*** sabeen has joined #openstack-infra | 14:14 | |
*** bknudson has joined #openstack-infra | 14:14 | |
-openstackstatus- NOTICE: CI is currently recovering from an outage overnight. It is safe to recheck results with NOT_REGISTERED errors. It may take some time for zuul to work through the backlog. | 14:15 | |
*** ChanServ changes topic to "CI is currently recovering from an outage overnight. It is safe to recheck results with NOT_REGISTERED errors. It may take some time for zuul to work through the backlog." | 14:15 | |
openstackgerrit | Paul Belanger proposed openstack-infra/puppet-dashboard: Add Gemfile and puppet 4 checks https://review.openstack.org/204241 | 14:15 |
AJaeger | palendae: not recheck - ask to add another +A or toggle the +A | 14:16 |
*** pblaho_ has joined #openstack-infra | 14:16 | |
*** boris-42 has joined #openstack-infra | 14:16 | |
*** ParsectiX has quit IRC | 14:16 | |
palendae | AJaeger: Asked, added my own +A | 14:16 |
AJaeger | palendae: ok, then let's wait - just saw the "recheck" comment by someboy | 14:16 |
palendae | That was me | 14:16 |
leseb_ | hi all, https://review.openstack.org/#/c/188768/ :) it'd be nice if someone could merge this, thanks in advance | 14:17 |
AJaeger | s/someboy/somebody/ | 14:17 |
palendae | :) | 14:17 |
openstackstatus | jhesketh: finished sending alert | 14:17 |
leseb_ | AJaeger: I don't really know who to ping for this but I'm sure you do, some if you have a minute :), thanks! | 14:17 |
*** pblaho has quit IRC | 14:17 | |
openstackgerrit | Paul Belanger proposed openstack-infra/puppet-zuul: Update from puppet-modules-sync https://review.openstack.org/204163 | 14:18 |
boris-42 | AJaeger: hi | 14:18 |
*** glauco has quit IRC | 14:18 | |
boris-42 | AJaeger: is infra going to change back topic of rally chat? | 14:18 |
boris-42 | AJaeger: * Topic for #openstack-rally set by openstackstatus ... | 14:19 |
boris-42 | AJaeger: I mean we have quite important information in it and I believe it shouldn't be changed without any questions of rally team.. | 14:19 |
*** sigmavirus24_awa is now known as sigmavirus24 | 14:20 | |
jhesketh | boris-42: normal topics will be restored once the alert has finished | 14:20 |
openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: [WIP] Switch overcloud-f21-nonha to instack https://review.openstack.org/185151 | 14:21 |
*** mtanino has joined #openstack-infra | 14:21 | |
boris-42 | jhesketh: great thanks | 14:21 |
AJaeger | boris-42: the infra team changes the topic of all changes when it's important. | 14:22 |
AJaeger | pleia2: could you review leseb_'s change, please? | 14:22 |
leseb_ | pleia2: https://review.openstack.org/#/c/188768/ :) thanks in advance | 14:23 |
pleia2 | sure | 14:23 |
openstackgerrit | Marek Denis proposed openstack/requirements: Constrain oauthlib versions https://review.openstack.org/204633 | 14:23 |
AJaeger | pleia2, leseb_ : I just wonder whether this shouldn't be part of openstack namespace... | 14:24 |
pleia2 | AJaeger: do you know how the plugin: field works in projects.yaml? seems to be inconsistent usage among the devstack plugins | 14:25 |
*** shardy_ has joined #openstack-infra | 14:25 | |
clarkb | greghaynes dhcp ? | 14:26 |
leseb_ | AJaeger: well russellb discussed that with dtroyer_zz and the outcome was to first push this here | 14:26 |
AJaeger | pleia2: yes, there's inconsistency and I don't know what's best. Should we ask jeblair for help? | 14:26 |
leseb_ | perhaps russellb can elaborate if needed | 14:26 |
pleia2 | AJaeger: I'll make a note inline and maybe someone else can have a look | 14:27 |
*** shardy has quit IRC | 14:27 | |
AJaeger | leseb_: moving from stackforge to openstack is expensive, that's why we try to move it to openstack if possible. | 14:27 |
AJaeger | pleia2: could you ping jeblair once he's awake? I'm going offline soonish... | 14:27 |
greghaynes | clarkb: ohai. So, dhcp to where? The thing is - we dont need a full devstack so this is just one off run qemu-system-x86_64 | 14:28 |
*** koolhead17 has joined #openstack-infra | 14:28 | |
*** koolhead17 has quit IRC | 14:28 | |
*** koolhead17 has joined #openstack-infra | 14:28 | |
pleia2 | AJaeger: I'm off to the convention center soon, but I'll see what I can do | 14:28 |
*** zeih has quit IRC | 14:28 | |
greghaynes | clarkb: I am considering just add a br, slap some static network in the image, boot with -net tap | 14:28 |
clarkb | greghaynes I eould devstack just for that but elazy and using virsh is bad for my sanity | 14:28 |
greghaynes | heh | 14:28 |
clarkb | you could just do the arp check | 14:29 |
AJaeger | pleia2: understood - thanks. Enjoy oscon! | 14:29 |
clarkb | thst dhcp from libvirt? | 14:29 |
*** yamamoto has quit IRC | 14:29 | |
openstackgerrit | Tony Breeds proposed openstack-infra/irc-meetings: qa-team-meeting: Remove unneeded comment https://review.openstack.org/204636 | 14:29 |
pleia2 | AJaeger: thanks :) this morning is it for me, have a flight in 6 hours | 14:29 |
greghaynes | clarkb: well, im not even running a dhcp at that point, I could add one... | 14:29 |
clarkb | if you use virsh I think it dors it | 14:30 |
greghaynes | ah | 14:30 |
greghaynes | I wish you had replied about 10 mins sooner because I might have just gotten this br deal working | 14:30 |
greghaynes | virsh might still be a good idea though | 14:30 |
*** shardy_ has quit IRC | 14:30 | |
*** u_glide has left #openstack-infra | 14:31 | |
jhesketh | #status ok CI is slowly recovering, please be patient while the backlog is worked through. | 14:31 |
openstackstatus | jhesketh: sending ok | 14:31 |
*** shardy has joined #openstack-infra | 14:31 | |
*** sabeen3 has joined #openstack-infra | 14:32 | |
*** yamamoto has joined #openstack-infra | 14:32 | |
*** sabeen has quit IRC | 14:32 | |
*** yolanda has quit IRC | 14:32 | |
-openstackstatus- NOTICE: CI is slowly recovering, please be patient while the backlog is worked through. | 14:33 | |
*** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/" | 14:33 | |
openstackgerrit | greghaynes proposed openstack/diskimage-builder: Perform a booting test for our images https://review.openstack.org/204639 | 14:35 |
greghaynes | clarkb: ^ | 14:35 |
openstackgerrit | Merged openstack-infra/irc-meetings: Change the alternate QA meeting time to 9:00 UTC https://review.openstack.org/204628 | 14:35 |
openstackstatus | jhesketh: finished sending ok | 14:35 |
*** koolhead17 has quit IRC | 14:36 | |
*** pal has quit IRC | 14:38 | |
*** bexelbie has quit IRC | 14:38 | |
AJaeger | wow, 248 jobs in the check queue ;( | 14:39 |
Sam-I-Am | chugga | 14:39 |
*** pcrews has quit IRC | 14:41 | |
*** sherif has quit IRC | 14:41 | |
elmiko | ooph | 14:42 |
*** pcrews has joined #openstack-infra | 14:42 | |
AJaeger | 301... | 14:42 |
Daviey | AJaeger: if it were running on some sort of cloud, we could scale out more workers to help with the temp higher demand .. :) | 14:42 |
AJaeger | Daviey: do you have a spare cloud? WE use Rackspace and HP public clouds already... | 14:43 |
Daviey | AJaeger: Sure, I have one under my desk. Help yourself :) | 14:44 |
AJaeger | Daviey: just plug it in - send patches for project-config and system-config so that we can use it;) | 14:44 |
*** ayoung has joined #openstack-infra | 14:45 | |
Daviey | :) | 14:45 |
pleia2 | our infrastructure does support easy scaling across the cloud, but as AJaeger says, we don't have the resources to expand further on the clouds provided to the project | 14:45 |
Daviey | pleia2: Oh i know.. I am familiar with nodepool.. | 14:46 |
AJaeger | Daviey: see http://specs.openstack.org/openstack-infra/infra-specs/specs/infra-cloud.html for next steps - you should plugin your rack in that framework... | 14:47 |
jhesketh | AJaeger: yes, I'm re-enqueuing the jobs from earlier | 14:47 |
Daviey | AJaeger: well that backfired... I didn't know about that spec | 14:47 |
*** baoli has quit IRC | 14:48 | |
AJaeger | jhesketh: Thanks! Now passed 400... | 14:48 |
pleia2 | Daviey: no more jokes from you! | 14:48 |
*** baoli_ has joined #openstack-infra | 14:49 | |
Daviey | pleia2: Indeed. | 14:49 |
AJaeger | Daviey: we have it on record now that you're going to help with that ;) | 14:49 |
pleia2 | :) | 14:49 |
*** dguitarbite has quit IRC | 14:49 | |
pleia2 | hehe | 14:49 |
*** vlaza has left #openstack-infra | 14:49 | |
*** davidjc has joined #openstack-infra | 14:51 | |
*** signed8bit is now known as signed8bit_ZZZzz | 14:52 | |
*** signed8bit_ZZZzz is now known as signed8bit | 14:53 | |
*** tonytan4ever has joined #openstack-infra | 14:53 | |
marekd | lifeless: Hi. I added another comment on https://review.openstack.org/#/c/204633/ . So, keystoneclint fails on master version as oauthlib changes signatures of few of their functions. I doubt they will get back to the old signatures (or those that are compatible with ksc) in future versions, so I think we should keep <1.0.0 until we fix keystone. | 14:53 |
*** glauco has joined #openstack-infra | 14:55 | |
Daviey | AJaeger / pleia2: Hold me to it, and i'll provide you a cloud running on underclocked raspberry pi's :) | 14:55 |
elmiko | lol | 14:55 |
AJaeger | Daviey: I heard that compute nodes with raspberry pi's are not that great since the CPU misses some virtualization support. So, you can use them as controllers... | 14:56 |
zaro | hashar: had a question for you on this, https://review.openstack.org/#/c/130166/ | 14:57 |
*** maishsk has quit IRC | 14:58 | |
*** AJaeger has quit IRC | 14:58 | |
openstackgerrit | Sean Dague proposed openstack-infra/devstack-gate: grenade partial multinode support https://review.openstack.org/199091 | 14:59 |
openstackgerrit | Sean Dague proposed openstack-infra/devstack-gate: convert variables to local in setup_multinode_connectivity https://review.openstack.org/204655 | 14:59 |
*** andrey-mp has joined #openstack-infra | 14:59 | |
*** marzif has quit IRC | 15:00 | |
*** mfmcdonagh has quit IRC | 15:00 | |
*** pelix has quit IRC | 15:01 | |
*** patrickeast has joined #openstack-infra | 15:02 | |
*** dizquierdo has quit IRC | 15:02 | |
*** palendae has left #openstack-infra | 15:02 | |
*** jsavak has quit IRC | 15:03 | |
hashar | zaro: good morning :) | 15:04 |
hashar | zaro: ahh | 15:05 |
hashar | zaro: so that is really a lame wrapper I added in my local copy to easily build the plugin | 15:05 |
*** pal has joined #openstack-infra | 15:07 | |
jeblair | jhesketh: thanks for dealing with the outage | 15:08 |
jeblair | jhesketh, mordred: i think we should revert the zuul patch and use a different solution there -- i think zuul should have a configuration for what gearman server to connect to, and a separate config option for the ip address to listen on | 15:08 |
mordred | jeblair: wfm | 15:08 |
*** spfifxp is now known as spiffxp | 15:09 | |
*** jamesmcarthur has joined #openstack-infra | 15:09 | |
*** koolhead17 has joined #openstack-infra | 15:09 | |
*** david-lyle has joined #openstack-infra | 15:09 | |
jeblair | (mostly because making outgoing connections to 0.0.0.0 seems weird to me) | 15:10 |
openstackgerrit | Matt Riedemann proposed openstack/requirements: Bump minimum os-brick>=0.3.1 https://review.openstack.org/203193 | 15:10 |
mriedem | thingee: hemnafk: ^ | 15:10 |
mriedem | adds the depends-on to the release change | 15:10 |
hashar | zaro: gave it a try. git describe yields 0.1.1-1-g9fea5a0 , your script 0.1.1.1.9fea5a0 | 15:10 |
hashar | zaro: seems good :) | 15:11 |
*** tsg has quit IRC | 15:11 | |
mordred | jeblair: the internet should be able to infer your intentions | 15:11 |
jhesketh | jeblair: no trouble | 15:11 |
jhesketh | jeblair: I agree | 15:12 |
*** dprince has quit IRC | 15:12 | |
jeblair | k, i'll write that patch | 15:12 |
jhesketh | thanks | 15:12 |
jhesketh | I'm going to get these last jobs back into the queue and then probably head to bed now things seem to be stabilizing | 15:13 |
openstackgerrit | Antoine Musso proposed openstack-infra/gearman-plugin: Script to easily build a package https://review.openstack.org/130166 | 15:13 |
*** mrmartin has quit IRC | 15:13 | |
*** annegentle has quit IRC | 15:13 | |
hashar | zaro: done! | 15:13 |
*** annegentle has joined #openstack-infra | 15:14 | |
*** vlaza has joined #openstack-infra | 15:14 | |
*** nijaba has quit IRC | 15:16 | |
*** mwagner_lap has quit IRC | 15:16 | |
*** nijaba has joined #openstack-infra | 15:16 | |
openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: [WIP] Switch overcloud-f21-nonha to instack https://review.openstack.org/185151 | 15:18 |
*** tsekiyama has joined #openstack-infra | 15:19 | |
*** glauco has quit IRC | 15:20 | |
*** koolhead17 has quit IRC | 15:20 | |
*** annegentle has quit IRC | 15:20 | |
*** arxcruz has joined #openstack-infra | 15:21 | |
jeblair | there are a lot of unused ready nodes, mostly on jenkins07, that probably missed being actually attached to jenkins; i'm deleting them | 15:22 |
*** kushal has joined #openstack-infra | 15:22 | |
*** laughterwym has joined #openstack-infra | 15:23 | |
openstackgerrit | greghaynes proposed openstack/diskimage-builder: Perform a booting test for our images https://review.openstack.org/204639 | 15:24 |
zaro | hashar: great, thanks. | 15:24 |
*** glauco has joined #openstack-infra | 15:25 | |
*** austin81 has joined #openstack-infra | 15:26 | |
*** pelix has joined #openstack-infra | 15:26 | |
*** sshnaidm has quit IRC | 15:27 | |
*** doug-fish has joined #openstack-infra | 15:28 | |
*** pblaho_ is now known as pblaho | 15:28 | |
openstackgerrit | greghaynes proposed openstack/diskimage-builder: Perform a booting test for our images https://review.openstack.org/204639 | 15:28 |
*** harlowja_at_home has joined #openstack-infra | 15:29 | |
*** pcrews has quit IRC | 15:29 | |
*** glauco has quit IRC | 15:30 | |
*** sherif has joined #openstack-infra | 15:31 | |
*** hashar is now known as hasharConfcall | 15:31 | |
thingee | anteaya: ping | 15:32 |
anteaya | thingee: hi | 15:32 |
anteaya | thanks | 15:32 |
*** BharatK has joined #openstack-infra | 15:32 | |
anteaya | so you have some cis you would like disabled | 15:33 |
anteaya | let's start there | 15:33 |
*** hemnafk is now known as hemna | 15:33 | |
*** jsavak has joined #openstack-infra | 15:34 | |
thingee | anteaya: yes, IBM XIV CI | 15:34 |
*** spredzy is now known as spredzy|afk | 15:34 | |
thingee | anteaya: and Microsoft iSCSI CI | 15:34 |
*** baoli_ has quit IRC | 15:35 | |
anteaya | great thanks | 15:36 |
anteaya | have we reasons why? | 15:36 |
anteaya | you asking is good enough for me but when we announce to the mailing list it is nice to have something to point to | 15:36 |
*** baoli has joined #openstack-infra | 15:36 | |
*** abregman has quit IRC | 15:37 | |
jeblair | jenkins07 is not sending zmq events, so i have put it into prepare-for-shutdown mode and will restart it laster when jobs its running are finished | 15:37 |
*** armax has joined #openstack-infra | 15:37 | |
anteaya | jeblair: thank you | 15:37 |
*** gsagie_ has joined #openstack-infra | 15:38 | |
*** kiran-r has joined #openstack-infra | 15:38 | |
*** dtantsur is now known as dtantsur|afk | 15:38 | |
Swanson | Is the lovely logs.openstack.org log viewer something that is available? Or whatever it is that converts the logs into that html? | 15:39 |
*** andersonvom has quit IRC | 15:39 | |
*** ashleighfarnham has joined #openstack-infra | 15:39 | |
anteaya | I have a problem, when I run curl -X GET 'https://review.openstack.org/accounts/alonma@il.ibm.com' to get the gerrit account id for IBM XIV CI I get not found | 15:39 |
*** sergmelikyan has quit IRC | 15:40 | |
anteaya | not sure who is around to look at the gerrit db to find and disable IBM XIV CI but my usual method of finding their gerrit id is failing | 15:40 |
jeblair | anteaya: i'll give it a shot | 15:40 |
*** _nadya_ has quit IRC | 15:41 | |
anteaya | jeblair: thanks | 15:41 |
jeblair | anteaya: that email is associated with both 10213 and 12493 | 15:41 |
*** _nadya_ has joined #openstack-infra | 15:42 | |
zaro | hasharConfcall: i think we are deadlocked on this change since pelix and i both worked on it. no other cores to move it along. wondering if you could help out when you have a moment? https://review.openstack.org/#/c/151472/ | 15:42 |
anteaya | jeblair: wonderful | 15:42 |
jeblair | one of them looks like an old-style manually created account (12493) and the other one looks like a launchpad account | 15:43 |
anteaya | the microsoft ci gerrit id is 13394, it that also could be disabled: http://paste.openstack.org/show/400440/ | 15:43 |
anteaya | jeblair: hmmmm | 15:44 |
*** ihrachyshka has quit IRC | 15:44 | |
*** chmouel has quit IRC | 15:44 | |
*** baoli has quit IRC | 15:45 | |
*** andersonvom has joined #openstack-infra | 15:45 | |
*** jsavak has quit IRC | 15:46 | |
*** _nadya_ has quit IRC | 15:46 | |
*** jsavak has joined #openstack-infra | 15:46 | |
*** baoli has joined #openstack-infra | 15:46 | |
jhesketh | okay all the queues have been restored... not a fun backlog sorry :-( | 15:46 |
thingee | anteaya: oh sorry, I sent some emails to the third party list, but I can resend them if they got stuck for some reason. | 15:47 |
jeblair | i think i should delete the email address from 12493 and then disable it | 15:47 |
anteaya | jhesketh: thanks for all your help | 15:47 |
anteaya | jeblair: I support that direction | 15:47 |
*** Guest23745 has joined #openstack-infra | 15:47 | |
anteaya | thingee: yes please resend | 15:47 |
*** scheuran has quit IRC | 15:47 | |
jhesketh | no trouble, that's why I'm here :-) | 15:47 |
jhesketh | (I hope) | 15:47 |
*** Guest23745 has quit IRC | 15:47 | |
anteaya | jhesketh: :) | 15:47 |
anteaya | jhesketh: so glad you're here | 15:47 |
jhesketh | anyway, see y'all in the morning | 15:47 |
anteaya | jhesketh: night | 15:48 |
jeblair | jhesketh: thanks, and sleep well! :) | 15:48 |
*** achanda has joined #openstack-infra | 15:48 | |
anteaya | thingee: they were sent the same day as you told me to ignore your post to third-party-announce, so I rejected them | 15:48 |
anteaya | thingee: I didn't realize there were some you wanted posted and some you wanted ignored, my fault | 15:49 |
jeblair | flushing gerrit's caches | 15:49 |
anteaya | thingee: so please resend now that I understand better | 15:49 |
anteaya | jeblair: did you disable microsoft too? | 15:49 |
anteaya | gerrit id 13394? | 15:49 |
jeblair | anteaya: not yet, still doing db repair | 15:49 |
anteaya | ah k, thank you | 15:49 |
*** tsg has joined #openstack-infra | 15:49 | |
thingee | anteaya: done | 15:50 |
anteaya | thingee: thank you | 15:50 |
anteaya | once they are disabled I will approve them | 15:50 |
anteaya | approve the ml posts | 15:50 |
*** patrickeast has quit IRC | 15:50 | |
*** abregman has joined #openstack-infra | 15:50 | |
*** Sukhdev has joined #openstack-infra | 15:50 | |
*** dimtruck is now known as zz_dimtruck | 15:51 | |
*** chmouel has joined #openstack-infra | 15:51 | |
odyssey4me | oh dear, we have several reports of review.o.o croaking across different regions - is anyone else seeing this? | 15:51 |
*** DericHorn-HP has quit IRC | 15:51 | |
anteaya | odyssey4me: we just flushed the db caches | 15:51 |
jeblair | odyssey4me: can you elaborate on 'croaking'? | 15:51 |
thingee | anteaya: ah that's right. yeah it's my fault. Someone told me they were already disabled, but I kept seeing them reporting so I'm requesting again for them to be disabled | 15:52 |
*** bswartz1 has joined #openstack-infra | 15:52 | |
thingee | anteaya: thank you! | 15:52 |
odyssey4me | it seems to have come back to life :) | 15:52 |
*** mpaolino has quit IRC | 15:52 | |
*** agentle_ has joined #openstack-infra | 15:52 | |
anteaya | we had a small hiccup at the nova mid-cycle too, but is likely due to the gerrit db cache jeblair performed | 15:52 |
jeblair | if you mean "was unresponsive within the last 3 minutes", yes, it's because we're doing a tiny bit of maintenance | 15:52 |
odyssey4me | jeblair I tried to refresh and it gave a server error. | 15:52 |
*** achanda has quit IRC | 15:52 | |
jeblair | if it's something more than that, it's news to us | 15:52 |
odyssey4me | ah, makes sense - fair enough :) | 15:52 |
anteaya | thingee: glad we have reconnected again | 15:53 |
thingee | thorst: no reason to talk privately. IBM XIV has been failing for sometime. | 15:53 |
thingee | thorst: reps from IBM are unsure why. Until then, I don't want things reporting until that's figured out. | 15:53 |
anteaya | thingee: I suggested he talk to you in -cinder, he just got up for break here at nova mid-cycle | 15:53 |
anteaya | he will be back in a minute | 15:53 |
*** bswartz has quit IRC | 15:54 | |
thingee | thorst: I'm also unsure why people are leaving something running to give false reports. | 15:54 |
anteaya | he told me he will try on his end to impress upon the operators of this ci the responsiblity that is theirs | 15:54 |
*** shashank_hegde has joined #openstack-infra | 15:54 | |
*** mgershen has joined #openstack-infra | 15:54 | |
jeblair | anteaya: 12493 and 13394 are now inactive | 15:54 |
anteaya | thingee: thorst seems to get it | 15:54 |
anteaya | jeblair: thank you! | 15:54 |
* anteaya goes to approve ml posts | 15:54 | |
jeblair | now we seem to be over quota in dfw, looking into that | 15:55 |
thingee | anteaya: I heard that from Jay Bryant, Alon, etc...it has been weeks now | 15:55 |
*** patrickeast has joined #openstack-infra | 15:55 | |
anteaya | thingee: yup, got it | 15:55 |
*** achanda has joined #openstack-infra | 15:55 | |
*** romcheg has quit IRC | 15:55 | |
anteaya | thingee: you have way more patience than I do, that is for sure | 15:56 |
*** coolsvap|away is now known as coolsvap | 15:56 | |
jeblair | it's because we have 50 'DELETED' instances in rax | 15:57 |
nibalizer | morning yall | 15:57 |
*** hasharConfcall is now known as hashar | 15:57 | |
anteaya | jeblair: fun | 15:57 |
anteaya | nibalizer: morning | 15:57 |
jeblair | i'll open a ticket | 15:57 |
*** tonytan4ever has quit IRC | 15:58 | |
nibalizer | jeblair: I'd like to approve some more puppet-httpd changes, if you're up for being infra-root-on-hand | 15:58 |
*** jamesmcarthur has quit IRC | 15:58 | |
*** MarkAtwood has joined #openstack-infra | 15:58 | |
*** achanda has quit IRC | 15:58 | |
jeblair | nibalizer: sure thing, but only do ones that have existing verified +1s, the zuul backlog is huge and it'll probably be late today before we see any check results return | 15:59 |
pabelanger | nibalizer: I like your moxie | 15:59 |
*** SergK has quit IRC | 15:59 | |
nibalizer | sure, thing | 15:59 |
thorst | thingee: sorry - stepped away for a bit. Didn't see this. Didn't mean pinging privately to be odd. I'm just trying to track down the owner to get awareness and the owner. | 16:00 |
openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for neutron ipv6 ping failure bug 1477192 https://review.openstack.org/204652 | 16:00 |
uvirtbot | Launchpad bug 1477192 in neutron "neutron test_multi_prefix_slaac failing in the gate with ping failures starting around 7/22" [High,Confirmed] https://launchpad.net/bugs/1477192 | 16:00 |
openstack | bug 1477192 in neutron "neutron test_multi_prefix_slaac failing in the gate with ping failures starting around 7/22" [High,Confirmed] https://launchpad.net/bugs/1477192 | 16:00 |
uvirtbot | Launchpad bug 1477192 in neutron "neutron test_multi_prefix_slaac failing in the gate with ping failures starting around 7/22" [High,Confirmed] | 16:00 |
thingee | thorst: ok | 16:00 |
*** jlanoux has quit IRC | 16:00 | |
*** deepakcs has joined #openstack-infra | 16:00 | |
thorst | sorry about that! | 16:00 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul: Specify gearman server listen address with separate option https://review.openstack.org/204682 | 16:01 |
pabelanger | pleia2: when you have some time, do you mind +2 on review 179208? That's to merge our grafana.o.o node | 16:01 |
nibalizer | pabelanger: I think https://review.openstack.org/#/c/202618/ is irrelevant since http://git.openstack.org/cgit/openstack-infra/puppet-dashboard/tree/README.markdown#n3 | 16:02 |
*** SergK has joined #openstack-infra | 16:03 | |
*** agentle_ has quit IRC | 16:03 | |
*** mpaolino has joined #openstack-infra | 16:04 | |
*** yamamoto has quit IRC | 16:06 | |
*** yfried has quit IRC | 16:07 | |
*** patrickeast has quit IRC | 16:07 | |
*** patrickeast has joined #openstack-infra | 16:07 | |
*** ivar-lazzaro has joined #openstack-infra | 16:07 | |
*** wznoinsk has quit IRC | 16:07 | |
*** erikmwilson_ has joined #openstack-infra | 16:07 | |
nibalizer | pabelanger: on https://review.openstack.org/#/c/200204/5/manifests/site.pp the other changes seem to be direct switchout not a check/switch behaviour, why is that one special? | 16:07 |
nibalizer | maybe that class is applied to a node along with other classes with apache resources? | 16:08 |
jeblair | okay, rax ticket opened about the 'DELETED' servers | 16:09 |
*** geoffarnold has joined #openstack-infra | 16:11 | |
pabelanger | nibalizer: for 200204 because we have a circular dependency on system-config. | 16:11 |
pabelanger | once merged, we can remove the system-config include apache then update this | 16:12 |
*** gyee has joined #openstack-infra | 16:12 | |
*** tangchen_ has quit IRC | 16:12 | |
pabelanger | nibalizer: as for puppet-dashboard, we should move it into attic, if we are not adding commits | 16:12 |
pabelanger | I think you have something upstream | 16:12 |
*** mwagner_lap has joined #openstack-infra | 16:13 | |
*** geoffarnold has quit IRC | 16:13 | |
nibalizer | http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/eavesdrop.pp#n18 | 16:14 |
nibalizer | yep right there | 16:14 |
*** mjturek1 has quit IRC | 16:14 | |
jeblair | pabelanger, nibalizer: except instead of moving it into the attic, we should merge a change to the readme saying the project is no longer maintained and change the status to read-only in gerrit | 16:14 |
*** fhubik is now known as fhubik_afk | 16:15 | |
hashar | zaro: sure. Will give it a shot tomorrow :-) | 16:16 |
pabelanger | I am fine with that | 16:16 |
*** arxcruz has quit IRC | 16:16 | |
hashar | zaro: gotta figure out some Zuul layout madness on Wikimedia setup first :D | 16:16 |
*** geoffarnold has joined #openstack-infra | 16:16 | |
*** wznoinsk has joined #openstack-infra | 16:16 | |
nibalizer | jeblair: readme change has already happened http://git.openstack.org/cgit/openstack-infra/puppet-dashboard/tree/README.markdown#n3 | 16:17 |
*** harlowja_at_home has quit IRC | 16:17 | |
nibalizer | ill work on the readonly script | 16:18 |
*** changbl has quit IRC | 16:18 | |
jeblair | nibalizer: oh good. if that's not enough, we can merge a change that removes all the content and leave _only_ the readme with that paragraph. | 16:18 |
jeblair | i'm not sure we've established a preference for whether to do that or not. | 16:18 |
openstackgerrit | Spencer Krum proposed openstack-infra/project-config: Puppet-dashboard is read only now https://review.openstack.org/204685 | 16:19 |
*** k4n0 has joined #openstack-infra | 16:19 | |
* fungi is glad to see the ci crazyness has been resolved successfully | 16:19 | |
*** jistr has quit IRC | 16:19 | |
*** ivar-lazzaro has quit IRC | 16:19 | |
nibalizer | jeblair: 204685 is the readonly patch | 16:20 |
*** warpc__ has joined #openstack-infra | 16:20 | |
*** ivar-lazzaro has joined #openstack-infra | 16:20 | |
*** Guest3787 has quit IRC | 16:20 | |
jeblair | so many 500 errors from hpcloud | 16:20 |
jeblair | i'm running the port cleanup script, just in case, but i ran it yesterday too | 16:21 |
pabelanger | removing all the content and leaving readme, gets my vote. That pretty much means project is dead IMO | 16:21 |
*** fhubik_afk is now known as fhubik | 16:21 | |
*** jcoufal has quit IRC | 16:21 | |
jeblair | fungi: have a preference for something like http://git.openstack.org/cgit/openstack-infra/puppet-dashboard/tree/README.markdown#n3 or the stronger "rm *" and leave only a 1 sentence readme? | 16:22 |
*** esmute_ has joined #openstack-infra | 16:23 | |
*** achanda has joined #openstack-infra | 16:24 | |
*** dprince has joined #openstack-infra | 16:24 | |
jeblair | i'm looking for leaked fips on hpcloud too | 16:24 |
*** ivar-lazzaro has quit IRC | 16:24 | |
*** unicell has quit IRC | 16:24 | |
*** alexsyip has joined #openstack-infra | 16:25 | |
fungi | yeah, content removal ftw. it's not like someone can't git checkout at earlier state of the frozen repository | 16:25 |
*** mrmartin has joined #openstack-infra | 16:25 | |
*** signed8bit is now known as signed8bit_ZZZzz | 16:25 | |
jeblair | nibalizer: ^ that work for you? | 16:25 |
*** amitgandhinz has quit IRC | 16:25 | |
*** kiran-r has quit IRC | 16:26 | |
jeblair | nibalizer: (might be good to propose content-removal then have the read-only change depends-on that since i'm sure we won't be merging either of them today) | 16:26 |
fungi | the only downside i can think of is if someone is continually deploying the branch tip, then we break them... but that also has the benefit of bringing its end-of-life state to their attention | 16:26 |
jeblair | fungi: yeah, i think i would be both annoyed and appreciative if i were on the receiving end of that. :) | 16:27 |
*** signed8bit_ZZZzz is now known as signed8bit | 16:27 | |
*** pal has quit IRC | 16:28 | |
jeblair | deleting what i hope are leaked floating ips now | 16:28 |
*** glauco_ has joined #openstack-infra | 16:30 | |
anteaya | jeblair: yay | 16:31 |
*** ihrachyshka has joined #openstack-infra | 16:31 | |
openstackgerrit | greghaynes proposed openstack/diskimage-builder: Perform a booting test for our images https://review.openstack.org/204639 | 16:31 |
*** jsavak has quit IRC | 16:31 | |
*** changbl has joined #openstack-infra | 16:32 | |
sc68cal | lifeless: http://paste.openstack.org/show/400488/ | 16:33 |
sc68cal | lifeless: the cookiecutter thing we just chatted about | 16:33 |
*** jsavak has joined #openstack-infra | 16:33 | |
*** achanda has quit IRC | 16:34 | |
*** DericHorn-HP has joined #openstack-infra | 16:34 | |
*** k4n0 has quit IRC | 16:34 | |
*** DericHorn-HP has quit IRC | 16:35 | |
lifeless | sc68cal: so yeah, old mock | 16:35 |
lifeless | sc68cal: now to find the actual minimum version | 16:35 |
*** jungleboyj has joined #openstack-infra | 16:35 | |
*** dannywilson has joined #openstack-infra | 16:36 | |
lifeless | sc68cal: six 1.6 includes wraps | 16:36 |
*** reed_ has joined #openstack-infra | 16:36 | |
*** reed_ has quit IRC | 16:38 | |
lifeless | sc68cal: try patching the mock.py to have a try:/ImportError: import six; print(six.__file__) | 16:38 |
*** coolsvap is now known as coolsvap|away | 16:38 | |
*** kiran-r has joined #openstack-infra | 16:38 | |
*** sergmelikyan has joined #openstack-infra | 16:38 | |
*** pblaho has quit IRC | 16:38 | |
*** laughterwym has quit IRC | 16:39 | |
andrey-mp | Hi all. Core, please review this https://review.openstack.org/#/c/202070/3 This review is adding one more experimental job, because we need it to move futher with our functional testing in stackforge/ec2-api project | 16:39 |
*** eharney_ has joined #openstack-infra | 16:39 | |
jungleboyj | anteaya: Word on the street is that you are looking for me. | 16:39 |
*** eharney has quit IRC | 16:39 | |
*** eharney_ is now known as eharney | 16:40 | |
*** eharney has quit IRC | 16:40 | |
*** eharney has joined #openstack-infra | 16:40 | |
anteaya | jungleboyj: less so than getting the ibm folks to take care of teh currently disabled ci | 16:40 |
*** vnod has joined #openstack-infra | 16:41 | |
anteaya | jungleboyj: if you are able to do that, thingee would be happy | 16:41 |
jeblair | jenkins07 is shut down; deleting nodes attached to it | 16:41 |
anteaya | jeblair: ack | 16:41 |
jungleboyj | anteaya: Yes, I apologize for that. I have been pinging on them for the last week. Escalated the yesterday. | 16:41 |
anteaya | jungleboyj: understood | 16:41 |
jungleboyj | anteaya: Actually longer than that. | 16:41 |
anteaya | jungleboyj: general tiredness with ibm lack of listening to you here | 16:41 |
jungleboyj | anteaya: It is being actively worked at the moment. | 16:41 |
anteaya | jungleboyj: I know you get it, ibm should listen to you better | 16:41 |
*** yamahata has joined #openstack-infra | 16:41 | |
*** hashar has quit IRC | 16:42 | |
anteaya | jungleboyj: I hope they start listening to you | 16:42 |
jungleboyj | anteaya: :-) Yeah. Working on that. | 16:42 |
anteaya | jungleboyj: you carry a good message | 16:42 |
anteaya | jungleboyj: let me know if I can help | 16:42 |
openstackgerrit | Merged openstack-infra/project-config: VPNaaS: Experimental grenade job https://review.openstack.org/204114 | 16:42 |
openstackgerrit | Merged openstack-infra/project-config: VPNaaS: Experimental grenade job https://review.openstack.org/204114 | 16:42 |
mescanef | hi, please review changes in project-config : https://review.openstack.org/#/c/203582/ and https://review.openstack.org/#/c/204057/ | 16:42 |
mescanef | SergeyLukjanov: ^^ :-) | 16:42 |
mescanef | or maybe fungi | 16:43 |
mescanef | :-) | 16:43 |
*** sergmelikyan has quit IRC | 16:43 | |
jeblair | the floating ip delete in hpcloud is complete | 16:43 |
nibalizer | jeblair: fungi see conversation here: https://review.openstack.org/#/c/189591/ I'm fine with total removal | 16:43 |
jungleboyj | anteaya: Ok, just was surprised when Andrew started pinging me about it. I wasn't aware that it had been escalated so much in the community. | 16:44 |
anteaya | jeblair: yay | 16:44 |
anteaya | jungleboyj: he just happens to be here at the nova mid-cycle | 16:44 |
*** ivar-lazzaro has joined #openstack-infra | 16:44 | |
anteaya | if thorst is andrew | 16:44 |
anteaya | we were talking last night about ci operators taking reponsibility for their work | 16:44 |
anteaya | it was a heartening exchange for me | 16:44 |
anteaya | when I saw an ibm ci was about to be disabled, I asked him if he knew anything about it | 16:45 |
anteaya | since he is sitting beside me | 16:45 |
thorst | thorst is andrew ;-) | 16:45 |
anteaya | :) | 16:45 |
jeblair | nibalizer: *nod* so it's worth a conversation about that. but i bet we can come up with some readme text that would make jhesketh happy -- maybe somethnig like "hey we don't maintain this, but if you want to pick it up, feel free to branch it from the previous commit" or something. | 16:45 |
*** ihrachyshka has quit IRC | 16:45 | |
jungleboyj | Oh, I was going to mention you thorst but wasn't sure. :-) | 16:46 |
anteaya | jungleboyj: :) | 16:46 |
*** mpaolino has quit IRC | 16:46 | |
jungleboyj | anteaya: Gotcha. I need to come over and say hi since you are in town. | 16:46 |
*** sdake has joined #openstack-infra | 16:47 | |
anteaya | jungleboyj: you are in rochester? yes please | 16:47 |
*** ivar-lazzaro has quit IRC | 16:47 | |
jungleboyj | Yep ... That is home base for me. | 16:47 |
anteaya | good | 16:47 |
*** ivar-lazzaro has joined #openstack-infra | 16:47 | |
anteaya | jungleboyj: dinner tonight or tomorrow? | 16:47 |
*** sdake_ has joined #openstack-infra | 16:48 | |
*** bharath_ has joined #openstack-infra | 16:48 | |
*** MarkAtwood has quit IRC | 16:48 | |
jungleboyj | anteaya: Not sure I will be able to do dinner this week. Have family commitments in the evenings. | 16:49 |
anteaya | jungleboyj: fair enough, time for a hug would be great though | 16:49 |
jungleboyj | anteaya: Of course. Will stop over there today. | 16:50 |
*** maishsk has joined #openstack-infra | 16:50 | |
anteaya | jungleboyj: grand | 16:50 |
openstackgerrit | Merged openstack-infra/project-config: Remove non-voting gate jobs https://review.openstack.org/203410 | 16:50 |
*** jasondotstar has quit IRC | 16:50 | |
anteaya | jungleboyj: https://wiki.openstack.org/wiki/Sprints/NovaLibertySprint#IBM_Specifics | 16:50 |
*** dahoo has joined #openstack-infra | 16:50 | |
*** dahoo is now known as jerryz | 16:50 | |
*** MarkAtwood has joined #openstack-infra | 16:51 | |
jeblair | jenkins07 appears to be operating normally now | 16:51 |
anteaya | ya | 16:52 |
anteaya | y! | 16:52 |
*** sdake has quit IRC | 16:52 | |
*** kjnelson has joined #openstack-infra | 16:52 | |
*** erikmwilson_ has quit IRC | 16:53 | |
*** kiran-r has quit IRC | 16:54 | |
*** spredzy|afk is now known as spredzy | 16:54 | |
jeblair | hpcloud-b1 and hpcloud-b2 nodepool managers are stuck | 16:55 |
*** jsavak has quit IRC | 16:56 | |
*** pal has joined #openstack-infra | 16:56 | |
*** fhubik has quit IRC | 16:57 | |
*** sputnik13 has joined #openstack-infra | 16:57 | |
jeblair | stuck on an ssl read in requests/urllib3 | 16:57 |
*** unicell has joined #openstack-infra | 16:58 | |
anteaya | there was a new release of urllib3 yesterday | 16:58 |
jeblair | http://paste.openstack.org/show/400649/ | 16:58 |
jeblair | it's the urllib3 bundled in requests 2.7.0 that's being used here | 16:59 |
nibalizer | crinkle: on https://review.openstack.org/#/c/203868/2 i think we decided the default posix provider was sufficient ya? | 16:59 |
jeblair | i'm going to resatart nodepool | 16:59 |
nibalizer | I keep looking at that change and reading about selinux and trying the commands out | 17:00 |
clarkb | I thought stuck on ssl was why we put the api timeouts in place | 17:00 |
nibalizer | i'm becomming pretty confident that the change is good | 17:00 |
jeblair | hrm, let me see where we are on dib builds | 17:00 |
*** madhu_ak has joined #openstack-infra | 17:00 | |
*** soren has joined #openstack-infra | 17:00 | |
clarkb | are they set for those clouds? | 17:00 |
nibalizer | and ofc using an actual selinux module instead of execs would be better, but one thing at a time | 17:00 |
crinkle | nibalizer: i think we both tested it and it seems sufficient, but after reading the documentation i don't know /why/ it's sufficient | 17:00 |
jeblair | clarkb: i'm pretty sure they are, i'll double check | 17:00 |
crinkle | nibalizer: so i'm a bit nervous about it | 17:01 |
openstackgerrit | litong01 proposed openstack-infra/project-config: Add new monitor repository for OpenStack monitoring service https://review.openstack.org/203259 | 17:01 |
jeblair | api-timeout: 60 | 17:01 |
jeblair | clarkb: ^ | 17:01 |
*** deepakcs has quit IRC | 17:01 | |
*** Ala has quit IRC | 17:01 | |
clarkb | ok so possible the client/requests/urllib3 isnt respectibg that | 17:01 |
*** jsavak has joined #openstack-infra | 17:01 | |
jbernard | mriedem: has multiple cinder backends in gate been discussed already? | 17:02 |
jeblair | clarkb: i know we've seen it do so in the past | 17:02 |
nibalizer | crinkle: im more worried that I don't really know anything about selinux network controls so that change could be bad without me knowing why | 17:02 |
*** zz_dimtruck is now known as dimtruck | 17:02 | |
jeblair | clarkb: like when hpcloud was taking forever to respond to api calls | 17:02 |
jeblair | clarkb: so i suspect this is an edge condition it doesn't handle :/ | 17:02 |
jbernard | mriedem: i would really like to add coverage for the new vol migration changes | 17:02 |
jeblair | clarkb: (maybe connection dropped in middle of tls packet or something) | 17:02 |
anteaya | jbernard: if you post in -nova more folks will see your words | 17:03 |
jbernard | anteaya: thanks, will do | 17:03 |
clarkb | jeblair gotcha | 17:03 |
*** mrmartin has quit IRC | 17:03 | |
openstackgerrit | Victor Ryzhenkin proposed openstack-infra/project-config: Add opportunity to install muranoclient from change https://review.openstack.org/204159 | 17:03 |
*** derekh has quit IRC | 17:04 | |
crinkle | nibalizer: i tested it on an hpcloud centos 6 vm, would be easy enough to reproduce my tests | 17:04 |
openstackgerrit | Victor Ryzhenkin proposed openstack-infra/project-config: Make job gate-murano*-requirements use job template https://review.openstack.org/203177 | 17:04 |
jeblair | i suspect we are approximately 1 hour from dib image builds being complete | 17:04 |
openstackgerrit | Merged openstack-infra/puppet-bandersnatch: Migrate to puppet-httpd module https://review.openstack.org/202609 | 17:04 |
jeblair | i think i'd rather have the extra 200 instances today, so i'm going to restart nodepool | 17:04 |
*** andrey-mp has quit IRC | 17:04 | |
*** tsg has quit IRC | 17:04 | |
*** soren has quit IRC | 17:04 | |
jeblair | stopping nodepool when it's running dib doesn't work like you expect | 17:06 |
clarkb | jeblair go for it we can retrigger builds if necessary | 17:06 |
clarkb | jeblair it should, the builds are killed | 17:07 |
jeblair | root@nodepool:~# /etc/init.d/nodepool restart | 17:07 |
jeblair | * Restarting Nodepool nodepool .... waiting ............................................................................... | 17:07 |
clarkb | well the build it only does one ata time | 17:07 |
clarkb | oh restart doesnt work at all iirc. hashar was working on it | 17:07 |
jeblair | ooooh | 17:07 |
crinkle | nibalizer: i'm going to test it again and make sure it does the right thing when behind_proxy is true/false | 17:07 |
nibalizer | ok | 17:08 |
jeblair | i wanted to restart to minimize zmq downtime (since we leak completed nodes during a restart) | 17:08 |
mriedem | jbernard: ? | 17:08 |
jeblair | anyway, stop/started now | 17:08 |
*** _nadya_ has joined #openstack-infra | 17:09 | |
*** gsagie_ has quit IRC | 17:09 | |
jbernard | mriedem: just curious if there as an existing effort (or discussion) on multipe backends for cinder in the gate | 17:09 |
jbernard | mriedem: if i had it, i could add test coverage for volume migration | 17:10 |
*** Krinkle_ is now known as Krinkle | 17:10 | |
jeblair | so now we should just be waiting for rax to delete the 83 deleted servers | 17:10 |
clarkb | jeblair https://review.openstack.org/#/c/203232/ | 17:10 |
clarkb | thats hashars change | 17:11 |
*** thorst has quit IRC | 17:11 | |
*** shardy_ has joined #openstack-infra | 17:11 | |
*** vnod has quit IRC | 17:12 | |
*** puranamr has joined #openstack-infra | 17:12 | |
*** jasondotstar has joined #openstack-infra | 17:13 | |
*** shardy has quit IRC | 17:13 | |
*** puranamr has quit IRC | 17:13 | |
nibalizer | http://puppetboard.openstack.org/report/git02.openstack.org/d0ca097885d666b64ee26ce9a44f7456563af2c1 my crystal ball says that is caused by us losing our symlink from `which pip` to /usr/local/bin/python-pip | 17:13 |
nibalizer | a few other git servers are affected | 17:13 |
clarkb | nibalizer where did the symlink disappear to? | 17:14 |
*** Hal has joined #openstack-infra | 17:14 | |
*** Hal is now known as Guest57966 | 17:14 | |
*** marzif has joined #openstack-infra | 17:15 | |
jeblair | nibalizer: 06-08 are newer; is that part of the pattern? | 17:15 |
*** puranamr has joined #openstack-infra | 17:15 | |
nibalizer | clarkb: I'm guessing thats the issue, can you hop on that server and confirm | 17:15 |
nibalizer | if memory serves we create that symlink in install_puppet.sh but don't ensure it in puppet itself | 17:15 |
nibalizer | at laest thats a problem gozer had once | 17:15 |
*** shardy_ has quit IRC | 17:16 | |
jeblair | [corvus@git02 ~]$ ls -ld `which pip` | 17:17 |
jeblair | -rwxr-xr-x. 1 root root 281 Jul 1 09:51 /usr/bin/pip | 17:17 |
nibalizer | right and puppet, on redhat, tries to use pip-python and fails when it doesn't find it | 17:17 |
*** shardy has joined #openstack-infra | 17:17 | |
nibalizer | which is fine if you're using system packages, but we use get-pip | 17:17 |
clarkb | nibalizer I cannot. not on machine with keys currebtly | 17:17 |
openstackgerrit | David Cheperdak proposed openstack-infra/project-config: Add OpenStack project Fusion. https://review.openstack.org/204434 | 17:18 |
clarkb | nbalizer right did we change or remove the symlink install? | 17:18 |
nibalizer | clarkb: maybe the newer git machines didn't have install_puppet.sh run on them? | 17:18 |
jeblair | all the git servers are like that | 17:19 |
openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Ensure a symlink for pip-python https://review.openstack.org/204697 | 17:20 |
jeblair | the new ones were created normally so should have had install_puppet run | 17:20 |
*** MarkAtwood has quit IRC | 17:20 | |
jeblair | oooh | 17:21 |
nibalizer | jeblair: `which pip-python` on git02? | 17:21 |
jeblair | /usr/bin/pip-python exists on 1,6,7,8, but not 2,3,4,5 | 17:21 |
jeblair | i'm unaware of a pattern that explains that | 17:21 |
fungi | nibalizer: sounds like puppet needs fixing somewhere? red hat stopped using pip-python (after a brief stint of calling it python-pip too) | 17:22 |
*** auxian has quit IRC | 17:23 | |
openstackgerrit | Merged openstack-infra/puppet-openstackid: Migrate to puppet-httpd module https://review.openstack.org/201737 | 17:23 |
openstackgerrit | Merged openstack-infra/puppet-mailman: Migrate to puppet-httpd module https://review.openstack.org/201743 | 17:23 |
fungi | nibalizer: now the rpm only provides pip, pip2 and pip2.6 | 17:23 |
openstackgerrit | Merged openstack-infra/project-config: Remove non-voting pypy gate jobs https://review.openstack.org/203411 | 17:23 |
*** BharatK has quit IRC | 17:23 | |
nibalizer | fungi: ya I know where the fix to puppet-core would go | 17:23 |
nibalizer | I'm supposed to be in there fixing their 'upgrade' code as well | 17:23 |
*** e0ne has quit IRC | 17:24 | |
fungi | nibalizer: you can see by running `rpm -ql python-pip-7.1.0-1.el6.noarch` on an updated centos 6.x system | 17:24 |
jeblair | the node graph looks like we're doing something now. | 17:24 |
* nibalizer -> afk for a bit | 17:25 | |
*** abregman has quit IRC | 17:25 | |
*** marzif has quit IRC | 17:25 | |
*** gmurphy has joined #openstack-infra | 17:26 | |
*** afazekas_ has quit IRC | 17:26 | |
*** marzif has joined #openstack-infra | 17:26 | |
openstackgerrit | Merged openstack-infra/project-config: remove check-gerrit-buck https://review.openstack.org/202727 | 17:26 |
fungi | nibalizer: mainly just wondering how that module works at all on current centos 6 | 17:26 |
openstackgerrit | Merged openstack-infra/puppet-drupal: Migrate to puppet-httpd module https://review.openstack.org/201747 | 17:26 |
openstackgerrit | Merged openstack-infra/puppet-diskimage_builder: Add Gemfile and puppet 4 checks https://review.openstack.org/202661 | 17:26 |
fungi | or whether everyone who uses it puts in a similar workaround | 17:27 |
*** bswartz1 is now known as bswartz | 17:27 | |
gmurphy | this seems like a bit of a glitch visiting - https://tarballs.openstack.org/ presents an invalid certificate, if i choose to proceed i hit the security.openstack.org page. should i create a bug for this or is it a known issue? | 17:27 |
*** dannywilson has quit IRC | 17:28 | |
clarkb | gmurphy I dont think there is an https vhost for that name | 17:30 |
fungi | gmurphy: they're hosted on the same server | 17:30 |
fungi | gmurphy: security.o.o is the only https site on it | 17:30 |
gmurphy | oh right. that makes sense. | 17:30 |
fungi | gmurphy: there is no https for tarballs.o.o | 17:30 |
clarkb | so bug with our apache setup? | 17:30 |
gmurphy | think so. | 17:31 |
*** marzif has quit IRC | 17:31 | |
openstackgerrit | Oleksii Zamiatin proposed openstack-infra/project-config: Make zeromq functional gate check non-voting https://review.openstack.org/202095 | 17:31 |
fungi | is it? i always just assumed that's how hosting multiple sites on one server works. you have a *:443 site and then *:80 sites with name-based virtual hosting | 17:31 |
clarkb | fungi apache can distinguish and should probably return an error? | 17:32 |
*** marzif has joined #openstack-infra | 17:32 | |
fungi | can't host multiple https sites on the same ip address and port number without jumping through extra hoops | 17:32 |
fungi | clarkb: well, yes and no. apache can't know that until after it's presented the cert to the browser | 17:32 |
clarkb | oh right | 17:32 |
fungi | clarkb: ssl negotiation happens first, then the browser tells the server the hostname it's looking for | 17:32 |
jeblair | well, sni is mostly a thing in browsers now | 17:32 |
gmurphy | oh fair enough. it's been way too long since i've messed with apache config to figure this one out. leave it with you guys. | 17:33 |
*** tjones1 has joined #openstack-infra | 17:33 | |
*** ildikov has quit IRC | 17:33 | |
jeblair | so we probably could do better | 17:33 |
fungi | maybe. i think sni would require us to add all the vhost names for static.o.o's apache as subject altnames for that cert? | 17:33 |
*** Sukhdev has quit IRC | 17:33 | |
fungi | otherwise visiting any of them with https:// would lead to an invalid cert error in the browser | 17:34 |
*** koolhead17 has joined #openstack-infra | 17:34 | |
openstackgerrit | greghaynes proposed openstack/diskimage-builder: Perform a booting test for our images https://review.openstack.org/204639 | 17:35 |
jeblair | fungi: no, sni lets apache pick different certs based on the handshake | 17:35 |
greghaynes | fungi: sni? | 17:35 |
*** madhu_ak has quit IRC | 17:35 | |
greghaynes | yea | 17:35 |
fungi | so, yes. we could do better by either a) getting ssl cert coverage for all sites on static.o.o, or b) move security.o.o to a separate server | 17:35 |
fungi | unless i'm misunderstanding | 17:35 |
greghaynes | Unless you need to support letting people from nintendo wii's hit your website | 17:36 |
fungi | okay, so not subjectaltname in a common cert, but multiple certs for them still | 17:36 |
jeblair | fungi: i am not certain whether it would let us c) return some kind of error condition for non-security.o.o vhosts | 17:36 |
fungi | what i'm saying is i don't think sni solves the "invalid cert for https://tarballs.openstack.org/" problem unless we actually provide ssl cert coverage for that vhost name | 17:37 |
jeblair | fungi: unless c is somehow possible, i agree. | 17:37 |
jeblair | i should amend (c) to say "some kind of non-confusing error condition" | 17:38 |
jeblair | since it's already returning a confusing one :) | 17:38 |
fungi | i've witnessed plenty of issues in the past where proxies relying on sni which neglect to add the right vhost name for the thing they've cnamed to it, and you get an invalid cert error in the browser | 17:38 |
openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add a section to the readme about removing bug signatures https://review.openstack.org/204707 | 17:39 |
fungi | so at least the browsers i use provide basically the same error regardless of whether the cert is for one hostname which isn't the thing you've requested, or the server isn't providing certs for the host you've requested | 17:39 |
jeblair | oh good, rax deleted the 83 deleted servers | 17:40 |
jeblair | so i think we're at max capacity now | 17:40 |
*** thorst has joined #openstack-infra | 17:40 | |
*** changbl has quit IRC | 17:41 | |
*** ericksonsantos has quit IRC | 17:42 | |
*** esmute_ has quit IRC | 17:43 | |
*** davideagnello has joined #openstack-infra | 17:43 | |
*** ajmiller has quit IRC | 17:44 | |
*** davideagnello has quit IRC | 17:45 | |
*** pelix has quit IRC | 17:45 | |
nibalizer | fungi: https://github.com/puppetlabs/puppet/blob/3.x/lib/puppet/provider/package/pip.rb#L38 | 17:47 |
*** timrc_ has joined #openstack-infra | 17:47 | |
nibalizer | fungi: so I would expect latest centos 6.x and latest puppet 3.x do not have a functional pip provider | 17:48 |
*** sergmelikyan has joined #openstack-infra | 17:48 | |
*** maishsk has quit IRC | 17:48 | |
*** maishsk has joined #openstack-infra | 17:49 | |
*** davidjc has quit IRC | 17:49 | |
*** kjnelson has quit IRC | 17:49 | |
*** pvaneck has joined #openstack-infra | 17:50 | |
*** heyongli has quit IRC | 17:52 | |
*** esmute_ has joined #openstack-infra | 17:52 | |
*** changbl has joined #openstack-infra | 17:52 | |
*** heyongli has joined #openstack-infra | 17:52 | |
*** amickus has joined #openstack-infra | 17:53 | |
*** ajmiller has joined #openstack-infra | 17:55 | |
*** yfried has joined #openstack-infra | 17:55 | |
*** pal has quit IRC | 17:55 | |
SpamapS | jeblair: so, if we merge https://review.openstack.org/201666 .. I can rework the storyboard a bit and instead of making a single node ironic puppet manifest, I can start making OpenStack controllers and compute node manifests... | 17:55 |
SpamapS | jeblair: also we likely want a bifrost manifest instead of the ironic puppet manifest. | 17:56 |
SpamapS | which is just o_O .. "Yo dawg, I heard you like puppet so I wrote some puppet to deploy our ansible that deploys your puppet" | 17:56 |
*** esmute_ has quit IRC | 17:57 | |
*** dimtruck is now known as zz_dimtruck | 17:57 | |
*** jasondotstar has quit IRC | 17:57 | |
openstackgerrit | Boris Pavlovic proposed openstack-infra/project-config: Add non voting fuel agent python jobs https://review.openstack.org/204711 | 17:58 |
*** ajmiller has quit IRC | 17:58 | |
*** sergmelikyan has quit IRC | 17:59 | |
*** baoli has quit IRC | 17:59 | |
jeblair | SpamapS: yeah, we told EmilienM|off that and it made him go on vacation | 18:00 |
SpamapS | jeblair: doh! ;) | 18:01 |
SpamapS | but he's french, they go on vacation if you tell them there is a place that has vacations. | 18:01 |
*** baoli_ has joined #openstack-infra | 18:01 | |
*** kjnelson has joined #openstack-infra | 18:01 | |
*** sergmelikyan has joined #openstack-infra | 18:01 | |
* SpamapS is jealous | 18:01 | |
*** thorst has quit IRC | 18:02 | |
*** marzif has quit IRC | 18:02 | |
*** bhunter71 has joined #openstack-infra | 18:02 | |
*** larainema has quit IRC | 18:02 | |
jeblair | SpamapS: i _think_ that change says that ansible can talk directly to ironic to launch our cloud? | 18:03 |
*** ericksonsantos has joined #openstack-infra | 18:03 | |
openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Set docimpact group of os-ansible-deployment https://review.openstack.org/204712 | 18:03 |
*** Kennan has joined #openstack-infra | 18:04 | |
*** Kennan2 has quit IRC | 18:04 | |
*** zz_dimtruck is now known as dimtruck | 18:04 | |
*** marzif has joined #openstack-infra | 18:04 | |
*** esmute_ has joined #openstack-infra | 18:04 | |
harlowja | hi guys, for anvil (stackforge package building, blah blah); the gate it has is currently testing venv building (on ubuntu); which is great, but not always sufficent, it also builds rpms and was wondering if its possible to get a custom script that gets activated on a rhel infra-vm instance (anvil needs rpmbuild installed, yum-build-dep...) so that the rpm building can also be a gate test... (vs right now its been somewhat manual | 18:04 |
harlowja | pita verification...) | 18:04 |
*** davidjc has joined #openstack-infra | 18:05 | |
bhunter71 | a question; on my centos6 builders, I seem to get pip 1.5.4 in my py26 virtualenvs. That is, when I activate and pip --version I see 1.5.4. The system has pip 7.1 installed globally. Is this normal, and if not, how do I get a recent pip into my centos6 builder? thanks! | 18:05 |
harlowja | anvil can setup these things, but it needs sudo access (that likely requires something special ?) | 18:06 |
*** Sukhdev has joined #openstack-infra | 18:06 | |
*** krtaylor has quit IRC | 18:06 | |
fungi | harlowja: the job just needs to not invoke the revoke-pip builder macro | 18:07 |
*** changbl has quit IRC | 18:07 | |
harlowja | hmmm | 18:07 |
jeblair | SpamapS: sorry, let me rephrase that as a real question: once puppet has configured bifrost+ironic on the bare metal node, and we have an ansible inventory describing how to turn other bare metal instances into our cloud (by having ansible run puppet on them), ansible will talk to bifrost's ironic to launch those nodes, right? | 18:07 |
*** mgershen has quit IRC | 18:07 | |
jeblair | fungi, harlowja: 'revoke-sudo' | 18:07 |
*** ajmiller has joined #openstack-infra | 18:07 | |
*** marzif_ has joined #openstack-infra | 18:07 | |
harlowja | kk, let me see if i can scrap something together that might allow this to work | 18:08 |
fungi | er, yes, that's what i meant ;) | 18:08 |
jeblair | harlowja: and yeah, there are centos and fedora nodes available | 18:08 |
harlowja | * http://paste.openstack.org/show/400732/ (the stuff that gets boostraped) | 18:08 |
fungi | bhunter71: yes, the version of pip in a virtualenv is determined by the version of virtualenv you installed | 18:08 |
fungi | virtualenv provides its own pip, it doesn't use the pip you have installed on your system | 18:09 |
SpamapS | jeblair: correct. | 18:09 |
bhunter71 | fungi: ok. Is my virtualenv likely stale? This isn't something I've knowingly modified in my environment. Images are up to date | 18:09 |
jeblair | SpamapS: okay cool. anyone else we should get to weigh in on that? greghaynes? | 18:09 |
jeblair | clarkb, fungi: you may be interested in https://review.openstack.org/201666 | 18:10 |
fungi | bhunter71: if you have pip 1.5.4 in your virtualenv then that implies you've got virtualenv 11.something installed i think | 18:10 |
SpamapS | jeblair: We skip the part where we put keystone, nova, glance, and neutron in place, and rely on the API being available only on 127.0.0.1. | 18:10 |
SpamapS | Which I actually don't love | 18:10 |
*** changbl has joined #openstack-infra | 18:10 | |
*** marzif has quit IRC | 18:11 | |
*** Sukhdev has quit IRC | 18:11 | |
bhunter71 | fungi: yes, virtulenv --version == 1.11.4 | 18:11 |
jeblair | SpamapS: oh, interesting... so we could not run ansible->bifrost from puppetmaster? | 18:11 |
*** alex_xu has quit IRC | 18:11 | |
jeblair | SpamapS: we'd have to run that ansible locally | 18:11 |
SpamapS | jeblair: we could, but any node that could talk to ironic would have full control over everything on that node. | 18:12 |
harlowja | jeblair fungi does that include centos6 and centos7 envs? | 18:12 |
*** mwagner_lap has quit IRC | 18:12 | |
harlowja | anvil should work with both, and having both would be sweet | 18:12 |
SpamapS | jeblair: so I'm thinking we keep it 127.0.0.1 and maybe even put in place iptables owner rules | 18:12 |
jeblair | SpamapS: oh, okay, so we could configure iptables to let puppetmaster ansible talk to bifrost? | 18:12 |
SpamapS | jeblair: right we could also just poke a hole for the puppetmaster to do it. | 18:13 |
SpamapS | jeblair: but bifrost is not being developed this way. It's entirely theoretical that these playbooks work on !localhost | 18:13 |
*** puranamr has quit IRC | 18:13 | |
*** alex_xu has joined #openstack-infra | 18:13 | |
SpamapS | TheJulia: ^ confirm that? | 18:14 |
SpamapS | NobodyCam: or you? ^ | 18:14 |
*** woodster_ has joined #openstack-infra | 18:14 | |
fungi | harlowja: yes we have both centos 6 and 7 workers | 18:14 |
jeblair | SpamapS: okay, worst case is i guess we have a playbook we run on puppetmaster that runs a playbook on bifrost-host, or tell ansible bifrost-host is an ssh jump host or something. | 18:15 |
harlowja | fungi nice, let's see if i can figure out the project-config magic for this | 18:15 |
fungi | harlowja: though we're likely to drop centos 6 within the year | 18:15 |
harlowja | k | 18:15 |
harlowja | fair enough | 18:15 |
greghaynes | jeblair: The only reasoning I heard for doing a full bm cloud rather than standalone ironic was so our ansible launch node future code could talk to nova | 18:15 |
jeblair | greghaynes: yeah, i think mordred is satisfied that similar functionality can happen with launch-node talking to ironic | 18:16 |
NobodyCam | SpamapS: oh I have not tested such a thing, but I think the playbooks could easily be adapted to work from !localhost | 18:17 |
SpamapS | Indeed, the launching part of bifrost is actually really simple. | 18:17 |
SpamapS | NobodyCam: yeah, this is one of the advantages of using Ansible. :) | 18:17 |
*** amitgandhinz has joined #openstack-infra | 18:17 | |
NobodyCam | TheJulia: do you know (off the top of your head) if there is something that would block ^^^^^^^^^ | 18:18 |
SpamapS | I'm reading the playbooks | 18:18 |
SpamapS | I see nothing | 18:18 |
NobodyCam | ya | 18:18 |
SpamapS | just have to change localhost to a variable or a group name. | 18:18 |
TheJulia | on the phone atm, will read/context switch in a minute | 18:18 |
greghaynes | SGTM, I think its preferrable to do the conservative thing in terms of effort for what were trying to do right now, and if later on it doesnt work well for launch-node we can reevaluate then | 18:19 |
*** _nadya_ has quit IRC | 18:19 | |
SpamapS | and move the 'connection: local' into the inventory. | 18:19 |
NobodyCam | TheJulia: ack TY | 18:19 |
*** marzif_ has quit IRC | 18:19 | |
SpamapS | yeah | 18:19 |
*** puranamr has joined #openstack-infra | 18:19 | |
*** _nadya_ has joined #openstack-infra | 18:19 | |
SpamapS | it's probably worth noting that this is going to jump start the cloud deployment. I can probably have all the working nodes deployed in a couple hours. | 18:19 |
greghaynes | famous last words | 18:20 |
SpamapS | note the qualification, "working nodes" | 18:20 |
*** soren has joined #openstack-infra | 18:20 | |
*** changbl has quit IRC | 18:20 | |
SpamapS | Oh! Let's talk about hostnames. Does infra have a schema that I need to adhere to? | 18:21 |
* SpamapS suspects this is covered on ci.openstack.org | 18:21 | |
*** timrc_ has quit IRC | 18:22 | |
*** bharath_ has quit IRC | 18:22 | |
jeblair | SpamapS: i don't think we've written anything about it... what do you have in mind? | 18:22 |
*** tonytan4ever has joined #openstack-infra | 18:23 | |
jeblair | SpamapS: i suspect we'd probably want to do something like X.<name of cloud instance>.openstack.org | 18:23 |
TheJulia | SpamapS: NobodyCam brain still occupied, wrapping up shortly | 18:23 |
*** _nadya_ has quit IRC | 18:24 | |
ttx | jeblair: quick question while i think about it... could we technically separate openstack and stackforge/experimental things wherever it gets presented as a catalog (think github or cgit) | 18:24 |
SpamapS | I was actually just going to pick an incrementing offset. baremetal###.infracloud0.openstack.org or something like that. | 18:24 |
jeblair | ttx: no | 18:24 |
*** thorst has joined #openstack-infra | 18:24 | |
ttx | I'm trying to think of ways to reduce confusion as to what is an official thing and what is not (or not yet) | 18:24 |
jeblair | SpamapS: that sounds good to me | 18:24 |
SpamapS | jeblair: just need to name all the nodes. | 18:24 |
*** mwagner_lap has joined #openstack-infra | 18:24 | |
jeblair | SpamapS: we could shorten that a bit too if you want | 18:25 |
clarkb | maybe controllerXX and computeXX or woll they be more fluid than that? | 18:25 |
ttx | jeblair: does the prefix we are using in Gerrit have to be the same in github ? | 18:25 |
clarkb | barrmetalXX doesnt say much beyond what you already know | 18:25 |
ttx | jeblair: and is there a way to split things between a cgit.o.o and a other-cgit.o.o | 18:26 |
SpamapS | clarkb: they're all identical except the two with access to the ilos, so it will be somewhat arbitrary how we choose. | 18:26 |
jeblair | ttx: yes prefixes must be the same | 18:26 |
*** agentle_ has joined #openstack-infra | 18:26 | |
ttx | jeblair: ok | 18:26 |
SpamapS | clarkb: I'm fine with setting the roles up front like that. | 18:26 |
ttx | so apatr from "forgetting to sync some" there is no real option on the guthub side | 18:27 |
jeblair | ttx: and the cgit split may be possible but with tremendous difficulty i think | 18:27 |
*** sergmelikyan has quit IRC | 18:27 | |
clarkb | SpamapS we can always update it later too | 18:27 |
clarkb | so baremetalXX wfm | 18:27 |
jeblair | ttx: yeah, the github org moves are a big part of the driver here | 18:27 |
jeblair | ttx: (they are actually less automatable than gerrit) | 18:28 |
ttx | jeblair: I mean, technically, we could only sync "openstack" repos | 18:28 |
SpamapS | clarkb: I guarantee we'll re-deploy these nodes at _least_ once since they have network cards we don't like and RAID w/o battery backed write cache at the moment. | 18:28 |
jeblair | ttx: to attempt to put your mind at ease, i actually think it's been a very long time since someone may have attempted to use github.com/openstack as a catalog | 18:28 |
ttx | and let stackforge/experimental things do whatever they want | 18:28 |
*** sergmelikyan has joined #openstack-infra | 18:28 | |
jeblair | there's already just way too much there and it's unavigable | 18:28 |
*** davideagnello has joined #openstack-infra | 18:28 | |
*** sergmelikyan has quit IRC | 18:29 | |
ttx | jeblair: yeah. ok, thanks for the answer | 18:29 |
*** yamamoto has joined #openstack-infra | 18:31 | |
*** Sukhdev has joined #openstack-infra | 18:31 | |
TheJulia | SpamapS: so enroll/deploy such playbooks can be run remotely, the module must execute locally, or concievably on a remote host where the moudle and dependencies (shade) are located | 18:31 |
jeblair | SpamapS: so i'd either go with your proposal, or bm00.ic0.openstack.org if you want to type less. :) | 18:32 |
* greghaynes nitpicks and points out that we should have at least 3 numerical digits | 18:32 | |
jeblair | greghaynes: very good point | 18:33 |
*** changbl has joined #openstack-infra | 18:33 | |
clarkb | hundreds of machines? no way | 18:33 |
greghaynes | cue dr. evil pinky | 18:33 |
clarkb | mwahahaha | 18:33 |
*** HeOS has quit IRC | 18:33 | |
jeblair | how many digits should we have for clouds? :) | 18:34 |
greghaynes | 10 clouds ought to be enough for anyone ;) | 18:35 |
*** MarkAtwood has joined #openstack-infra | 18:35 | |
*** yamamoto has quit IRC | 18:36 | |
* jesusaurus 's clouds go to 11 | 18:36 | |
*** ayoung has quit IRC | 18:36 | |
jeblair | wow, hpcloud leaks floating ips like a sieve | 18:36 |
*** jasondotstar has joined #openstack-infra | 18:36 | |
anteaya | :( | 18:37 |
*** ivar-lazzaro has quit IRC | 18:38 | |
asselin | how do the leaks occur? I have some too locally.... | 18:38 |
jeblair | and now hpcloud-b2 and -b3 have stopped working, likely also due to the requests problem | 18:38 |
*** Hal has joined #openstack-infra | 18:39 | |
*** Hal is now known as Guest75053 | 18:39 | |
patrickeast | when you say leaks, what does that mean? i’m seeing issues with my 3rd party ci where nodepool is not releasing floating ip’s when its done with single use nodes, is that what you guys are seeing? | 18:40 |
jeblair | sigmavirus24: hey, we're regularly seeing threads get stuck here, despite having a 60 second timeout: http://paste.openstack.org/show/400649/ | 18:40 |
*** achanda has joined #openstack-infra | 18:40 | |
jeblair | sigmavirus24: do you have any suggestions about how to avoid that? | 18:41 |
*** ZZelle_ has joined #openstack-infra | 18:41 | |
jeblair | patrickeast: well, not sure whether it's nodepool or openstack at fault, but yes | 18:41 |
sigmavirus24 | jeblair: so timeouts are not overall timeouts | 18:41 |
sigmavirus24 | or what's commonly referred to as a wall timeout | 18:41 |
clarkb | I think openstack | 18:41 |
*** krtaylor has joined #openstack-infra | 18:42 | |
sigmavirus24 | They're applied to connection and read. So if it's hanging that's because data is trickling back to the client | 18:42 |
*** sergmelikyan has joined #openstack-infra | 18:42 | |
sigmavirus24 | In other words, your timeout is long enough that just a little data comes back and resets the timeout | 18:42 |
clarkb | we say delete fip, get positive response without delete then delete the node attached to it which removes our way of knowing about the fip | 18:42 |
*** warpc__ has quit IRC | 18:42 | |
asselin | I wrote a script to work around the leaking floating-ips: https://git.openstack.org/cgit/stackforge/third-party-ci-tools/tree/nodepool/clean_floating_ip.sh | 18:42 |
*** dims_ has quit IRC | 18:42 | |
clarkb | since in nodepool its all node oriented and we dont mamage pools of fips too | 18:43 |
patrickeast | gotcha | 18:43 |
jeblair | sigmavirus24: it's been an hour, so i'm almost certain no data has been received (this is an openstack api call) | 18:43 |
sigmavirus24 | jeblair: https://github.com/pnpnpn/timeout-decorator seems to be a good wall timeout solution | 18:43 |
sigmavirus24 | jeblair: well the socket thinks there's data | 18:43 |
sigmavirus24 | *thinks it is receiving data | 18:43 |
patrickeast | asselin: do you just run that as a cron job? | 18:43 |
jeblair | sigmavirus24: are you sure it isn't the ssl-wrapper waiting for a complete packet? | 18:44 |
asselin | patrickeast, yes | 18:44 |
sigmavirus24 | jeblair: it could be that too | 18:44 |
sigmavirus24 | We set the timeouts on the socket object itself | 18:44 |
*** e0ne has joined #openstack-infra | 18:45 | |
asselin | patrickeast, I've been meaning to see if nodepool can do that, but didn't get a chance. And seems perhaps it's not really nodepool's issue.... | 18:46 |
clarkb | it would be nice if A) deleting an fip actually deleted it and B) deleting a node deleted the unneded fip (at least as an option) | 18:47 |
*** bharath_ has joined #openstack-infra | 18:47 | |
*** dims_ has joined #openstack-infra | 18:47 | |
openstackgerrit | David Cheperdak proposed openstack-infra/project-config: Add OpenStack project Fusion. https://review.openstack.org/204434 | 18:47 |
asselin | or c) creating a node used an unallocated floating-ip | 18:47 |
clarkb | but iirc the second thing isnt atomic yet | 18:47 |
clarkb | so you can get them out of sync and leak regardless | 18:48 |
fungi | right, given that you can't attach metadata to a fip and it doesn't have a distinct uuid (afaik) and a fip can reappear later because it's only a limited pool of available addresses anyway, i don't think there's much hope of nodepool being able to detect that a fip it asked to have deleted wasn't actually deleted even though the api claims it would do so | 18:49 |
nibalizer | asselin: can you vote/+a https://review.openstack.org/#/c/184905/21 and https://review.openstack.org/#/c/198463/ ? | 18:49 |
jeblair | sigmavirus24: i think that solution will be problematic -- we have about 10 threads that are pretty much continuously making api calls; multi-threaded means we have to use the multiprocessing variant of that, and it looks like it will end up spawning a process per call... | 18:49 |
asselin | nibalizer, looking | 18:50 |
jeblair | sigmavirus24: we could rework nodepool to use multiprocessing internally for its managers, and then use the signal strategy in each of those (relatively long-lived) processes | 18:50 |
openstackgerrit | gordon chung proposed openstack/requirements: block oslo.messaging 1.17.x due to memory leak https://review.openstack.org/204727 | 18:50 |
jeblair | sigmavirus24: but that will make database interaction very complicated | 18:51 |
sigmavirus24 | jeblair: yeah this is one of those things that causes some headaches for people (not having a wall timeout) | 18:51 |
jeblair | that's sort of at the 'rewrite nodepool level of complexity' | 18:51 |
jeblair | sigmavirus24: i think this is why we were barking up the tcp keepalive tree a while a go | 18:51 |
jeblair | sigmavirus24: because we're pretty sure that everyone thinks this socket is dead except our program | 18:52 |
sigmavirus24 | jeblair: so if you want TCP keep-alive you can use https://toolbelt.readthedocs.org/en/latest/adapters.html#tcpkeepaliveadapter | 18:52 |
asselin | nibalizer, I don't know enough about ruby to approve those | 18:52 |
jeblair | sigmavirus24: was this where we got as far as "we need support in the openstack clients" to use it? | 18:52 |
jeblair | clarkb, fungi: ^? | 18:53 |
sigmavirus24 | jeblair: so keystoneclient's session should set it up by default iirc | 18:53 |
sigmavirus24 | Both novaclient sans keystoneclient and keystoneclient should be auto-using tcp keep-alive | 18:53 |
fungi | i vaguely remember that clarkb might remember this more accurately than i do ;) | 18:54 |
sigmavirus24 | jeblair: https://git.openstack.org/cgit/openstack/python-keystoneclient/tree/keystoneclient/session.py#n901 | 18:54 |
jeblair | oh interesting; so i should see if we're running a too-old novaclient. and perhaps when we fully switch to shade, since shade is using keystone sessions, we'll get it there too | 18:54 |
clarkb | at the time I dont think the clients supported it | 18:54 |
clarkb | so this would be new | 18:54 |
sigmavirus24 | If novaclient uses a keystoneclient session, it supports it. If it doesn't use a ksc session, it supports it | 18:54 |
sigmavirus24 | This happened before the summit (these changes) | 18:55 |
crinkle | nibalizer: i'm more confident in my selinux puppet-cgit change now, i'm wondering if it would be easy to add a centos6 beaker job just for that module so we can all be as confident? | 18:55 |
*** krtaylor has quit IRC | 18:55 | |
clarkb | yes and when we looked at it was bwfore the summit | 18:55 |
clarkb | since after summit has been lolbusy for me | 18:55 |
sigmavirus24 | https://git.openstack.org/cgit/openstack/python-novaclient/tree/novaclient/client.py#n61 | 18:55 |
sigmavirus24 | clarkb: I know what you mean | 18:55 |
*** bswartz has quit IRC | 18:56 | |
nibalizer | crinkle: im hoping we can land the openstackci stuff before we start copying it out to the swarm | 18:56 |
sigmavirus24 | Ah, we're not setting keepidle in those adapters | 18:56 |
nibalizer | avoid 30-odd patches because of some trivial bug/nitpick in the spec_helper_acceptance.rb in openstackci | 18:56 |
sigmavirus24 | That could be what's causing problems | 18:56 |
crinkle | nibalizer: makes sense | 18:56 |
jeblair | sigmavirus24: we are running a version of novaclient with that | 18:56 |
*** glauco__ has joined #openstack-infra | 18:57 | |
anteaya | paste has an inbound spike: http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=14 | 18:57 |
*** baoli_ has quit IRC | 18:57 | |
*** baoli has joined #openstack-infra | 18:59 | |
sigmavirus24 | I wrote the toolbelt implementation of the TCPKeepAliveAdapter a month or two after doing it for novaclient/keystoneclient | 18:59 |
sigmavirus24 | I wonder if we should just depend on that instead of copying it into two clients | 18:59 |
nibalizer | asselin: I'll ask for some reviews from the openstack/puppet-* folks and see if we can get some more qualified eyes on it that way | 18:59 |
*** sherif has quit IRC | 19:00 | |
SpamapS | sigmavirus24: very seldom is "copying it into ..." the most optimal long term solution to code issues. | 19:01 |
crinkle | nibalizer: asselin if it helps we have nearly the same file in all the openstack/puppet-* modules | 19:01 |
sigmavirus24 | SpamapS: relying on the toolbelt just for one adapter seems a bit silly though | 19:01 |
*** davideagnello has quit IRC | 19:01 | |
sigmavirus24 | I mean there are lots of other goodies in there | 19:01 |
* asselin looks | 19:01 | |
sigmavirus24 | But I've never seen openstack need streaming multipart/form-data uploads | 19:01 |
SpamapS | sigmavirus24: could it be a standalone improvement that toolbelt relies on? | 19:01 |
SpamapS | or rather, standalone library | 19:02 |
*** glauco__ has quit IRC | 19:02 | |
sigmavirus24 | So we've actually been consolidating recipes in the toolbelt that we had previously written extemporaneously into issue comments in the toolbelt with tests | 19:02 |
sigmavirus24 | The idea was to keep everything in one place | 19:03 |
*** glauco__ has joined #openstack-infra | 19:03 | |
*** Krinkle is now known as Krinkle_ | 19:03 | |
sigmavirus24 | I wrote the code so doing a separate package isn't a problem in reality, I just don't want to maintain two packages instead of one | 19:03 |
sigmavirus24 | #lazy | 19:03 |
jeblair | sigmavirus24: i haven't found a definitive answer, but from my reading so far, i do strongly suspect that keepidle is required in order for the others to have an effect | 19:04 |
sigmavirus24 | jeblair: yeah, the other fun thing is that depending on the OS, some options may not even be available | 19:05 |
sigmavirus24 | So the code we have in the toolbelt does "the right thing" by checking for them before trying to set them | 19:05 |
jeblair | sigmavirus24: or it may be using a very high default value, like 2 hours | 19:05 |
sigmavirus24 | jeblair: yeah the keep-alive documentation is pretty much non-existent last I checked | 19:05 |
anteaya | the paste inbound spike seems to be decreasing | 19:06 |
openstackgerrit | greghaynes proposed openstack/diskimage-builder: Perform a booting test for our images https://review.openstack.org/204639 | 19:06 |
*** maishsk_ has joined #openstack-infra | 19:06 | |
*** maishsk has quit IRC | 19:06 | |
*** maishsk_ is now known as maishsk | 19:06 | |
jeblair | cat /proc/sys/net/ipv4/tcp_keepalive_time | 19:06 |
jeblair | 7200 | 19:06 |
anteaya | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=152&rra_id=all | 19:06 |
sigmavirus24 | woah | 19:06 |
clarkb | thats default iirc | 19:07 |
jeblair | sigmavirus24: so yeah, that looks like the linux system-wide default | 19:07 |
*** glauc____ has joined #openstack-infra | 19:07 | |
*** glauco_ has quit IRC | 19:07 | |
jeblair | sigmavirus24: if it would be useful, i could leave nodepool running as-is for another 40 minutes and we may be able to confirm that's what's happening (if it times out at 19:43 UTC) | 19:07 |
*** glauco_ has joined #openstack-infra | 19:08 | |
*** glauco__ has quit IRC | 19:08 | |
*** bharath__ has joined #openstack-infra | 19:09 | |
jeblair | (let me see if i can dig in the logs for an earlier instance) | 19:09 |
sigmavirus24 | jeblair: just for confirmation, that'd be good. in the meantime I'll write up patches for ksc and novaclient. Novaclient can lean on ksc's adapter now that the requirements should be sufficient going forward | 19:09 |
*** shardy_ has joined #openstack-infra | 19:10 | |
*** koolhead17 has quit IRC | 19:10 | |
*** andersonvom has quit IRC | 19:10 | |
*** shardy has quit IRC | 19:11 | |
openstackgerrit | Merged openstack/diskimage-builder: Use official mirror name for debian-minimal https://review.openstack.org/200345 | 19:12 |
*** changbl has quit IRC | 19:12 | |
asselin | nibalizer, done | 19:12 |
*** changbl has joined #openstack-infra | 19:12 | |
*** bharath_ has quit IRC | 19:13 | |
*** Krinkle_ is now known as Krinkle | 19:13 | |
*** shardy_ has quit IRC | 19:15 | |
*** kjnelson has quit IRC | 19:16 | |
*** shardy has joined #openstack-infra | 19:16 | |
jeblair | i can't prove it with the logs, so we'll just wait it out | 19:16 |
asselin | btw, nova - juno doesn't delete the fip when the instance is deleted. They're left in the pool, but nodepool/nova doesn't re-use them: http://paste.openstack.org/show/400961/ | 19:19 |
*** esmute_ has quit IRC | 19:19 | |
*** afazekas has joined #openstack-infra | 19:19 | |
clarkb | asselin yes nodepool does an explicit delete first iirc | 19:19 |
clarkb | so its delete fip && delete node | 19:19 |
asselin | clarkb, why does it need to delte the fip? why not leave it in the pool and get it from there? | 19:20 |
clarkb | asselin because it doesnt pool fips | 19:21 |
clarkb | so doesnt have a way to travk them beyond node attachment | 19:21 |
clarkb | it could pool them but that would require a bit of rework in nodepool | 19:21 |
*** crc32 has joined #openstack-infra | 19:22 | |
ZZelle_ | mordred, hi | 19:22 |
asselin | clarkb, but nova pools them, nodepool just needs to associate one of them http://paste.openstack.org/show/400962/ | 19:23 |
patrickeast | asselin: i was just looking at this to try and figure out why i’m not getting any back (like 0% success rate releasing them) it frees them here https://github.com/openstack-infra/nodepool/blob/master/nodepool/provider_manager.py#L605 | 19:23 |
clarkb | asselin but you have to know which arent used or bwing deleted | 19:23 |
clarkb | and you have to know when to get more and free unneeded IPs | 19:23 |
clarkb | its not as simple as just use IP | 19:24 |
asselin | i see, devil in the details | 19:24 |
openstackgerrit | Joshua Harlow proposed openstack-infra/project-config: Adjust anvil gate/check jobs to test venv and rpm building https://review.openstack.org/204734 | 19:24 |
*** amickus has quit IRC | 19:24 | |
clarkb | (and this is why nwutron should just give you working networks) | 19:24 |
harlowja | fungi jeblair ^ if u get a chance, think i did that right, ha | 19:24 |
*** sergmelikyan has quit IRC | 19:25 | |
*** puranamr has quit IRC | 19:27 | |
*** puranamr has joined #openstack-infra | 19:28 | |
*** MarkAtwood has quit IRC | 19:29 | |
*** ivar-lazzaro has joined #openstack-infra | 19:29 | |
*** esmute_ has joined #openstack-infra | 19:30 | |
*** mriedem has quit IRC | 19:32 | |
sigmavirus24 | jeblair: I'm not sure if there's nodepool project on launchpad but you might want to mark https://bugs.launchpad.net/python-novaclient/+bug/1477275 as affecting it too | 19:33 |
openstack | Launchpad bug 1477275 in python-novaclient "KSC's TCPKeepAliveAdapter should specify keepidle to prevent connections from hanging for hours" [Undecided,New] - Assigned to Ian Cordasco (icordasc) | 19:33 |
uvirtbot | Launchpad bug 1477275 in python-novaclient "KSC's TCPKeepAliveAdapter should specify keepidle to prevent connections from hanging for hours" [Undecided,New] | 19:33 |
uvirtbot | Launchpad bug 1477275 in python-novaclient "KSC's TCPKeepAliveAdapter should specify keepidle to prevent connections from hanging for hours" [Undecided,New] https://launchpad.net/bugs/1477275 | 19:33 |
*** jsavak has quit IRC | 19:34 | |
jeblair | sigmavirus24: it's on storyboard, so can't make that link. thanks | 19:34 |
*** esmute_ has quit IRC | 19:34 | |
*** jsavak has joined #openstack-infra | 19:35 | |
*** jasondotstar has quit IRC | 19:36 | |
*** mriedem has joined #openstack-infra | 19:36 | |
harlowja | jeblair fungi in https://review.openstack.org/#/c/204734/1/jenkins/jobs/anvil-jobs.yaml does that shell script get executed in the directory of the checked out git code, or is there some other magic needed to do that? | 19:37 |
harlowja | that might be a change needed if so | 19:37 |
*** sergmelikyan has joined #openstack-infra | 19:37 | |
openstackgerrit | Boris Pavlovic proposed openstack-infra/project-config: Add non voting fuel agent python jobs https://review.openstack.org/204711 | 19:38 |
*** sergmelikyan has quit IRC | 19:39 | |
sigmavirus24 | jeblair: thoughts on a reasonable keepalive? | 19:40 |
sigmavirus24 | *keepidle | 19:40 |
mordred | clarkb: https://github.com/p8952/bocker docker written in 100 lines of bash | 19:40 |
*** sergmelikyan has joined #openstack-infra | 19:41 | |
*** AJaeger has joined #openstack-infra | 19:42 | |
AJaeger | could I get some reviews for https://review.openstack.org/204712 , please? openstack-manuals just got the third DocImpact bug today wrongly assigned by openstack-ansible, so let's fix their config... | 19:43 |
*** ayoung has joined #openstack-infra | 19:43 | |
SpamapS | Oh nice, our brain dead mellanox firmware fix is in Ironic master already. :-D | 19:43 |
SpamapS | TheJulia: nice work | 19:43 |
jeblair | sigmavirus24: can you set it based on the supplied timeout value? perhaps something like idle=timeout-(count*interval) (with some suitable range bounding) ? | 19:44 |
*** heyongli has quit IRC | 19:45 | |
sigmavirus24 | jeblair: not quite. urllib3 didn't implement them per-request, they're per-PoolManager | 19:45 |
*** AJaeger has quit IRC | 19:45 | |
*** heyongli has joined #openstack-infra | 19:47 | |
*** kushal has quit IRC | 19:47 | |
*** radez is now known as radez_g0n3 | 19:47 | |
*** kjnelson has joined #openstack-infra | 19:47 | |
jeblair | sigmavirus24: makes sense; maybe idle=60, interval=15, count=4, to give it up to 2 minutes? that seems reasonable to me for a cloud. | 19:48 |
sigmavirus24 | jeblair: sounds good | 19:49 |
openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for oslo.db KeyError bug 1476708 https://review.openstack.org/204123 | 19:49 |
uvirtbot | Launchpad bug 1476708 in oslo.db "KeyError: 'pid' during db pool connection" [High,In progress] https://launchpad.net/bugs/1476708 | 19:49 |
openstack | bug 1476708 in oslo.db "KeyError: 'pid' during db pool connection" [High,In progress] https://launchpad.net/bugs/1476708 - Assigned to Roman Podoliaka (rpodolyaka) | 19:49 |
uvirtbot | Launchpad bug 1476708 in oslo.db "KeyError: 'pid' during db pool connection" [High,In progress] | 19:49 |
*** HeOS has joined #openstack-infra | 19:50 | |
*** ZZelle_ has quit IRC | 19:51 | |
*** cdent has quit IRC | 19:52 | |
*** sergmelikyan has quit IRC | 19:53 | |
*** sherif has joined #openstack-infra | 19:55 | |
jeblair | sigmavirus24: ah, the default interval and probes are also very high, so we have a total of 131.25 minutes to wait for confirmation, which has just passed | 19:57 |
jeblair | 2015-07-22 17:43:52,898 DEBUG nodepool.ProviderManager: Manager hpcloud-b2 running task <nodepool.provider_manager.ListServersTask object at 0x7f018004f090> (queue: 0) | 19:57 |
jeblair | 2015-07-22 19:55:21,985 DEBUG nodepool.ProviderManager: Manager hpcloud-b2 ran task <nodepool.provider_manager.ListServersTask object at 0x7f018004f090> in 7889.07918096s | 19:57 |
jeblair | sigmavirus24: and right on time ^ | 19:57 |
sigmavirus24 | aha | 19:57 |
sigmavirus24 | Well I'm just writing the commit message for ksc now | 19:57 |
*** rlandy has quit IRC | 19:57 | |
sigmavirus24 | And then I'll send a dependent review to novaclient | 19:57 |
jeblair | sigmavirus24: awesome! and now we have a high degree of confidence that should help with this problem | 19:58 |
*** rlandy has joined #openstack-infra | 19:58 | |
jeblair | sigmavirus24: thank you very much! | 19:58 |
jeblair | clarkb, fungi: ^ | 19:58 |
jeblair | File "/usr/lib/python2.7/ssl.py", line 260, in read | 19:59 |
jeblair | return self._sslobj.read(len) | 19:59 |
jeblair | error: [Errno 110] Connection timed out | 19:59 |
clarkb | jeblair: neat so things got better after 2 hours | 19:59 |
jeblair | sigmavirus24: and that's the tail end of the traceback for the exception that gets thrown (which is fine for us, nodepool catches that moves on) | 20:00 |
jeblair | clarkb: well 2 hours, and 11 and a half minutes :) | 20:00 |
jeblair | (that extra bit almost had me thinking we were wrong) | 20:01 |
*** puranamr has quit IRC | 20:01 | |
sigmavirus24 | This is after all my fault for not doing it right the first time =P | 20:02 |
*** MarkAtwood has joined #openstack-infra | 20:02 | |
jeblair | well, things are definitely better than they used to be -- and this explains why this only happens when we're looking at it (unlike what used to happen, which is it would just get stuck and stop completely forever) | 20:02 |
*** MarkAtwood has quit IRC | 20:04 | |
jeblair | fungi: re presentty: doh. | 20:04 |
*** krtaylor has joined #openstack-infra | 20:04 | |
*** marzif_ has joined #openstack-infra | 20:05 | |
*** mrmartin has joined #openstack-infra | 20:07 | |
*** sdake_ is now known as sdake | 20:07 | |
*** Guest57966 has quit IRC | 20:07 | |
fungi | jeblair: clarkb: sigmavirus24: that makes a lot more sense now | 20:08 |
*** MarkAtwood has joined #openstack-infra | 20:08 | |
*** esmute_ has joined #openstack-infra | 20:08 | |
clarkb | so basically when we tried previously clients didn't support it. Now clients do but currently linux defaults are really terrible so overriding in clients as we speak? | 20:09 |
fungi | jeblair: i'm about to send you a patch for an exit key too. i was just going with q but if you have a preference, i'm happy to accommodate | 20:09 |
fungi | also for a vertical version of the pan transition, what do you think of "scroll" as a keyword? | 20:09 |
*** Guest75053 has quit IRC | 20:10 | |
*** jasondotstar has joined #openstack-infra | 20:12 | |
*** signed8bit is now known as signed8bit_ZZZzz | 20:13 | |
sigmavirus24 | jeblair: fungi clarkb https://review.openstack.org/#/c/204741/ https://review.openstack.org/#/c/204745/ | 20:15 |
*** dannywilson has joined #openstack-infra | 20:16 | |
*** pal has joined #openstack-infra | 20:19 | |
sigmavirus24 | Also if anyone cares to nominate https://bugs.launchpad.net/python-novaclient/+bug/1477275 for backports to the stable/kilo branches of both, I'll be happy to do cherry-picks of those commits to their stable branches so nodepool can use those clients sooner rather than later | 20:20 |
openstack | Launchpad bug 1477275 in python-keystoneclient "KSC's TCPKeepAliveAdapter should specify keepidle to prevent connections from hanging for hours" [Medium,In progress] - Assigned to Ian Cordasco (icordasc) | 20:20 |
*** mrmartin has quit IRC | 20:20 | |
uvirtbot | Launchpad bug 1477275 in python-keystoneclient "KSC's TCPKeepAliveAdapter should specify keepidle to prevent connections from hanging for hours" [Medium,In progress] | 20:20 |
uvirtbot | Launchpad bug 1477275 in python-keystoneclient "KSC's TCPKeepAliveAdapter should specify keepidle to prevent connections from hanging for hours" [Medium,In progress] https://launchpad.net/bugs/1477275 | 20:20 |
sigmavirus24 | (nothing like getting the bug info in triplicate =P) | 20:20 |
*** ZZelle_ has joined #openstack-infra | 20:20 | |
openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Remove nova ephemeral temprevert https://review.openstack.org/204747 | 20:21 |
*** signed8bit_ZZZzz is now known as signed8bit | 20:21 | |
*** mrmartin has joined #openstack-infra | 20:22 | |
*** pal has quit IRC | 20:24 | |
clarkb | we can probably kick uvirtbot | 20:25 |
jeblair | fungi: scroll works (tilt works too, to match pan) | 20:25 |
*** e0ne has quit IRC | 20:26 | |
*** dannywilson has quit IRC | 20:26 | |
fungi | hah. i like tilt | 20:26 |
fungi | potentially confusing, but it's an amusing reference | 20:26 |
anteaya | ha ha ha | 20:26 |
*** dannywilson has joined #openstack-infra | 20:26 | |
*** MarkAtwood has quit IRC | 20:27 | |
* anteaya hears a dop call for the 35mm using wheels on sticks | 20:27 | |
jeblair | i'm having a hard time finding a definitive list of film transitions (which is what i wanted to base those on; that's why it's pan and not wipe (which would have the images stationary)) | 20:28 |
jeblair | i'm thinking of manually setting the kernel defaults on nodepool | 20:28 |
*** peristeri has quit IRC | 20:29 | |
anteaya | jeblair: http://www.springhurst.org/cinemagic/glossary_terms.htm | 20:29 |
fungi | yep, i've switched it to tilt. still tweaking CanvasCombine not to crash on me though | 20:29 |
jeblair | urwid is fun like that | 20:30 |
*** Krinkle is now known as Krinkle_ | 20:30 | |
*** gildub has joined #openstack-infra | 20:30 | |
jeblair | okay, i manually changed the kernel keepalive values; not sure what's required for them to take effect (possibly a nodepool restart to open new sockets with them?) | 20:30 |
*** hashar has joined #openstack-infra | 20:31 | |
*** Sukhdev has quit IRC | 20:31 | |
jeblair | (or maybe the next time they fail it will take 2 hours, but then the time after that, they will take effect) | 20:32 |
*** kjnelson has quit IRC | 20:33 | |
*** MarkAtwood has joined #openstack-infra | 20:33 | |
fungi | jeblair: _probably_ sticky from socket creation, agreed, but worth finding out | 20:33 |
*** eharney has quit IRC | 20:35 | |
*** jasondotstar has quit IRC | 20:38 | |
openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: OIDC - OpenId Connect Implementation https://review.openstack.org/198458 | 20:38 |
*** soren has quit IRC | 20:38 | |
*** baoli has quit IRC | 20:38 | |
*** gildub has quit IRC | 20:40 | |
*** mestery has quit IRC | 20:42 | |
openstackgerrit | Antoine Musso proposed openstack-infra/zuul: Test please ignore https://review.openstack.org/204753 | 20:42 |
*** davideagnello has joined #openstack-infra | 20:43 | |
*** jsavak has quit IRC | 20:43 | |
*** jsavak has joined #openstack-infra | 20:43 | |
hashar | holly hell | 20:43 |
hashar | hello :-} | 20:43 |
clarkb | hashar: but now I can't ignore it :) | 20:44 |
* jeblair hands hashar some smelling salts | 20:44 | |
hashar | I just found out that gerrit review --label code-review=0 | 20:44 |
* anteaya hopes hashar is saying hello to someone who's first name is holly | 20:44 | |
greghaynes | jeblair: the 11min makes sense, I think the tineout is when kernel starts sending keepaluve probes, takes about 11min for the kernel to decide its deas | 20:44 |
jeblair | greghaynes: yep | 20:44 |
anteaya | s/who's/whose | 20:44 |
hashar | does not work on my setup :-D Causes some MySQL related stacktrace due to code-review being a dupe of Code-Review (note the case) | 20:44 |
greghaynes | wow my phone typing is bad | 20:45 |
hashar | but it works on openstack setup. I am wondering whether you guys messed up with the SQL backend database to make it case insensitive | 20:45 |
*** jeblair is now known as CarolChanning | 20:45 | |
CarolChanning | hello holly! | 20:45 |
*** CarolChanning is now known as jeblair | 20:45 | |
anteaya | ha ha ha | 20:46 |
anteaya | I love her voice | 20:47 |
*** jsavak has quit IRC | 20:47 | |
openstackgerrit | Merged openstack-dev/hacking: Updated from global requirements https://review.openstack.org/186829 | 20:48 |
ianw | seems i've once again slept through a crisis :) | 20:49 |
jeblair | ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin | 20:49 |
jeblair | hashar: ^ | 20:49 |
*** puranamr has joined #openstack-infra | 20:50 | |
hashar | jeblair: seems like case sensitive :D | 20:50 |
jeblair | hashar: i'm pretty sure that means we're case-sensitive | 20:50 |
jeblair | yeah | 20:50 |
jeblair | hashar: gerrit version difference? | 20:50 |
hashar | yeah potentially | 20:50 |
hashar | the root cause is definitely gerrit review being changed to use --label ( https://review.openstack.org/#/c/138132/1/zuul/lib/gerrit.py,unified ) | 20:51 |
*** kjnelson has joined #openstack-infra | 20:51 | |
hashar | I changed the layout.yaml file to use Verified: Code-Review: everywhere, but that causes a ton of side effects in zuul which normalizes them with lower() anyway. So approvals (verified) no more matches the submit records (Verified) | 20:52 |
jeblair | ugh | 20:52 |
hashar | which mean I am going to send a lot of patches hehe | 20:52 |
jeblair | hashar: yeah, so have zuul keep the correct case, and then lowercase as needed for comparisons? | 20:53 |
hashar | yup | 20:53 |
jeblair | hashar: or even go as far as to make zuul case-sensitive? | 20:53 |
hashar | but it only lower case the version coming from Gerrit API | 20:53 |
hashar | not the one from layout.yaml | 20:53 |
hashar | so you end up with Verified == verifie | 20:53 |
hashar | I got some details at https://phabricator.wikimedia.org/T106531 if you want | 20:53 |
*** thorst has quit IRC | 20:54 | |
hashar | if label['label'].lower() not in allow_needs <--- where allow_needs has Verified :-) | 20:54 |
bhunter71 | fungi: thanks for your help earlier. There's this thing $desired_virtualenv in system-config that's pretty important for centos6. My value here was out of date. | 20:55 |
*** sergmelikyan has joined #openstack-infra | 20:56 | |
*** agentle_ has quit IRC | 20:57 | |
*** tongli has quit IRC | 20:57 | |
*** tongli has joined #openstack-infra | 20:57 | |
*** thorst has joined #openstack-infra | 20:58 | |
hashar | jeblair: dont worry too much about it. The obvious fix for me is to revert https://review.openstack.org/#/c/138132/1/zuul/lib/gerrit.py,unified and switch back approvals to be lower cases. This way they will matches the lower case normalized categories | 21:00 |
*** erlon has quit IRC | 21:00 | |
openstackgerrit | Merged openstack-infra/irc-meetings: qa-team-meeting: Remove unneeded comment https://review.openstack.org/204636 | 21:00 |
*** austin81 has left #openstack-infra | 21:01 | |
*** davideagnello has quit IRC | 21:03 | |
*** DericHorn-HP has joined #openstack-infra | 21:03 | |
*** sergmelikyan has quit IRC | 21:04 | |
*** aysyd has quit IRC | 21:05 | |
*** sergmelikyan has joined #openstack-infra | 21:06 | |
*** krtaylor has quit IRC | 21:06 | |
*** vlaza1 has joined #openstack-infra | 21:06 | |
*** vlaza has quit IRC | 21:06 | |
*** agentle_ has joined #openstack-infra | 21:08 | |
*** timrc_ has joined #openstack-infra | 21:14 | |
*** ldnunes has quit IRC | 21:15 | |
fungi | bhunter71: right, we update that in our manifest over time. unfortunately the pip provider does unexpected things with ensure=>latest | 21:16 |
*** kgiusti has left #openstack-infra | 21:16 | |
*** david-lyle has quit IRC | 21:17 | |
*** timrc_ has quit IRC | 21:17 | |
*** doug-fish has left #openstack-infra | 21:17 | |
*** jsavak has joined #openstack-infra | 21:17 | |
*** cbader has quit IRC | 21:18 | |
*** vlaza1 has quit IRC | 21:22 | |
*** MarkAtwood has quit IRC | 21:22 | |
*** dkranz has quit IRC | 21:25 | |
*** dustins has quit IRC | 21:25 | |
*** marzif_ has quit IRC | 21:25 | |
*** timrc has quit IRC | 21:26 | |
*** timrc has joined #openstack-infra | 21:26 | |
*** MarkAtwood has joined #openstack-infra | 21:26 | |
*** dan_ has joined #openstack-infra | 21:27 | |
*** dan_ is now known as Guest56855 | 21:27 | |
*** mrmartin has quit IRC | 21:28 | |
*** timrc has quit IRC | 21:29 | |
*** timrc has joined #openstack-infra | 21:30 | |
*** Krinkle_ is now known as Krinkle | 21:30 | |
*** e0ne has joined #openstack-infra | 21:30 | |
*** heyongli has quit IRC | 21:32 | |
*** flepied has quit IRC | 21:33 | |
*** heyongli has joined #openstack-infra | 21:34 | |
*** kjnelson has quit IRC | 21:34 | |
*** tongli has quit IRC | 21:37 | |
*** tongli has joined #openstack-infra | 21:37 | |
*** notnownikki has quit IRC | 21:38 | |
anteaya | one more day of nova mid-cycle left tomorrow, so far it is very productive | 21:38 |
*** Sukhdev has joined #openstack-infra | 21:38 | |
*** tlian has quit IRC | 21:39 | |
*** sergmelikyan has quit IRC | 21:39 | |
*** tonytan4ever has quit IRC | 21:41 | |
*** heyongli has quit IRC | 21:41 | |
*** MarkAtwood has quit IRC | 21:43 | |
*** achanda has quit IRC | 21:43 | |
*** heyongli has joined #openstack-infra | 21:43 | |
*** agentle_ has quit IRC | 21:43 | |
*** tongli has quit IRC | 21:46 | |
*** annegentle has joined #openstack-infra | 21:46 | |
*** tongli has joined #openstack-infra | 21:46 | |
*** jaypipes has quit IRC | 21:46 | |
*** amitgandhinz has quit IRC | 21:47 | |
*** tongli has quit IRC | 21:48 | |
*** mriedem has quit IRC | 21:49 | |
*** timrc has quit IRC | 21:51 | |
*** timrc has joined #openstack-infra | 21:52 | |
*** dizquierdo has joined #openstack-infra | 21:52 | |
*** sergmelikyan has joined #openstack-infra | 21:53 | |
openstackgerrit | litong01 proposed openstack-infra/project-config: Add new monitor repository for OpenStack monitoring service https://review.openstack.org/203259 | 21:53 |
*** dims_ has quit IRC | 21:53 | |
*** markus_z has quit IRC | 21:54 | |
*** signed8bit has quit IRC | 21:54 | |
*** dizquierdo has quit IRC | 21:55 | |
openstackgerrit | greghaynes proposed openstack/diskimage-builder: Perform a booting test for our images https://review.openstack.org/204639 | 21:56 |
*** hashar has quit IRC | 21:57 | |
*** e0ne has quit IRC | 21:58 | |
*** claudiub has quit IRC | 21:58 | |
*** jungleboyj has quit IRC | 22:00 | |
*** patrickeast has quit IRC | 22:00 | |
*** jasondotstar has joined #openstack-infra | 22:01 | |
*** Kennan has quit IRC | 22:02 | |
*** sigmavirus24 is now known as sigmavirus24_awa | 22:03 | |
*** Kennan has joined #openstack-infra | 22:03 | |
*** heyongli has quit IRC | 22:04 | |
*** heyongli has joined #openstack-infra | 22:05 | |
*** gordc has quit IRC | 22:06 | |
timrc | jeblair: Hey so re: https://review.openstack.org/#/c/204682 - So does setting "host = None" tell gear to listen to localhost? | 22:07 |
*** sergmelikyan has quit IRC | 22:07 | |
*** bknudson has quit IRC | 22:08 | |
*** bnemec has quit IRC | 22:09 | |
*** thorst has quit IRC | 22:10 | |
*** sergmelikyan has joined #openstack-infra | 22:10 | |
openstackgerrit | litong01 proposed openstack-infra/project-config: Add new kiloeyes repository for OpenStack monitoring service https://review.openstack.org/203259 | 22:11 |
*** jsavak has quit IRC | 22:12 | |
*** earlephilhower has quit IRC | 22:13 | |
openstackgerrit | Merged openstack-infra/release-tools: Prototype script to release from the new YAML files https://review.openstack.org/202240 | 22:15 |
openstackgerrit | Merged openstack-infra/release-tools: translate glance-store to glance_store https://review.openstack.org/202789 | 22:15 |
openstackgerrit | Merged openstack-infra/release-tools: Move release_notes.py into releasetools package https://review.openstack.org/202257 | 22:15 |
*** tvanderwiel has quit IRC | 22:15 | |
*** achanda has joined #openstack-infra | 22:15 | |
openstackgerrit | Merged openstack-infra/release-tools: move list-repos into the releasetools project https://review.openstack.org/202320 | 22:15 |
openstackgerrit | litong01 proposed openstack-infra/project-config: Add new kiloeyes repository for OpenStack monitoring service https://review.openstack.org/203259 | 22:16 |
openstackgerrit | Merged openstack-infra/release-tools: move commands into their own package https://review.openstack.org/202321 | 22:16 |
greghaynes | timrc: None makes it default to whatever getaddrinfo() returns | 22:16 |
greghaynes | timrc: so, should be something localhost-like | 22:16 |
*** jasondotstar has quit IRC | 22:18 | |
*** heyongli has quit IRC | 22:19 | |
*** glauc____ has quit IRC | 22:21 | |
openstackgerrit | Kyle Partridge proposed openstack-infra/jenkins-job-builder: Add "commit-status-context" to the build option supporting the most recent version of GHPRB https://review.openstack.org/204779 | 22:22 |
openstackgerrit | greghaynes proposed openstack/diskimage-builder: Perform a booting test for our images https://review.openstack.org/204639 | 22:24 |
*** glauco_ has quit IRC | 22:24 | |
*** heyongli has joined #openstack-infra | 22:24 | |
*** jasondotstar has joined #openstack-infra | 22:26 | |
*** dprince has quit IRC | 22:31 | |
*** rlandy has quit IRC | 22:31 | |
*** jasondotstar has quit IRC | 22:33 | |
*** rossella_s has quit IRC | 22:37 | |
*** rossella_s has joined #openstack-infra | 22:37 | |
jeblair | should be something like 'all addresses' | 22:38 |
jeblair | timrc, greghaynes: ^ | 22:38 |
openstackgerrit | greghaynes proposed openstack/diskimage-builder: Only warn about missing map files with debug https://review.openstack.org/204783 | 22:39 |
*** ashleighfarnham has quit IRC | 22:43 | |
*** sergmelikyan has quit IRC | 22:43 | |
*** DericHorn-HP has quit IRC | 22:45 | |
*** sergmelikyan has joined #openstack-infra | 22:51 | |
*** DericHorn-HP has joined #openstack-infra | 22:51 | |
*** DericHorn-HP has quit IRC | 22:53 | |
*** prad has quit IRC | 22:53 | |
openstackgerrit | greghaynes proposed openstack/diskimage-builder: Perform a booting test for our images https://review.openstack.org/204639 | 22:53 |
jhesketh | Morning | 22:54 |
*** ivar-lazzaro has quit IRC | 22:55 | |
*** dimtruck is now known as zz_dimtruck | 22:59 | |
pvaneck | Hi, can this review https://review.openstack.org/#/c/198910/ be approved? It has two +2s and mainly just adds a .gitreview file to a new puppet project | 23:03 |
*** krtaylor has joined #openstack-infra | 23:04 | |
*** ddieterly has joined #openstack-infra | 23:07 | |
*** camunoz_gone is now known as camunoz | 23:08 | |
*** sergmelikyan has quit IRC | 23:08 | |
*** davideagnello has joined #openstack-infra | 23:09 | |
jeblair | pvaneck: done | 23:11 |
*** larainema has joined #openstack-infra | 23:11 | |
pvaneck | jeblair: Thanks :) | 23:12 |
*** shardy_ has joined #openstack-infra | 23:12 | |
*** shardy has quit IRC | 23:14 | |
*** Krinkle has left #openstack-infra | 23:14 | |
*** shardy_ has quit IRC | 23:17 | |
*** shardy has joined #openstack-infra | 23:18 | |
*** jungleboyj has joined #openstack-infra | 23:21 | |
*** sherif has quit IRC | 23:22 | |
openstackgerrit | greghaynes proposed openstack/diskimage-builder: Perform a booting test for our images https://review.openstack.org/204639 | 23:23 |
*** jasondotstar has joined #openstack-infra | 23:25 | |
*** esmute_ has quit IRC | 23:25 | |
openstackgerrit | Merged openstack-infra/puppet-refstack: Add .gitreview & fix puppet-lint errors/warnings https://review.openstack.org/198910 | 23:27 |
*** esmute_ has joined #openstack-infra | 23:27 | |
*** sergmelikyan has joined #openstack-infra | 23:28 | |
*** Guest56855 has quit IRC | 23:29 | |
*** sergmelikyan has quit IRC | 23:30 | |
*** gildub has joined #openstack-infra | 23:35 | |
*** Sukhdev has quit IRC | 23:38 | |
greghaynes | clarkb: https://jenkins07.openstack.org/job/gate-dib-dsvm-functests-devstack-f21/115/console ! | 23:43 |
greghaynes | SpamapS: ^ | 23:43 |
greghaynes | SpamapS: That is a test that qemu boots a dib built image and verifies we can ping | 23:44 |
*** crc32 has quit IRC | 23:45 | |
greghaynes | only took about 2mins to boot too | 23:46 |
clarkb | nice | 23:46 |
*** vlaza has joined #openstack-infra | 23:46 | |
*** ajmiller has quit IRC | 23:47 | |
*** sabeen3 has quit IRC | 23:48 | |
*** xyang1 has quit IRC | 23:48 | |
*** achanda has quit IRC | 23:49 | |
*** davideagnello has quit IRC | 23:53 | |
openstackgerrit | Ghanshyam Mann proposed openstack-infra/project-config: Move multinode job from experimental to non-voting for Tempest https://review.openstack.org/203923 | 23:53 |
*** rm_work is now known as rm_work|away | 23:53 | |
*** gildub has quit IRC | 23:54 | |
*** FallenPegasus has joined #openstack-infra | 23:56 | |
*** hemna is now known as hemnafk | 23:57 | |
*** dingyichen has joined #openstack-infra | 23:57 | |
SpamapS | greghaynes: I like it! | 23:58 |
*** sergmelikyan has joined #openstack-infra | 23:58 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!