Wednesday, 2015-07-22

crinklethe posix provider "prevents the use of globbing and shell built-ins" --> seems like it shouldn't work00:00
crinklebut it worked for me00:00
*** tbarron has joined #openstack-infra00:00
*** tsekiyam_ has quit IRC00:00
*** hichihara has joined #openstack-infra00:01
*** jasondotstar has quit IRC00:01
*** PpPyg has joined #openstack-infra00:01
*** boris-42 has joined #openstack-infra00:02
*** camunoz_mtg has quit IRC00:02
*** sergmelikyan has quit IRC00:02
*** shashank_hegde has quit IRC00:04
*** sergmelikyan has joined #openstack-infra00:04
*** pcrews has quit IRC00:05
*** madhu_ak has quit IRC00:05
*** jamesmcarthur has quit IRC00:06
*** DericHorn-HP_ has joined #openstack-infra00:07
*** nimrodsun_ has joined #openstack-infra00:07
*** yfried_ has joined #openstack-infra00:07
*** yfried_ is now known as yfried|afk00:07
*** dimtruck is now known as zz_dimtruck00:08
*** pahuang__ has quit IRC00:09
*** mwagner_lap has joined #openstack-infra00:09
*** achanda has quit IRC00:16
*** jasondotstar has joined #openstack-infra00:16
*** jasondotstar has quit IRC00:16
*** davideagnello has quit IRC00:16
*** jasondotstar has joined #openstack-infra00:16
*** DericHorn-HP has quit IRC00:17
*** ildikov has quit IRC00:17
*** yfried has quit IRC00:17
*** linkedinyou has quit IRC00:17
*** nimrodsun has quit IRC00:17
*** DericHorn-HP_ is now known as DericHorn-HP00:17
akraminsky_afazekas: im having some issues with sshing into my subnodes using their private IP's. Can you explain why you use the private ip's instead of public in devstack-gate-vm.sh line 499? :)00:18
*** nimrodsun_ is now known as nimrodsun00:18
*** shardy_ has joined #openstack-infra00:18
akraminsky_afazekas: https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate.sh00:18
*** shardy has quit IRC00:19
*** shardy_ has quit IRC00:23
*** shardy has joined #openstack-infra00:23
*** ashleighfarnham has quit IRC00:24
*** ildikov has joined #openstack-infra00:25
*** pahuang has joined #openstack-infra00:26
*** tbarron has quit IRC00:27
*** esmute_ has quit IRC00:29
*** esmute__ has joined #openstack-infra00:29
*** tbarron has joined #openstack-infra00:30
*** patrickeast has quit IRC00:30
*** geoffarnold has quit IRC00:32
*** tbarron has quit IRC00:33
*** sdake has quit IRC00:34
*** pvaneck has quit IRC00:36
*** sergmelikyan has quit IRC00:37
*** sdake has joined #openstack-infra00:37
*** Swami has quit IRC00:39
*** tbarron has joined #openstack-infra00:40
*** cody-somerville has quit IRC00:42
*** salv-orlando has quit IRC00:43
*** shashank_hegde has joined #openstack-infra00:43
*** esmute__ has quit IRC00:45
*** sergmelikyan has joined #openstack-infra00:46
*** rlandy has quit IRC00:46
*** rlandy has joined #openstack-infra00:47
*** sergmelikyan has quit IRC00:47
*** DericHorn-HP has quit IRC00:54
*** sergmelikyan has joined #openstack-infra00:55
*** jasondotstar has quit IRC00:56
*** sergmelikyan has quit IRC00:56
openstackgerritliusheng proposed openstack/requirements: Bump gabbit to version 1.1.0  https://review.openstack.org/20195300:58
openstackgerritliusheng proposed openstack/requirements: Bump gabbi to version 1.1.0  https://review.openstack.org/20195300:59
*** ZZelle_ has quit IRC00:59
*** serg_melikyan has joined #openstack-infra01:00
*** erlon has quit IRC01:00
*** shashank_hegde has quit IRC01:05
*** kutija has quit IRC01:07
*** ddieterl_ has joined #openstack-infra01:08
*** ddieterly has quit IRC01:08
*** amitgandhinz has joined #openstack-infra01:09
*** Krinkle has joined #openstack-infra01:11
*** shashank_hegde has joined #openstack-infra01:11
*** shashank_hegde has quit IRC01:11
openstackgerritColleen Murphy proposed openstack-infra/puppet-cgit: Add Gemfile and puppet 4 checks  https://review.openstack.org/20387001:12
*** esmute_ has joined #openstack-infra01:12
openstackgerritColleen Murphy proposed openstack-infra/puppet-diskimage_builder: Add Gemfile and puppet 4 checks  https://review.openstack.org/20266101:12
openstackgerrittakehirokaneko proposed openstack-infra/project-config: Add coverage test to check pipeline of stackforge/rack  https://review.openstack.org/20397601:13
*** DericHorn-HP has joined #openstack-infra01:13
*** amickus has quit IRC01:14
*** serg_melikyan has quit IRC01:15
*** puranamr has joined #openstack-infra01:16
*** camunoz has joined #openstack-infra01:17
*** sergmelikyan has joined #openstack-infra01:18
openstackgerritIWAMOTO Toshihiro proposed openstack/requirements: Add ryu  https://review.openstack.org/15435401:20
*** sergmelikyan has quit IRC01:23
*** esmute_ has quit IRC01:24
*** alexsyip has quit IRC01:26
*** asselin_ has joined #openstack-infra01:27
*** bdemers has joined #openstack-infra01:33
*** esmute_ has joined #openstack-infra01:33
*** shashank_hegde has joined #openstack-infra01:35
*** rlandy has quit IRC01:37
*** thorst has joined #openstack-infra01:38
*** flepied has joined #openstack-infra01:38
*** rlandy has joined #openstack-infra01:38
*** changbl has joined #openstack-infra01:43
openstackgerritColleen Murphy proposed openstack-infra/puppet-diskimage_builder: Add Gemfile and puppet 4 checks  https://review.openstack.org/20266101:43
*** salv-orlando has joined #openstack-infra01:43
*** pcrews has joined #openstack-infra01:44
*** sergmelikyan has joined #openstack-infra01:44
*** armax has quit IRC01:44
*** yamamoto has quit IRC01:45
openstackgerritColleen Murphy proposed openstack-infra/puppet-cgit: Add Gemfile and puppet 4 checks  https://review.openstack.org/20387001:45
*** jamesmcarthur has joined #openstack-infra01:46
*** salv-orlando has quit IRC01:46
*** amitgandhinz has quit IRC01:46
*** sergmelikyan has quit IRC01:46
openstackgerritColleen Murphy proposed openstack-infra/puppet-drupal: Add Gemfile and puppet 4 checks  https://review.openstack.org/20388101:46
*** pal has joined #openstack-infra01:47
*** puranamr has quit IRC01:47
StevenKpleia2: If you're still around, https://review.openstack.org/203927 needs a review, pretty please.01:47
StevenKnibalizer, crinkle: Did either of you poke at the manifest log file I uploaded, by the way?01:48
nibalizerStevenK: ah no, relink?01:48
StevenKnibalizer: http://wedontsleep.org/~steven/manifest.log.gz01:49
openstackgerritColleen Murphy proposed openstack-infra/puppet-elastic_recheck: Add Gemfile and puppet 4 checks  https://review.openstack.org/20388401:50
*** bdemers has quit IRC01:50
openstackgerritColleen Murphy proposed openstack-infra/puppet-elasticsearch: Add Gemfile and puppet 4 checks  https://review.openstack.org/20388601:50
nibalizerStevenK: what puppet version?01:50
StevenKnibalizer: Whatever the latest from Trusty is01:51
StevenKI think01:51
StevenKLet me boot the VM01:51
nibalizerok01:51
*** DericHorn-HP has quit IRC01:52
openstackgerritTimo Tijhof proposed openstack-infra/zuul: Cloner: Implement cache-no-hardlinks argument  https://review.openstack.org/11762601:54
*** bradjones has quit IRC01:54
*** sergmelikyan has joined #openstack-infra01:54
*** bdemers has joined #openstack-infra01:55
*** sergmelikyan has quit IRC01:55
*** sergmelikyan has joined #openstack-infra01:56
*** bradjones has joined #openstack-infra01:56
*** bradjones has quit IRC01:56
*** bradjones has joined #openstack-infra01:56
*** sergmelikyan has quit IRC01:57
StevenKnibalizer: Sorry about that. It's 3.8.101:59
nibalizerno worries02:00
nibalizerjust as long as its not 4 lol02:00
*** DericHorn-HP has joined #openstack-infra02:00
nibalizeroh you know what that looks like to me02:01
nibalizeris a weird puppet way of saying the superior directory doesn't exist02:01
nibalizerdoes /opt/wildfly/standalone/deployments/ exist?02:01
StevenKdrwxr-xr-x 2 wildfly wildfly 4.0K Jul 21 04:16 deployments02:01
StevenKYup02:01
nibalizerhrm02:01
*** sergmelikyan has joined #openstack-infra02:02
*** julim has joined #openstack-infra02:03
* nibalizer strokes nonexistent goatee02:03
*** rlandy has quit IRC02:04
*** annegent_ has quit IRC02:04
crinkleI don't understand why it's referring to a .lock file, is this a different patchset than master?02:04
nibalizeryes this question ^02:05
nibalizerit looks like it should be looking for zanata-war-3.6.2.war02:05
*** annegentle has joined #openstack-infra02:06
crinkleI thought it was looking for ROOT.war http://git.openstack.org/cgit/openstack-infra/puppet-zanata/tree/manifests/init.pp#n8502:06
StevenKThat's what it's trying to create02:07
StevenK-rw-r--r-- 1 wildfly wildfly 97M Apr  2 07:04 /home/wildfly/zanata-war-3.6.2.war02:07
StevenKcrinkle: And no02:07
*** sergmelikyan has quit IRC02:07
StevenKroot@zanata-server:/etc/puppet/modules/zanata# git log --pretty=format:%h -n 102:08
StevenK52d2ebc02:08
*** esmute_ has quit IRC02:08
*** yamamoto has joined #openstack-infra02:09
nibalizerStevenK: okay the beers are calling to me02:09
*** annegentle has quit IRC02:10
nibalizersorry I didn't remember to debug this before my day ended02:10
nibalizerfungi: pleia2 yall osconpartying? ping me if you want to meetup02:10
crinkleStevenK: and adding file:// to the value in the source parameter fixes it?02:11
StevenKcrinkle: That makes it work, yes02:12
crinklethat's really weird02:12
nibalizeri just want to say one last time that this behaviour is pretty o.O02:12
nibalizerlike wat02:12
nibalizermaybe we should just open a jira ticket for puppet like 'wat'02:12
*** esmute_ has joined #openstack-infra02:13
*** DericHorn-HP has quit IRC02:14
*** puranamr has joined #openstack-infra02:18
*** puranamr has quit IRC02:18
*** ivar-laz_ has quit IRC02:18
*** ivar-lazzaro has joined #openstack-infra02:19
*** thorst has quit IRC02:20
*** flepied has quit IRC02:24
*** linkedinyou has joined #openstack-infra02:25
*** dims_ has joined #openstack-infra02:25
*** koolhead17 has joined #openstack-infra02:30
*** unicell has quit IRC02:31
*** timrc_ has joined #openstack-infra02:32
*** pcrews has quit IRC02:33
*** achanda has joined #openstack-infra02:33
*** bharath has joined #openstack-infra02:35
*** pahuang has quit IRC02:35
*** rossella_s has quit IRC02:37
*** rossella_s has joined #openstack-infra02:37
*** glauco has joined #openstack-infra02:39
*** bharath has quit IRC02:39
*** gyee has quit IRC02:39
openstackgerritColleen Murphy proposed openstack-infra/puppet-accessbot: Remove puppet-lint-numericvariable gem  https://review.openstack.org/20434302:42
openstackgerritColleen Murphy proposed openstack-infra/puppet-ansible: Remove puppet-lint-numericvariable gem  https://review.openstack.org/20434402:42
openstackgerritColleen Murphy proposed openstack-infra/puppet-apparmor: Remove puppet-lint-numericvariable gem  https://review.openstack.org/20434502:42
openstackgerritColleen Murphy proposed openstack-infra/puppet-apps_site: Remove puppet-lint-numericvariable gem  https://review.openstack.org/20434602:42
openstackgerritColleen Murphy proposed openstack-infra/puppet-askbot: Remove puppet-lint-numericvariable gem  https://review.openstack.org/20434702:42
openstackgerritColleen Murphy proposed openstack-infra/puppet-asterisk: Remove puppet-lint-numericvariable gem  https://review.openstack.org/20434802:42
openstackgerritColleen Murphy proposed openstack-infra/puppet-bandersnatch: Remove puppet-lint-numericvariable gem  https://review.openstack.org/20434902:42
openstackgerritColleen Murphy proposed openstack-infra/puppet-bugdaystats: Remove puppet-lint-numericvariable gem  https://review.openstack.org/20435002:42
openstackgerritColleen Murphy proposed openstack-infra/puppet-bup: Remove puppet-lint-numericvariable gem  https://review.openstack.org/20435102:42
*** pal has quit IRC02:43
*** pcrews has joined #openstack-infra02:45
*** pahuang has joined #openstack-infra02:47
*** mkj_wk_ is now known as mkj_wk02:53
*** julim has quit IRC02:53
openstackgerritTimothy R. Chavez proposed openstack-infra/puppet-cgit: Add better defaults for clone-prefix  https://review.openstack.org/20424502:54
*** yamahata has quit IRC02:54
*** unicell has joined #openstack-infra02:54
*** Krinkle is now known as Krinkle_02:55
*** amitgandhinz has joined #openstack-infra02:57
*** DericHorn-HP has joined #openstack-infra02:57
*** DericHorn-HP has quit IRC02:58
*** unicell1 has joined #openstack-infra02:58
*** sabeen has joined #openstack-infra02:59
*** unicell has quit IRC02:59
*** amitgandhinz has quit IRC03:01
*** auxian has joined #openstack-infra03:01
*** DericHorn-HP has joined #openstack-infra03:02
*** jerryz has quit IRC03:07
*** dingyichen has joined #openstack-infra03:07
openstackgerritVadivel Poonathan proposed openstack-infra/project-config: Add ALE Omniswitch Networking project to Openstack  https://review.openstack.org/20347703:08
*** linkedinyou|2 has joined #openstack-infra03:09
*** kiran-r has joined #openstack-infra03:10
*** linkedinyou|2 has quit IRC03:11
*** linkedinyou has quit IRC03:12
*** heyongli has quit IRC03:12
*** DericHorn-HP has quit IRC03:13
*** bdemers has quit IRC03:14
*** heyongli has joined #openstack-infra03:14
*** bdemers has joined #openstack-infra03:14
*** ivar-laz_ has joined #openstack-infra03:15
*** dims_ has quit IRC03:15
*** bdemers has quit IRC03:18
*** ivar-lazzaro has quit IRC03:18
*** bdemers has joined #openstack-infra03:21
*** koolhead17 has quit IRC03:21
*** salv-orlando has joined #openstack-infra03:22
*** tlian has quit IRC03:24
*** bdemers has quit IRC03:26
*** bdemers has joined #openstack-infra03:27
*** salv-orlando has quit IRC03:27
*** kiran-r has quit IRC03:28
*** pal has joined #openstack-infra03:29
*** linkedinyou has joined #openstack-infra03:31
*** glauco has quit IRC03:31
*** thorst has joined #openstack-infra03:31
*** crc32 has joined #openstack-infra03:32
*** pahuang has quit IRC03:33
*** bhunter71 has quit IRC03:33
*** heyongli has quit IRC03:35
*** sherif has joined #openstack-infra03:35
*** DericHorn-HP has joined #openstack-infra03:36
*** salv-orlando has joined #openstack-infra03:37
*** glauco_ has joined #openstack-infra03:37
*** salv-orlando has quit IRC03:37
*** heyongli has joined #openstack-infra03:40
*** bdemers has quit IRC03:41
*** bdemers has joined #openstack-infra03:41
openstackgerritlitong01 proposed openstack-infra/project-config: Add new monitor repository for OpenStack monitoring service  https://review.openstack.org/20325903:43
*** jamesmcarthur has quit IRC03:45
*** thorst has quit IRC03:45
*** gus_ is now known as gus03:45
*** glauco_ has quit IRC03:47
*** dims_ has joined #openstack-infra03:49
*** pahuang has joined #openstack-infra03:50
*** achanda has quit IRC03:50
*** bdemers has quit IRC03:55
*** dims_ has quit IRC03:56
*** bdemers has joined #openstack-infra03:56
*** gildub has quit IRC03:57
*** coolsvap|away is now known as coolsvap04:01
*** rfolco has quit IRC04:03
*** jamesmcarthur has joined #openstack-infra04:07
*** sherif has quit IRC04:13
*** Kennan2 has joined #openstack-infra04:14
*** Kennan has quit IRC04:15
openstackgerritMerged openstack-infra/gearman-plugin: Fix deadlock from a WORK_FAIL event  https://review.openstack.org/19242904:16
*** bdemers has quit IRC04:17
*** bdemers has joined #openstack-infra04:18
*** gildub has joined #openstack-infra04:19
*** linkedinyou has quit IRC04:22
openstackgerritMerged openstack/requirements: Add keystoneauth to projects.txt  https://review.openstack.org/20325604:22
*** VivCheri has joined #openstack-infra04:22
*** VivCheri has quit IRC04:24
*** camunoz is now known as camunoz_mtg04:24
lifelessmordred: file a bug on pbr, I'll look at it and see if I can eithe rdo it trivially, or mentor someone on it04:24
*** puranamr has joined #openstack-infra04:24
openstackgerritMerged openstack/requirements: Bumping Cinder client to 1.3.1  https://review.openstack.org/19934804:26
openstackgerritMerged openstack-infra/project-config: Remove gate-tempest-dsvm-docker from nova's check queue  https://review.openstack.org/20426404:27
*** tlbr_ has quit IRC04:31
*** tlbr has joined #openstack-infra04:32
*** glauco_ has joined #openstack-infra04:32
*** boris-42 has quit IRC04:32
*** bdemers has quit IRC04:34
*** bdemers has joined #openstack-infra04:35
*** DericHorn-HP has quit IRC04:35
*** glauco_ has quit IRC04:36
*** puranamr has quit IRC04:37
openstackgerritMerged openstack/requirements: mock 1.2 has the 1.1.4 regression fixed and is 2.6 compatible  https://review.openstack.org/20323104:37
*** achanda has joined #openstack-infra04:37
*** puranamr has joined #openstack-infra04:37
*** bdemers has quit IRC04:42
*** bdemers has joined #openstack-infra04:42
openstackgerritMerged openstack/diskimage-builder: debian: install DHCP client and ifconfig packages  https://review.openstack.org/20003004:45
*** harlowja_at_home has joined #openstack-infra04:47
openstackgerritMerged openstack/requirements: Bump lower/upper constraint of futurist to newly released 0.1.2  https://review.openstack.org/20170904:47
*** larainema has quit IRC04:48
*** larainema has joined #openstack-infra04:48
*** tsg has quit IRC04:52
openstackgerritMerged openstack/requirements: Adding angular-gettext to global requirements  https://review.openstack.org/19934904:53
openstackgerritOpenStack Proposal Bot proposed openstack-dev/hacking: Updated from global requirements  https://review.openstack.org/18682904:54
openstackgerritOpenStack Proposal Bot proposed openstack-dev/pbr: Updated from global requirements  https://review.openstack.org/20438604:54
openstackgerritOpenStack Proposal Bot proposed openstack/diskimage-builder: Updated from global requirements  https://review.openstack.org/20185204:54
*** bdemers has quit IRC04:55
*** bdemers has joined #openstack-infra04:55
*** maishsk has quit IRC04:57
*** dalgaaf has quit IRC04:58
*** jamesmcarthur has quit IRC05:03
*** kutija has joined #openstack-infra05:03
*** mrmartin has joined #openstack-infra05:05
*** shashank_hegde has quit IRC05:06
funginibalizer: oh, sorry, clarkb drug us out looking for a brewery which wasn't overrun by a tool concert05:07
fungiwe skipped from ex novo over to lompoc, then called it an early evening05:08
nibalizerhah. sounds like clarkb05:08
fungiwell, he has babies now05:08
nibalizerill be conferencing properly tomorrow so we will likely see each lther05:08
fungithose apparently are lots of work05:08
fungiyep, i'll be there all day. see you there!05:08
openstackgerritMerged openstack-infra/project-config: Switch default pattern in create-zanata-xml.py  https://review.openstack.org/20392705:10
*** puranamr has quit IRC05:10
*** esmute_ has quit IRC05:13
*** esmute_ has joined #openstack-infra05:16
crinkleStevenK: I think nibalizer was right about the parent directory not being created05:16
crinkleStevenK: it gets created by an exec that untars a tarball in the wildfly module https://github.com/biemond/biemond-wildfly/blob/v0.2.4/manifests/install.pp#L16-L2305:16
crinkleStevenK: adding a require relationship on that resource fixed it for me http://paste.openstack.org/show/398518/05:17
*** deepakcs has joined #openstack-infra05:17
openstackgerritDavid Cheperdak proposed openstack-infra/project-config: Add Stackforge project Fusion.  https://review.openstack.org/20443405:19
*** ivar-laz_ has quit IRC05:21
*** yamamoto has quit IRC05:21
*** ivar-lazzaro has joined #openstack-infra05:21
*** yamamoto has joined #openstack-infra05:23
clarkbTool and oscon at tge same time whats going on05:24
fungiso much crazy05:25
*** ivar-lazzaro has quit IRC05:26
*** glauco has joined #openstack-infra05:26
*** harlowja_at_home has quit IRC05:26
afazekasakraminsky_: The nodes expected to be able to see each other on the private net with less trouble. The private_ip is not necessarily directly reachable for external user.05:30
*** davidjc has joined #openstack-infra05:30
*** abregman has joined #openstack-infra05:30
StevenKcrinkle: Ah! An ordering problem, neat05:31
StevenKcrinkle: Thanks for digging into it :-)05:31
*** glauco has quit IRC05:31
crinkleStevenK: no problem :)05:31
StevenKRight, and that's why file:// works -- it isn't that it works per-se, but it's a second run, and the directory exists.05:32
StevenKSo blah05:32
crinkleright05:32
*** salv-orlando has joined #openstack-infra05:33
StevenKcrinkle: Hmm, that's another place that the wildfly version gets hardcoded.05:35
*** salv-orlando has quit IRC05:38
crinkleStevenK: hmm i think you could use $zanata_file in place of the file name in that exec title05:38
StevenKcrinkle: That's the zanata WAR, not the wildfly tarball05:39
StevenKBut I may just copy that pattern05:39
crinkleoh right05:39
crinkleer no, i think it is the tarball in that exec05:40
*** mrmartin has quit IRC05:40
StevenKYup, which comes from zanata_wildfly_install_url05:41
crinkleoh I was looking at the wrong variable05:41
crinkleyou're right05:42
*** shashank_hegde has joined #openstack-infra05:43
StevenKcrinkle: I have a diff locally, I'm just spawning an unsullied VM so we can see that the module behaves correctly for first ever run05:43
*** davidjc has quit IRC05:45
*** heyongli has quit IRC05:49
*** heyongli has joined #openstack-infra05:50
*** afazekas has quit IRC05:55
*** BharatK has joined #openstack-infra05:56
*** _nadya_ has joined #openstack-infra05:56
*** yfried|afk has quit IRC06:05
*** dchen has joined #openstack-infra06:07
*** amotoki has joined #openstack-infra06:09
*** dingyichen has quit IRC06:09
*** timrc_ has quit IRC06:10
*** puranamr has joined #openstack-infra06:11
*** camunoz_mtg has quit IRC06:11
*** jtomasek has joined #openstack-infra06:12
*** pahuang has quit IRC06:12
*** andrey-mp has joined #openstack-infra06:13
*** maishsk has joined #openstack-infra06:13
*** puranamr has quit IRC06:15
*** dims_ has joined #openstack-infra06:17
StevenK2015-07-22 06:17:01 +0000 Puppet (err): Could not find dependency Exec[tar ${wildfly_file} in /var/tmp] for File. Blah.06:18
*** AJaeger has joined #openstack-infra06:18
*** ParsectiX has joined #openstack-infra06:19
AJaegerany root still (already) around? jhesketh, fungi, clarkb, jeblair, pleia2, SergeyLukjanov: Looks like zuul is not removing jobs from the queue at all. Jobs are finished in the check queue but stay there according to status.openstack.org/zuul06:20
AJaeger(same with gate queue)06:21
*** dims_ has quit IRC06:22
*** pahuang has joined #openstack-infra06:23
*** heyongli has quit IRC06:24
*** dingyichen has joined #openstack-infra06:24
*** camunoz_mtg has joined #openstack-infra06:24
*** scheuran has joined #openstack-infra06:24
*** dtantsur|afk is now known as dtantsur06:25
*** kiran-r has joined #openstack-infra06:26
*** dingyichen has quit IRC06:27
*** jyuso1 has quit IRC06:29
*** jyuso has joined #openstack-infra06:29
*** boris-42 has joined #openstack-infra06:29
*** caphrim007 has quit IRC06:30
*** sdake has quit IRC06:31
*** shardy_ has joined #openstack-infra06:31
*** shardy has quit IRC06:32
*** koolhead17 has joined #openstack-infra06:32
openstackgerritMarc Koderer proposed openstack-infra/project-config: Add experimental job to test Manila Tempest plugin  https://review.openstack.org/20445406:34
*** heyongli has joined #openstack-infra06:34
*** kushal has joined #openstack-infra06:35
*** shardy_ has quit IRC06:36
*** tangchen_ has joined #openstack-infra06:36
tangchen_Hi, all. :)06:37
tangchen_anyone here ?06:37
*** shardy has joined #openstack-infra06:37
*** rossella_s has quit IRC06:37
tangchen_I'd like to ask something about 3rd party CI.06:37
*** rossella_s has joined #openstack-infra06:38
*** afazekas_ has joined #openstack-infra06:39
jheskethAJaeger: hmm, I'll take a look06:39
AJaegerjhesketh: interesting that the first job in the check queue (nova, 187857,14) does not expand at all06:41
AJaegerjhesketh: thanks!06:41
AJaegertangchen_: seems nobody with that knowledge is awake06:41
*** mrunge has joined #openstack-infra06:42
*** mfalatic has quit IRC06:42
openstackgerritVadivel Poonathan proposed openstack-infra/project-config: Add ALE Omniswitch Networking project to Openstack  https://review.openstack.org/20347706:44
*** vlaza has joined #openstack-infra06:45
*** mattt has left #openstack-infra06:45
jheskethtangchen_: I might be able to help later after I've looked at the gate06:46
*** dims_ has joined #openstack-infra06:47
openstackgerritSteve Kowalik proposed openstack-infra/puppet-zanata: Add second requirement for the ROOT.war deployment  https://review.openstack.org/20194306:48
tangchen_jhesketh, AJaeger: Hi, Thanks for reply.06:49
jheskethzuul's internal database seems to have gotten into a bad state06:50
jheskethI'm not sure if there's a fix that isn't a zuul restart06:50
jheskethbut I'd like to figure out how it got there first06:50
tangchen_I'm setting up my first CI, and when I send a patch to gerrit, it cannot be merged.06:50
jheskethlots of http://paste.openstack.org/show/398833/ in the logs06:50
tangchen_It reports: "Unable to find change queue for change <Change 0x7fa7ef8b6250 204446,1> in project openstack-dev/sandbox"06:51
*** heyongli has quit IRC06:51
*** deepakcs has quit IRC06:51
tangchen_Do you guys have any idea of this ?06:51
*** andrey-mp has quit IRC06:53
*** dims_ has quit IRC06:53
*** heyongli has joined #openstack-infra06:53
*** k4n0 has joined #openstack-infra06:55
*** jyuso has quit IRC06:56
openstackgerritDavid Cheperdak proposed openstack-infra/project-config: Add Stackforge project Fusion.  https://review.openstack.org/20443406:56
*** r-mibu has joined #openstack-infra06:57
*** coolsvap is now known as coolsvap|away07:01
*** _nadya_ has quit IRC07:04
AJaegerjhesketh: I have no idea about Zuul logs, sorry. But this looks indeed like some data structure is broken.07:07
*** bharath_ has joined #openstack-infra07:07
jheskethAJaeger: it appears to be due to a race condition in the zuul reload07:07
jheskethAJaeger: I'm just prepping for a restart and re-queue07:07
AJaegerjhesketh: ok07:08
*** kushal has quit IRC07:08
*** sergmelikyan has joined #openstack-infra07:09
*** heyongli has quit IRC07:10
*** dims_ has joined #openstack-infra07:12
*** esmute_ has quit IRC07:12
*** ifarkas has joined #openstack-infra07:12
*** deepakcs has joined #openstack-infra07:13
*** glauco_ has joined #openstack-infra07:14
*** dims_ has quit IRC07:16
*** heyongli has joined #openstack-infra07:17
jheskethrestarting zuul07:18
*** glauco_ has quit IRC07:19
AJaegerjhesketh: what about sending a notification via IRC?07:19
jheskethgood idea07:21
*** jerryz has joined #openstack-infra07:22
jhesketh#status notice zuul is undergoing an emergency restart. Jobs will be re-queued but some events may be lost.07:22
openstackstatusjhesketh: sending notice07:22
*** e0ne has joined #openstack-infra07:22
*** ildikov has quit IRC07:22
-openstackstatus- NOTICE: zuul is undergoing an emergency restart. Jobs will be re-queued but some events may be lost.07:23
*** cody-somerville has joined #openstack-infra07:23
*** kushal has joined #openstack-infra07:24
openstackstatusjhesketh: finished sending notice07:24
*** salv-orlando has joined #openstack-infra07:26
*** sshnaidm has joined #openstack-infra07:26
AJaegerthanks, jhesketh! Hope that everything is fine.07:26
AJaegerRight now I wonder why stackforge/fuel-agent is on the top of the check queue - with only a noop change.07:27
jheskethI think the mergers are busy07:27
AJaegerLet's hope ;)07:27
jheskethheh, yeah07:27
*** camunoz_mtg is now known as camunoz_gone07:28
*** maishsk has quit IRC07:29
*** achanda has quit IRC07:30
*** fhubik has joined #openstack-infra07:31
openstackgerritliusheng proposed openstack/requirements: Bump gabbi to version 1.1.0  https://review.openstack.org/20195307:31
*** pal has quit IRC07:33
*** heyongli has quit IRC07:33
*** odyssey4me has quit IRC07:34
*** odyssey4me has joined #openstack-infra07:37
*** heyongli has joined #openstack-infra07:38
*** koolhead17 has quit IRC07:38
*** achanda has joined #openstack-infra07:39
*** dizquierdo has joined #openstack-infra07:39
*** jlanoux has joined #openstack-infra07:41
openstackgerritAngus Lees proposed openstack-infra/project-config: Add new oslo.privsep project  https://review.openstack.org/20447707:43
*** dims_ has joined #openstack-infra07:44
*** zeih has joined #openstack-infra07:44
*** afazekas_ has quit IRC07:45
AJaegerjhesketh: doesn't look like any job is moving forward ;( - or am I to impatient?07:47
jheskethAJaeger: no07:48
jheskethclients are failing to reconnect to the gearman service07:48
jhesketheg mergers and jenkins07:48
*** dguerri` is now known as dguerri07:48
jheskethI can telnet to the gearman service on localhost, but not remotely despite the firewall rules looking correct07:48
jheskethso I'm possibly missing something07:48
*** achanda has quit IRC07:49
jheskethSergeyLukjanov, jeblair, fungi, clarkb, mordred, pleia2: anybody around per chance?07:49
openstackgerritAndrei V. Ostapenko proposed openstack-infra/project-config: Move from stackforge/magnetodb to openstack/magnetodb  https://review.openstack.org/20352407:51
openstackgerritAndrei V. Ostapenko proposed openstack-infra/project-config: Changes jobs description for MagnetoDB  https://review.openstack.org/20447807:51
kiran-rThere is a problem in jenkins07:54
kiran-rALl the tests are queued07:54
kiran-rNone are running07:54
*** dims_ has quit IRC07:54
*** afazekas_ has joined #openstack-infra07:58
flaper87AJaeger: is gertty working properly for you? (do you use it?)07:58
flaper87I'm having troubles to sync stuff07:58
jheskethkiran-r: yes there are some problems at the moment07:58
flaper87but I couldn't find a bug reporting issues07:58
flaper87jeblair: ^07:58
AJaegerflaper87: I'm not using gertty, still on my todo list.08:01
*** HeOS has quit IRC08:02
flaper87AJaeger: gotcha08:03
*** jistr has joined #openstack-infra08:03
*** gildub has quit IRC08:03
*** scheuran has quit IRC08:04
*** e0ne has quit IRC08:05
AJaegerflaper87: could you import glance translations again soonish, please? https://review.openstack.org/#/c/197206/08:06
*** dizquierdo has quit IRC08:06
flaper87AJaeger: Sir, yes sir.08:07
*** pal has joined #openstack-infra08:09
*** dims_ has joined #openstack-infra08:12
jhesketh#status alert zuul jobs are currently stuck while problems with gearman are debugged08:12
openstackstatusjhesketh: sending alert08:12
*** esmute_ has joined #openstack-infra08:13
*** Hal has joined #openstack-infra08:14
*** gaurangt has joined #openstack-infra08:14
-openstackstatus- NOTICE: zuul jobs are currently stuck while problems with gearman are debugged08:14
*** andrey-mp has joined #openstack-infra08:14
*** ChanServ changes topic to "zuul jobs are currently stuck while problems with gearman are debugged"08:14
*** Hal is now known as Guest8374008:14
*** dims_ has quit IRC08:16
openstackstatusjhesketh: finished sending alert08:16
*** esmute_ has quit IRC08:17
*** derekh has joined #openstack-infra08:17
*** scheuran has joined #openstack-infra08:19
*** crc32 has quit IRC08:19
*** mrmartin has joined #openstack-infra08:20
*** heyongli has quit IRC08:21
*** heyongli has joined #openstack-infra08:23
*** maishsk has joined #openstack-infra08:23
*** ihrachyshka has joined #openstack-infra08:23
*** ihrachyshka_ has joined #openstack-infra08:23
*** kiran-r has quit IRC08:24
*** u_glide has joined #openstack-infra08:24
*** heyongli has quit IRC08:27
*** heyongli has joined #openstack-infra08:28
*** ildikov has joined #openstack-infra08:29
*** mpaolino has joined #openstack-infra08:29
*** gaurangt1 has joined #openstack-infra08:30
*** gaurangt has quit IRC08:31
*** maishsk has quit IRC08:31
*** boris-42 has quit IRC08:32
*** maishsk has joined #openstack-infra08:32
openstackgerritAtsushi SAKAI proposed openstack/diskimage-builder: typos on the document  https://review.openstack.org/20449108:32
*** jerryz has quit IRC08:33
*** tmorin has joined #openstack-infra08:33
*** bharath_ has quit IRC08:33
tmorinI have an issue where launchpad bugs are not autatically commented on by hudson08:34
tmorinanyone could help ?08:34
tmorinthis happens for all changes of networking-bgpvpn project08:35
tmorinexample:  https://review.openstack.org/#/c/199617/  (networking-bgpvpn project)08:35
tmorinping Ajaeger : would you have a look, or suggest someone who would ?08:35
*** dtantsur is now known as dtantsur|brb08:36
*** dchen has quit IRC08:36
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/shade: Fix shade returning empty private_v4  https://review.openstack.org/20449508:38
*** akasatkin has joined #openstack-infra08:38
*** ZZelle has quit IRC08:39
*** abregman is now known as abregman|afk08:40
*** ZZelle has joined #openstack-infra08:40
*** vinay_yadhav has joined #openstack-infra08:40
*** zeih has quit IRC08:41
*** dims_ has joined #openstack-infra08:41
*** _nadya_ has joined #openstack-infra08:41
*** zeih has joined #openstack-infra08:41
openstackgerritAntoine Musso proposed openstack-infra/zuul: Document Gerrit labels being case sensitive  https://review.openstack.org/20449908:43
AJaegertmorin: your repo is networking-bgvpn, but launchpad uses bgvpn.08:45
*** dims_ has quit IRC08:45
AJaegertmorin: see the greay box in http://docs.openstack.org/infra/manual/creators.html#add-the-repository-to-the-master-repositories-list08:46
tmorinAjaeger: yes, I had identified at the time the fact that the launchpad project had not the right name (not created by me), but failed to see what had to be done to point openstack-infra to the right LP project08:46
jheskethOkay tracked down the issue with zuul, going to see if I can restart and enqueue everything08:47
tmorinAjaeger: actually, I did this fix for networking-bagpipe-l2, but networking-bgpvpn was not done by me and had not the "group: .." statement08:48
tmorinI will add it08:48
tmorinthanks08:48
*** hashar has joined #openstack-infra08:48
AJaegeryou're welcome, tmorin08:49
*** maishsk has quit IRC08:49
openstackgerritThomas Morin proposed openstack-infra/project-config: Point networking-bgpvpn to correct LP project  https://review.openstack.org/20450208:51
tmorinAJaeger: done08:52
*** vinay_yadhav has quit IRC08:52
tmorinAjaeger: for previous events on changes, I guess hudson will not add the comments and I have to update the bug by hand08:52
AJaegertmorin: thanks. Will review later...08:53
openstackgerritJoshua Hesketh proposed openstack-infra/puppet-zuul: Listen to all addresses by default  https://review.openstack.org/20450408:53
AJaegertmorin: Correct. Our CI (btw. we use Jenkins and Zuul and..., not hudson) will update bugs only at two times:08:53
AJaegerWhen the first patch is send and when a patch merges08:53
AJaegerSo once your change is in, all patches that merge will become "Resolved"08:54
AJaegerBut Progress is only done for the very first time a change is submitted08:54
tmorinAjaeger: thanks for the info (mentioned hudson because saw the hudson-openstack username in LP comments)08:55
*** yamamoto has quit IRC08:57
openstackgerritJoshua Hesketh proposed openstack-infra/system-config: Gearman should listen on 0.0.0.0  https://review.openstack.org/20450508:57
*** hashar_ has joined #openstack-infra08:57
*** maishsk has joined #openstack-infra08:58
*** hashar has quit IRC08:58
*** maishsk has quit IRC08:59
*** shashank_hegde has quit IRC09:00
*** zeih has quit IRC09:01
AJaegerjhesketh: thanks for all your debugging. I have to sign-off now...09:01
*** AJaeger has quit IRC09:01
*** dguitarbite has quit IRC09:02
*** glauco has joined #openstack-infra09:03
*** zeih has joined #openstack-infra09:05
*** yfried has joined #openstack-infra09:05
*** e0ne has joined #openstack-infra09:07
*** glauco has quit IRC09:07
*** abregman|afk has quit IRC09:08
*** scheuran has quit IRC09:08
*** abregman has joined #openstack-infra09:08
*** gus has quit IRC09:10
*** bexelbie has joined #openstack-infra09:11
*** maishsk has joined #openstack-infra09:12
*** maishsk has quit IRC09:13
*** s0nea has quit IRC09:14
*** ihrachyshka has quit IRC09:15
*** ihrachyshka_ is now known as ihrachyshka09:15
*** s0nea has joined #openstack-infra09:15
*** dims_ has joined #openstack-infra09:15
*** maishsk has joined #openstack-infra09:17
openstackgerritgreghaynes proposed openstack/diskimage-builder: Document what our stable interfaces are  https://review.openstack.org/19881409:19
*** HeOS has joined #openstack-infra09:19
*** abregman is now known as abregman|hulk_mo09:20
*** dims_ has quit IRC09:20
*** dims_ has joined #openstack-infra09:23
*** scheuran has joined #openstack-infra09:23
*** dguitarbite has joined #openstack-infra09:24
mescanefhi, please review changes in project-config : https://review.openstack.org/#/c/203582/ and https://review.openstack.org/#/c/204057/09:24
*** hichihara has quit IRC09:24
mescanef:-)09:25
*** puranamr has joined #openstack-infra09:25
*** dims_ has quit IRC09:27
*** abregman|hulk_mo is now known as abregman|afk09:28
openstackgerritgreghaynes proposed openstack/diskimage-builder: Fail functests if refusing to run tests  https://review.openstack.org/17700209:28
*** skolekonov has joined #openstack-infra09:28
*** gaurangt1 has quit IRC09:28
*** coolsvap|away is now known as coolsvap09:28
*** andrey-mp has quit IRC09:29
*** e0ne has quit IRC09:29
*** jasondotstar has joined #openstack-infra09:30
*** puranamr has quit IRC09:30
*** tmorin has quit IRC09:30
*** fhubik is now known as fhubik_afk09:32
openstackgerritgreghaynes proposed openstack/diskimage-builder: Support building ACIs  https://review.openstack.org/15548409:32
*** jasondotstar has quit IRC09:35
*** mattymo_ has joined #openstack-infra09:36
*** dizquierdo has joined #openstack-infra09:37
*** zeih has quit IRC09:43
openstackgerritgreghaynes proposed openstack-infra/glean: Dont fail is openssh is not installed  https://review.openstack.org/20452209:43
*** fhubik_afk is now known as fhubik09:45
*** pblaho has joined #openstack-infra09:46
*** pblaho has quit IRC09:47
*** dims_ has joined #openstack-infra09:47
*** pblaho has joined #openstack-infra09:47
*** pblaho has quit IRC09:47
*** Guest83740 has quit IRC09:48
*** hashar_ is now known as hashar09:48
*** bexelbie has quit IRC09:49
*** andersonvom has quit IRC09:50
openstackgerritDoron Chen proposed openstack-infra/project-config: adding a new stackforge/storlets project  https://review.openstack.org/20341809:51
*** ricky1 has joined #openstack-infra09:51
ricky1o-k09:51
ricky1'nuff said with the topic09:52
*** rediskin has joined #openstack-infra09:52
*** pblaho has joined #openstack-infra09:55
*** _nadya_ has quit IRC09:55
*** pblaho has quit IRC09:56
openstackgerritgreghaynes proposed openstack-infra/project-config: Create jobs for a wheel mirror  https://review.openstack.org/16492709:56
*** gus has joined #openstack-infra09:57
openstackgerritgreghaynes proposed openstack-infra/system-config: Add wheel_mirror_slaves that use pypi-mirror  https://review.openstack.org/16524009:57
*** _kiran_ has joined #openstack-infra09:59
_kiran_Isn't the jenkins fixed yet??10:00
*** bexelbie has joined #openstack-infra10:02
*** dtantsur|brb is now known as dtantsur10:07
kozhukalovajaeger, pleia2, others could you please add me into the group fuel-upgrade-core, I am an author of this patch https://review.openstack.org/#/c/202541/310:09
*** hashar_ has joined #openstack-infra10:14
*** kmartin has quit IRC10:16
*** coolsvap is now known as coolsvap|away10:16
*** hashar has quit IRC10:17
*** e0ne has joined #openstack-infra10:17
*** vlaza has quit IRC10:17
*** dims_ has quit IRC10:18
*** yfried is now known as yfried|afk10:18
*** abregman|afk has quit IRC10:19
*** DericHorn-HP has joined #openstack-infra10:20
*** hashar has joined #openstack-infra10:21
*** fhubik has quit IRC10:23
*** fhubik has joined #openstack-infra10:23
*** hashar_ has quit IRC10:24
*** pelix has joined #openstack-infra10:26
*** fhubik is now known as fhubik_afk10:26
*** cody-somerville has quit IRC10:29
*** ricky1 has quit IRC10:31
*** jyuso1 has joined #openstack-infra10:31
*** vlaza has joined #openstack-infra10:35
*** dims_ has joined #openstack-infra10:36
*** fhubik_afk is now known as fhubik10:37
*** rossella_s has quit IRC10:37
*** rossella_s has joined #openstack-infra10:37
*** yfried|afk is now known as yfried10:38
*** dulek has joined #openstack-infra10:40
*** dims_ has quit IRC10:41
*** dims_ has joined #openstack-infra10:41
*** davideagnello has joined #openstack-infra10:43
*** flepied has joined #openstack-infra10:46
*** jasondotstar has joined #openstack-infra10:48
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/system-config: Add fuel-library channel  https://review.openstack.org/20031010:49
*** glauco_ has joined #openstack-infra10:50
*** _kiran_ has quit IRC10:54
*** cdent has joined #openstack-infra10:57
*** glauco_ has quit IRC11:00
*** tmorin has joined #openstack-infra11:01
*** notnownikki has joined #openstack-infra11:02
openstackgerritgreghaynes proposed openstack/diskimage-builder: Fix test cleanup trap to cleanup tmpdir  https://review.openstack.org/20455011:02
*** davideagnello has quit IRC11:04
*** gildub has joined #openstack-infra11:04
*** tmorin has quit IRC11:05
*** mwagner_lap has quit IRC11:10
*** fhubik is now known as fhubik_afk11:12
*** MarkAtwood has joined #openstack-infra11:14
*** flepied has quit IRC11:16
*** eharney has quit IRC11:18
*** zeih has joined #openstack-infra11:18
*** bradjones has quit IRC11:18
*** glauco_ has joined #openstack-infra11:19
*** MarkAtwood has quit IRC11:19
*** flepied has joined #openstack-infra11:19
*** bradjones has joined #openstack-infra11:20
*** bradjones has quit IRC11:20
*** bradjones has joined #openstack-infra11:20
*** bexelbie has quit IRC11:20
*** koolhead17 has joined #openstack-infra11:22
*** fhubik_afk is now known as fhubik11:23
*** pal has quit IRC11:24
*** hashar has quit IRC11:25
*** gildub has quit IRC11:27
*** zeih has quit IRC11:28
*** julim has joined #openstack-infra11:30
*** DericHorn-HP has quit IRC11:30
*** andrey-mp has joined #openstack-infra11:30
*** k4n0 has quit IRC11:31
*** yfried has quit IRC11:32
odyssey4mejhesketh aren't you meant to be on holiday?11:33
*** AJaeger has joined #openstack-infra11:33
jheskethodyssey4me: that finished yesterday11:34
odyssey4mejhesketh ah - I take it that gearman is still broken? we have one check executing, but another showing as NOT_REGISTERED11:35
jheskethso I found the problem with gearman, but now jobs aren't re-registering and I'm stuck again :-(11:36
*** shardy_ has joined #openstack-infra11:36
odyssey4me:( is there anything I can do to help?11:37
*** shardy has quit IRC11:37
jheskethodyssey4me: I appreciate the offer, not entirely sure though sorry..11:38
jheskeththe missing jobs are registered with jenkins, and appear in gearman's status too11:39
* jhesketh is probably missing something obvious11:39
*** mpaolino has quit IRC11:39
*** yfried has joined #openstack-infra11:41
*** shardy_ has quit IRC11:41
*** shardy has joined #openstack-infra11:42
*** hashar has joined #openstack-infra11:43
*** gildub has joined #openstack-infra11:44
*** mgershen has joined #openstack-infra11:44
*** PpPyg has quit IRC11:44
mgershenI'm getting -1 form jenkins and a message of "NOT_REGISTERED", what should i do? https://review.openstack.org/#/c/201973/11:45
*** zeih has joined #openstack-infra11:46
AJaegermgershen: wait until everything is fixed, see topic11:46
AJaegermgershen: Once our CI infrastructure is fine, add a "recheck" comment11:46
mgershenI'm a newbie, so I didn't see the connection. thank you :)11:46
*** shardy_ has joined #openstack-infra11:48
*** shardy has quit IRC11:48
*** eharney has joined #openstack-infra11:49
*** eharney is now known as Guest738411:50
*** abregman has joined #openstack-infra11:52
*** david-lyle has joined #openstack-infra11:52
*** abregman_ has joined #openstack-infra11:53
*** shardy_ has quit IRC11:53
*** pblaho has joined #openstack-infra11:53
*** BharatK has quit IRC11:53
*** pblaho has quit IRC11:54
*** shardy has joined #openstack-infra11:54
*** abregman_ has quit IRC11:54
*** pblaho has joined #openstack-infra11:54
*** abregman_ has joined #openstack-infra11:54
*** abregman_ has quit IRC11:55
*** abregman_ has joined #openstack-infra11:55
*** erlon has joined #openstack-infra11:55
*** ayoung has quit IRC11:55
*** jaypipes has joined #openstack-infra11:55
*** ldnunes has joined #openstack-infra11:57
*** flepied has quit IRC11:58
*** dprince has joined #openstack-infra11:59
*** mpavone has joined #openstack-infra12:01
*** mpavone has quit IRC12:01
*** glauco_ has quit IRC12:03
*** Guest7384 is now known as eharney12:03
*** eharney has quit IRC12:03
*** eharney has joined #openstack-infra12:03
*** BharatK has joined #openstack-infra12:06
*** bexelbie has joined #openstack-infra12:07
*** zeih has quit IRC12:08
*** tlian has joined #openstack-infra12:08
*** mpaolino has joined #openstack-infra12:08
*** koolhead17 has quit IRC12:09
*** mrmartin has quit IRC12:10
*** doude has quit IRC12:11
*** doude has joined #openstack-infra12:11
*** hashar_ has joined #openstack-infra12:11
*** hashar has quit IRC12:13
marekdHello, I have a question: When Jenkins runs gate jobs - does it always run on a fresh setup on newly created VM or something (VM?, devstack deployment) is reused?12:14
AJaegermarekd: Each job is in a complete new VM12:15
AJaeger(there are only a few exceptions)12:15
AJaegerexceptions are jobs for creation of tarballs or uploading code to translation server12:16
*** BharatK has quit IRC12:16
AJaegermarekd: Does that help?12:16
marekdAJaeger: ok thanks. I was asking wrt functional tests and whether we should really care about cleaning after ourselves. But since VM will be killed either way, I guess we can leave 'polluted' backends.12:16
marekdAJaeger: yes, i think it helps, thanks!12:16
*** BharatK has joined #openstack-infra12:16
*** andersonvom has joined #openstack-infra12:17
AJaegermarekd: consider that you as a developer might want to run a functional test. Clean up in for that case12:17
*** zeih has joined #openstack-infra12:18
*** Hal has joined #openstack-infra12:18
jheskethhmm, looks like it's just happening to jobs configured to run on bare-precise nodes12:19
jheskethbut I can't see anything wrong with those nodes12:19
*** Hal is now known as Guest422512:19
*** kgiusti has joined #openstack-infra12:19
*** fhubik is now known as fhubik_afk12:20
*** bswartz has quit IRC12:21
*** shardy_ has joined #openstack-infra12:21
*** shardy has quit IRC12:22
samueldmqgood morning12:22
samueldmqI have a change and Jenkins checks there are stated as 'NOT_REGISTERED'12:22
samueldmqhttps://review.openstack.org/#/c/197980/12:22
samueldmqis there any known issue ?12:22
jheskethsamueldmq: yes, CI is currently broken12:22
*** fhubik_afk is now known as fhubik12:22
AJaegersamueldmq: see topic12:23
*** Guest4225 has quit IRC12:23
samueldmqjhesketh: k thanks12:23
samueldmqAJaeger: yes, sorry I haven't noticed12:24
*** dims_ has quit IRC12:24
*** dguerri is now known as dguerri`12:25
*** shardy_ has quit IRC12:26
*** shardy has joined #openstack-infra12:27
*** pblaho has quit IRC12:27
*** mfalatic has joined #openstack-infra12:27
*** pblaho has joined #openstack-infra12:28
*** pblaho has quit IRC12:28
openstackgerritEric Harney proposed openstack-infra/project-config: Move Cinder LIO from experimental to non-voting  https://review.openstack.org/20415812:29
*** mpaolino has quit IRC12:29
openstackgerritlitong01 proposed openstack-infra/project-config: Add new monitor repository for OpenStack monitoring service  https://review.openstack.org/20325912:30
*** tongli has joined #openstack-infra12:30
*** yamamoto has joined #openstack-infra12:31
*** pblaho has joined #openstack-infra12:31
*** pblaho has quit IRC12:31
*** shardy has quit IRC12:31
*** shardy has joined #openstack-infra12:32
*** zeih has quit IRC12:32
*** deepakcs has quit IRC12:32
*** mwagner_lap has joined #openstack-infra12:33
*** mpaolino has joined #openstack-infra12:34
*** pblaho has joined #openstack-infra12:35
*** gordc has joined #openstack-infra12:37
*** pblaho has quit IRC12:38
*** dkranz has joined #openstack-infra12:38
*** bexelbie has quit IRC12:40
*** heyongli has quit IRC12:40
*** pblaho has joined #openstack-infra12:41
*** heyongli has joined #openstack-infra12:42
*** prad has joined #openstack-infra12:42
openstackgerritZara proposed openstack-infra/storyboard-webclient: Add 'Remove this story' button to detail page  https://review.openstack.org/20458712:43
*** rfolco has joined #openstack-infra12:43
*** DericHorn-HP has joined #openstack-infra12:44
*** pal has joined #openstack-infra12:48
*** xyang1 has joined #openstack-infra12:48
*** jsavak has joined #openstack-infra12:49
*** thorst has joined #openstack-infra12:50
*** bcourt has joined #openstack-infra12:51
*** ParsectiX has quit IRC12:51
*** dguerri` is now known as dguerri12:51
*** zeih has joined #openstack-infra12:51
*** pblaho has quit IRC12:52
DavieyUhm, is something wrong with zuul?  Seeing NOT_REGISTERED on Jenkins jobs on https://review.openstack.org/#/c/204492/ .. and the Zuul queue looks really short.12:53
mordredDaviey: see channel topic12:54
*** pblaho has joined #openstack-infra12:54
Davieymordred: Schoolboy error.  Thanks12:54
*** bexelbie has joined #openstack-infra12:54
mordredDaviey: sho nuff!12:54
*** abregman_ has quit IRC12:55
*** abregman_ has joined #openstack-infra12:55
DavieyReading scrollback, i blush at this being question of the day.12:55
*** yamamoto has quit IRC12:55
*** annegentle has joined #openstack-infra12:56
*** unicell has joined #openstack-infra12:57
*** iurygregory has joined #openstack-infra12:57
jheskethmordred: oh heya, you're around12:57
jheskethmordred: have you got time to help look?12:57
*** unicell1 has quit IRC12:58
mordredjhesketh: I'm not really here (vacation) ... but I think those sorts of errors are usually due to something in jjb borking12:58
*** andrey-mp has quit IRC12:58
mordredjhesketh: I was just looking through project-config patches to see if anything landed recently12:58
jheskethmordred: okay, thanks for the hint12:58
mordredjhesketh: any sense as to whetheror not there are bare-precise nodes in existence?12:58
*** radez_g0n3 is now known as radez12:59
jheskethmordred: only the min, more aren't being spun up12:59
mordredjhesketh: you may also want to try restarting a jenkins master and seeing if it'll re-register the jobs it has12:59
jheskethie one12:59
jheskethmordred: already tried that12:59
*** yamamoto has joined #openstack-infra12:59
openstackgerritLimor Stotland proposed openstack-infra/project-config: Add puppet module for Mistral  https://review.openstack.org/20252712:59
*** bknudson has joined #openstack-infra12:59
mordredjhesketh: bong12:59
*** ddieterl_ has quit IRC12:59
mordredjhesketh: and it's just bare-precise? or just for jobs taht have bare-precise mentioned anywhere?13:00
jheskethmordred: looking at the nodes that jobs are registered for on gearman it appears to only be bare-precise13:00
jheskethbut it could be a red herring13:01
mordrednod13:01
* greghaynes is curious if theres more context than "gearman is broken"13:01
greghaynesdidnt see much in scrollback13:02
jheskethgreghaynes: earlier gearman was broken when I restarted zuul and a recent change to gear made gearman bind to a single host13:02
greghaynesah13:02
jheskethgreghaynes: my fixes for that are here https://review.openstack.org/#/c/204504/ https://review.openstack.org/#/c/204505/13:02
greghayneshah13:02
*** eharney has quit IRC13:02
jheskethin the recovery from the breakage not all jobs are registering with zuul13:02
mordredjhesketh: I think I don't have much more ideas in my vacation head - hopefully fungi will awaken soon13:03
AJaegerjhesketh: it started with Zuul having internal data structures broken, then gearman. Was their any cloud outage that hit us somehow?13:03
jheskethand I haven't gotten any further information to do a new status yet13:03
jheskethmordred: no trouble, thanks anyway! Go enjoy your time off :-)13:03
*** mgershen has quit IRC13:03
greghaynesjhesketh: ah, so those fixes got applied manually?13:03
*** MarkAtwood has joined #openstack-infra13:03
jheskethAJaeger: not that I can see... I suspect that was due to a race in reloading zuul but my intention was to unwedge the gate first :-(13:03
jheskethgreghaynes: yes13:04
openstackgerritMonty Taylor proposed openstack-infra/puppet-zuul: Listen to all addresses by default  https://review.openstack.org/20450413:04
AJaegerjhesketh: I'll send an email to the mailing list for those not reading IRC topics ;)13:04
jheskethAJaeger: thanks, appreciated13:04
openstackgerritMonty Taylor proposed openstack-infra/puppet-zuul: Listen to all addresses by default  https://review.openstack.org/20450413:04
mordredjhesketh: (those are just commit message fixes)13:04
jheskethmordred: okay thanks13:04
*** fhubik is now known as fhubik_afk13:04
*** glauco_ has joined #openstack-infra13:05
*** flepied has joined #openstack-infra13:05
*** glauco_ has quit IRC13:05
*** glauco_ has joined #openstack-infra13:06
openstackgerritImre Farkas proposed openstack-infra/project-config: Add openstack/python-dracclient project  https://review.openstack.org/20399113:06
*** dkranz has quit IRC13:06
*** glauco_ has quit IRC13:06
*** Hal has joined #openstack-infra13:07
*** glauco_ has joined #openstack-infra13:07
*** glauco_ has quit IRC13:07
*** Hal is now known as Guest378713:07
*** _nadya_ has joined #openstack-infra13:07
*** romcheg has joined #openstack-infra13:07
*** gildub has quit IRC13:08
*** koolhead17 has joined #openstack-infra13:08
*** MarkAtwood has quit IRC13:08
*** glauco has joined #openstack-infra13:09
*** skolekonov has quit IRC13:13
*** rlandy has joined #openstack-infra13:13
*** ParsectiX has joined #openstack-infra13:14
*** eharney has joined #openstack-infra13:15
*** eharney is now known as Guest5724113:15
*** funzo has joined #openstack-infra13:16
*** baoli has joined #openstack-infra13:18
*** dkranz has joined #openstack-infra13:19
*** DericHorn-HP has quit IRC13:19
*** baoli has quit IRC13:21
*** Guest57241 is now known as eharney13:21
*** eharney has quit IRC13:21
*** eharney has joined #openstack-infra13:21
*** amotoki has quit IRC13:25
*** ParsectiX has quit IRC13:26
*** david-lyle has quit IRC13:27
*** puranamr has joined #openstack-infra13:27
pleia2kozhukalov: I have added the account associated with vkozhukalov@mirantis.com to the fuel-upgrade-core group13:28
*** peristeri has joined #openstack-infra13:28
*** mriedem has joined #openstack-infra13:29
*** koolhead17 has quit IRC13:29
*** yolanda has joined #openstack-infra13:29
*** gildub has joined #openstack-infra13:29
jheskethpleia2: oh hey, would you be able to help debug the gate per-chance?13:30
*** mjturek1 has joined #openstack-infra13:31
*** puranamr has quit IRC13:32
AJaegermorning, pleia2 !13:32
greghaynesclarkb: Hey, so im playing with getting dib images booting with qemu in a test, curious wat you think about for networking13:33
annegentleAJaeger: so it's CI down just for doc builds?13:33
annegentleAJaeger: anything I can do to help?13:33
*** abregman_ has quit IRC13:34
*** abregman has quit IRC13:34
jheskethannegentle: it's down across the board sorry13:34
annegentlejhesketh: bummer13:34
AJaegerannegentle: down everywhere;)13:35
annegentleI can't help much debugging that :)13:35
*** gildub has quit IRC13:35
AJaegerannegentle: wake up fungi or jeblair ;)13:35
annegentleheh13:35
*** abregman has joined #openstack-infra13:35
odyssey4mejhesketh oddly, I just saw one of the previously 'NOT_REGISTERED' jobs run properly13:35
*** vlaza has quit IRC13:36
jheskethodyssey4me: oh, do you have a link?13:36
odyssey4mejhesketh https://review.openstack.org/20459713:36
AJaegerjhesketh: there are nova jobs now with registered jobs!13:36
odyssey4mejhesketh https://jenkins07.openstack.org/job/os-ansible-deployment-check-lint/489/ was previously unregistered13:36
AJaegerseems we have a self-healing cloud ;)13:37
jheskethyeah, looks like there are a few that have come up now13:37
AJaegerjhesketh: check the bottom of the check queue on status.o.o/zuul13:37
annegentlewoo13:37
AJaegerpleia2: what have you done? ;)13:37
AJaegeror was it the friendly presence of annegentle ? ;)13:38
greghayneshuh13:38
annegentleheh13:38
annegentledoubtful!13:38
greghaynesthats... odd13:38
*** heyongli has quit IRC13:40
*** jaypipes has quit IRC13:40
samueldmqso CI can repair itself .. looks like some advanced AI approach being used :)13:40
*** heyongli has joined #openstack-infra13:41
*** markus_z has joined #openstack-infra13:41
greghaynesI wonder if there was just some network connectivity issues on the gearman server?13:41
annegentlelook out it'll be self aware on J-Day13:41
AJaegersamueldmq: CI breaks itself, so it should heal itself as well ;)13:41
samueldmqAJaeger: yeah that's pretty fair :)13:42
fricklerjhesketh: did you do anything at 13:05UTC? seems most jobs that are younger than that are working13:42
*** aysyd has joined #openstack-infra13:42
*** ddieterly has joined #openstack-infra13:42
jheskethfrickler: hmm, I was poking at the precise nodes to see why they weren't registering13:43
jheskethI cycled a few of them through13:43
jheskethmaybe that kicked things off13:43
jheskethbut I couldn't say way13:43
jhesketh*why13:43
openstackgerritwill soula proposed openstack-infra/jenkins-job-builder: Add support for whitesource plugin  https://review.openstack.org/20461513:44
*** jcoufal has quit IRC13:45
AJaegerjhesketh, can you now requeue the jobs from this morning - or is that too late now?13:45
AJaegerOr is it too early to put more load on?13:45
greghaynesThe queue is huge13:45
greghaynesso... itll be a while13:45
*** BharatK has quit IRC13:45
jheskethyeah so I have the previous queues snapshotted, but want to make sure it's actually fixed before I place them back in13:46
fricklerjhesketh: maybe the jenkins masters needed the trigger of a new slave in order to refresh the gearman connection13:46
AJaegerjhesketh: understood13:46
jheskethand then the gate is going to take a while to catch up13:46
*** hashar_ has quit IRC13:47
jheskethfrickler: maybe, although I had tried that earlier, so not sure what I did differently13:47
*** ddieterly has quit IRC13:47
*** kushal has quit IRC13:48
*** dims_ has joined #openstack-infra13:48
mriedemare we ok to recheck?13:48
jheskethmriedem: probably13:48
*** tvanderwiel has joined #openstack-infra13:49
*** ParsectiX has joined #openstack-infra13:50
*** signed8bit has joined #openstack-infra13:52
*** mrunge has quit IRC13:52
*** vlaza has joined #openstack-infra13:52
mriedemok cool13:55
*** zz_dimtruck is now known as dimtruck13:55
*** pblaho has quit IRC13:56
*** sabeen has quit IRC13:56
*** fhubik_afk is now known as fhubik13:57
*** evgenyl has left #openstack-infra13:57
*** baoli has joined #openstack-infra13:57
*** pblaho has joined #openstack-infra13:58
*** bnemec has joined #openstack-infra13:58
*** DericHorn-HP has joined #openstack-infra13:58
*** mrmartin has joined #openstack-infra13:58
andersonvomhey folks, I'm trying to "run my own" (http://docs.openstack.org/infra/system-config/running-your-own.html) but I keep getting missing dependencies while installing puppet master (http://docs.openstack.org/infra/system-config/puppet.html#id2)13:59
*** sherif has joined #openstack-infra13:59
andersonvomis there any particular distro that is better suited to follow these instructions?13:59
openstackgerritTimothy R. Chavez proposed openstack-infra/puppet-cgit: Add better defaults for clone-prefix  https://review.openstack.org/20424514:00
*** dustins has joined #openstack-infra14:01
AJaegerjhesketh: do you want to answer my email message once you're happy? Please also update the topic14:02
*** claudiub has joined #openstack-infra14:03
*** btully has joined #openstack-infra14:03
pleia2jhesketh, AJaeger - sorry, stepped out to breakfast right after fulfilling kozhukalov's request, but glad to see it fixed itself!14:05
openstackgerritPaul Belanger proposed openstack-infra/puppet-exim: Add Gemfile and puppet 4 checks  https://review.openstack.org/20424314:05
jheskethAJaeger: will do, just waiting for the green to reach the top of the queue14:05
AJaegerthanks, jhesketh !14:06
*** bknudson has quit IRC14:06
*** tsg has joined #openstack-infra14:07
*** hashar has joined #openstack-infra14:07
bauzasheya, zuul still blocked or can we recheck ?14:07
*** bswartz has joined #openstack-infra14:07
*** jaypipes has joined #openstack-infra14:07
openstackgerritMerged openstack-infra/project-config: Rearrange the tempest jobs under right section  https://review.openstack.org/20431414:08
*** palendae has joined #openstack-infra14:08
palendaeHello - we have a review that's been +W since Friday, but doesn't appear to have shown up in Zuul's queue. It's https://review.openstack.org/#/c/202258/, and to my knowledge this change skips our defined gating14:09
*** amitgandhinz has joined #openstack-infra14:09
openstackgerritMatthew Treinish proposed openstack-infra/irc-meetings: Change the alternate QA meeting time to 9:00 UTC  https://review.openstack.org/20462814:10
jheskethbauzas: should be okay to recheck14:10
openstackgerritPaul Belanger proposed openstack-infra/puppet-etherpad_lite: Add Gemfile and puppet 4 checks  https://review.openstack.org/20424214:10
bauzasjhesketh: ack, cool, thanks for the update14:10
jheskethpalendae: since Friday is a bit odd, probably okay to do a recheck though14:10
fricklerpalendae: zuul was restarted earlier today, probably need to remove the +W and readd it14:10
jheskethpalendae: it won't be in zuul's queue anymore anyway since it was restarted14:11
palendaefrickler, jhesketh: When was it restarted? I show a new +W added at 6:25AM EDT today14:11
pleia2jhesketh: care to update status?14:11
AJaegerpalendae: afterwards14:11
*** mestery has joined #openstack-infra14:11
jheskethpleia2: yeah I'll put out another alert and then do an okay once green is at the top14:12
palendaeAJaeger: Ok, thanks14:12
pleia2jhesketh: great, thanks14:12
AJaegerpalendae: so, toggle the +A. And wait for a few hours14:12
palendaeAJaeger: Thanks, will do!14:12
*** jcoufal has joined #openstack-infra14:12
jhesketh#status alert CI is currently recovering from an outage overnight. It is safe to recheck results with NOT_REGISTERED errors. It may take some time for zuul to work through the backlog.14:12
openstackstatusjhesketh: sending alert14:12
mtreinishtonyb: want to review https://review.openstack.org/#/c/204628/14:13
mtreinishI assume you can do something on that patch14:13
*** sabeen has joined #openstack-infra14:14
*** bknudson has joined #openstack-infra14:14
-openstackstatus- NOTICE: CI is currently recovering from an outage overnight. It is safe to recheck results with NOT_REGISTERED errors. It may take some time for zuul to work through the backlog.14:15
*** ChanServ changes topic to "CI is currently recovering from an outage overnight. It is safe to recheck results with NOT_REGISTERED errors. It may take some time for zuul to work through the backlog."14:15
openstackgerritPaul Belanger proposed openstack-infra/puppet-dashboard: Add Gemfile and puppet 4 checks  https://review.openstack.org/20424114:15
AJaegerpalendae: not recheck - ask to add another +A or toggle the +A14:16
*** pblaho_ has joined #openstack-infra14:16
*** boris-42 has joined #openstack-infra14:16
*** ParsectiX has quit IRC14:16
palendaeAJaeger: Asked, added my own +A14:16
AJaegerpalendae: ok, then let's wait - just saw the "recheck" comment by someboy14:16
palendaeThat was me14:16
leseb_hi all, https://review.openstack.org/#/c/188768/ :) it'd be nice if someone could merge this, thanks in advance14:17
AJaegers/someboy/somebody/14:17
palendae:)14:17
openstackstatusjhesketh: finished sending alert14:17
leseb_AJaeger: I don't really know who to ping for this but I'm sure you do, some if you have a minute :), thanks!14:17
*** pblaho has quit IRC14:17
openstackgerritPaul Belanger proposed openstack-infra/puppet-zuul: Update from puppet-modules-sync  https://review.openstack.org/20416314:18
boris-42AJaeger: hi14:18
*** glauco has quit IRC14:18
boris-42AJaeger: is infra going to change back topic of rally chat?14:18
boris-42AJaeger: * Topic for #openstack-rally set by openstackstatus  ...14:19
boris-42AJaeger: I mean we have quite important information in it and I believe it shouldn't be changed without any questions of rally team..14:19
*** sigmavirus24_awa is now known as sigmavirus2414:20
jheskethboris-42: normal topics will be restored once the alert has finished14:20
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: [WIP] Switch overcloud-f21-nonha to instack  https://review.openstack.org/18515114:21
*** mtanino has joined #openstack-infra14:21
boris-42jhesketh: great thanks14:21
AJaegerboris-42: the infra team changes the topic of all changes when it's important.14:22
AJaegerpleia2: could you review leseb_'s change, please?14:22
leseb_pleia2: https://review.openstack.org/#/c/188768/ :) thanks in advance14:23
pleia2sure14:23
openstackgerritMarek Denis proposed openstack/requirements: Constrain oauthlib versions  https://review.openstack.org/20463314:23
AJaegerpleia2, leseb_ : I just wonder whether this shouldn't be part of openstack namespace...14:24
pleia2AJaeger: do you know how the plugin: field works in projects.yaml? seems to be inconsistent usage among the devstack plugins14:25
*** shardy_ has joined #openstack-infra14:25
clarkbgreghaynes dhcp ?14:26
leseb_AJaeger: well russellb discussed that with dtroyer_zz and the outcome was to first push this here14:26
AJaegerpleia2: yes, there's inconsistency and I don't know what's best. Should we ask jeblair for help?14:26
leseb_perhaps russellb can elaborate if needed14:26
pleia2AJaeger: I'll make a note inline and maybe someone else can have a look14:27
*** shardy has quit IRC14:27
AJaegerleseb_: moving from stackforge to openstack is expensive, that's why we try to move it to openstack if possible.14:27
AJaegerpleia2: could you ping jeblair once he's awake? I'm going offline soonish...14:27
greghaynesclarkb: ohai. So, dhcp to where? The thing is - we dont need a full devstack so this is just one off run qemu-system-x86_6414:28
*** koolhead17 has joined #openstack-infra14:28
*** koolhead17 has quit IRC14:28
*** koolhead17 has joined #openstack-infra14:28
pleia2AJaeger: I'm off to the convention center soon, but I'll see what I can do14:28
*** zeih has quit IRC14:28
greghaynesclarkb: I am considering just add a br, slap some static network in the image, boot with -net tap14:28
clarkbgreghaynes I eould devstack just for that but elazy and using virsh is bad for my sanity14:28
greghaynesheh14:28
clarkbyou could just do the arp check14:29
AJaegerpleia2: understood - thanks. Enjoy oscon!14:29
clarkbthst dhcp from libvirt?14:29
*** yamamoto has quit IRC14:29
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: qa-team-meeting: Remove unneeded comment  https://review.openstack.org/20463614:29
pleia2AJaeger: thanks :) this morning is it for me, have a flight in 6 hours14:29
greghaynesclarkb: well, im not even running a dhcp at that point, I could add one...14:29
clarkbif you use virsh I think it dors it14:30
greghaynesah14:30
greghaynesI wish you had replied about 10 mins sooner because I might have just gotten this br deal working14:30
greghaynesvirsh might still be a good idea though14:30
*** shardy_ has quit IRC14:30
*** u_glide has left #openstack-infra14:31
jhesketh#status ok CI is slowly recovering, please be patient while the backlog is worked through.14:31
openstackstatusjhesketh: sending ok14:31
*** shardy has joined #openstack-infra14:31
*** sabeen3 has joined #openstack-infra14:32
*** yamamoto has joined #openstack-infra14:32
*** sabeen has quit IRC14:32
*** yolanda has quit IRC14:32
-openstackstatus- NOTICE: CI is slowly recovering, please be patient while the backlog is worked through.14:33
*** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/"14:33
openstackgerritgreghaynes proposed openstack/diskimage-builder: Perform a booting test for our images  https://review.openstack.org/20463914:35
greghaynesclarkb: ^14:35
openstackgerritMerged openstack-infra/irc-meetings: Change the alternate QA meeting time to 9:00 UTC  https://review.openstack.org/20462814:35
openstackstatusjhesketh: finished sending ok14:35
*** koolhead17 has quit IRC14:36
*** pal has quit IRC14:38
*** bexelbie has quit IRC14:38
AJaegerwow, 248 jobs in the check queue ;(14:39
Sam-I-Amchugga14:39
*** pcrews has quit IRC14:41
*** sherif has quit IRC14:41
elmikoooph14:42
*** pcrews has joined #openstack-infra14:42
AJaeger301...14:42
DavieyAJaeger: if it were running on some sort of cloud, we could scale out more workers to help with the temp higher demand .. :)14:42
AJaegerDaviey: do you have a spare cloud? WE use Rackspace and HP public clouds already...14:43
DavieyAJaeger: Sure, I have one under my desk. Help yourself :)14:44
AJaegerDaviey: just plug it in - send patches for project-config and system-config so that we can use it;)14:44
*** ayoung has joined #openstack-infra14:45
Daviey:)14:45
pleia2our infrastructure does support easy scaling across the cloud, but as AJaeger says, we don't have the resources to expand further on the clouds provided to the project14:45
Davieypleia2: Oh i know.. I am familiar with nodepool..14:46
AJaegerDaviey: see http://specs.openstack.org/openstack-infra/infra-specs/specs/infra-cloud.html for next steps - you should plugin your rack in that framework...14:47
jheskethAJaeger: yes, I'm re-enqueuing the jobs from earlier14:47
DavieyAJaeger: well that backfired... I didn't know about that spec14:47
*** baoli has quit IRC14:48
AJaegerjhesketh: Thanks! Now passed 400...14:48
pleia2Daviey: no more jokes from you!14:48
*** baoli_ has joined #openstack-infra14:49
Davieypleia2: Indeed.14:49
AJaegerDaviey: we have it on record now that you're going to help with that ;)14:49
pleia2:)14:49
*** dguitarbite has quit IRC14:49
pleia2hehe14:49
*** vlaza has left #openstack-infra14:49
*** davidjc has joined #openstack-infra14:51
*** signed8bit is now known as signed8bit_ZZZzz14:52
*** signed8bit_ZZZzz is now known as signed8bit14:53
*** tonytan4ever has joined #openstack-infra14:53
marekdlifeless: Hi. I added another comment on https://review.openstack.org/#/c/204633/ . So, keystoneclint fails on master version as oauthlib changes signatures of few of their functions. I doubt they will get back to the old signatures (or those that are compatible with ksc) in future versions, so I think we should keep <1.0.0 until we fix keystone.14:53
*** glauco has joined #openstack-infra14:55
DavieyAJaeger / pleia2: Hold me to it, and i'll provide you a cloud running on underclocked raspberry pi's :)14:55
elmikolol14:55
AJaegerDaviey: I heard that compute nodes with raspberry pi's are not that great since the CPU misses some virtualization support. So, you can use them as controllers...14:56
zarohashar: had a question for you on this, https://review.openstack.org/#/c/130166/14:57
*** maishsk has quit IRC14:58
*** AJaeger has quit IRC14:58
openstackgerritSean Dague proposed openstack-infra/devstack-gate: grenade partial multinode support  https://review.openstack.org/19909114:59
openstackgerritSean Dague proposed openstack-infra/devstack-gate: convert variables to local in setup_multinode_connectivity  https://review.openstack.org/20465514:59
*** andrey-mp has joined #openstack-infra14:59
*** marzif has quit IRC15:00
*** mfmcdonagh has quit IRC15:00
*** pelix has quit IRC15:01
*** patrickeast has joined #openstack-infra15:02
*** dizquierdo has quit IRC15:02
*** palendae has left #openstack-infra15:02
*** jsavak has quit IRC15:03
hasharzaro: good morning :)15:04
hasharzaro: ahh15:05
hasharzaro: so that is really a lame wrapper I added in my local copy to easily build the plugin15:05
*** pal has joined #openstack-infra15:07
jeblairjhesketh: thanks for dealing with the outage15:08
jeblairjhesketh, mordred: i think we should revert the zuul patch and use a different solution there -- i think zuul should have a configuration for what gearman server to connect to, and a separate config option for the ip address to listen on15:08
mordredjeblair: wfm15:08
*** spfifxp is now known as spiffxp15:09
*** jamesmcarthur has joined #openstack-infra15:09
*** koolhead17 has joined #openstack-infra15:09
*** david-lyle has joined #openstack-infra15:09
jeblair(mostly because making outgoing connections to 0.0.0.0 seems weird to me)15:10
openstackgerritMatt Riedemann proposed openstack/requirements: Bump minimum os-brick>=0.3.1  https://review.openstack.org/20319315:10
mriedemthingee: hemnafk: ^15:10
mriedemadds the depends-on to the release change15:10
hasharzaro: gave it a try.   git describe yields  0.1.1-1-g9fea5a0  , your script 0.1.1.1.9fea5a015:10
hasharzaro: seems good :)15:11
*** tsg has quit IRC15:11
mordredjeblair: the internet should be able to infer your intentions15:11
jheskethjeblair: no trouble15:11
jheskethjeblair: I agree15:12
*** dprince has quit IRC15:12
jeblairk, i'll write that patch15:12
jheskeththanks15:12
jheskethI'm going to get these last jobs back into the queue and then probably head to bed now things seem to be stabilizing15:13
openstackgerritAntoine Musso proposed openstack-infra/gearman-plugin: Script to easily build a package  https://review.openstack.org/13016615:13
*** mrmartin has quit IRC15:13
*** annegentle has quit IRC15:13
hasharzaro: done!15:13
*** annegentle has joined #openstack-infra15:14
*** vlaza has joined #openstack-infra15:14
*** nijaba has quit IRC15:16
*** mwagner_lap has quit IRC15:16
*** nijaba has joined #openstack-infra15:16
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: [WIP] Switch overcloud-f21-nonha to instack  https://review.openstack.org/18515115:18
*** tsekiyama has joined #openstack-infra15:19
*** glauco has quit IRC15:20
*** koolhead17 has quit IRC15:20
*** annegentle has quit IRC15:20
*** arxcruz has joined #openstack-infra15:21
jeblairthere are a lot of unused ready nodes, mostly on jenkins07, that probably missed being actually attached to jenkins; i'm deleting them15:22
*** kushal has joined #openstack-infra15:22
*** laughterwym has joined #openstack-infra15:23
openstackgerritgreghaynes proposed openstack/diskimage-builder: Perform a booting test for our images  https://review.openstack.org/20463915:24
zarohashar: great, thanks.15:24
*** glauco has joined #openstack-infra15:25
*** austin81 has joined #openstack-infra15:26
*** pelix has joined #openstack-infra15:26
*** sshnaidm has quit IRC15:27
*** doug-fish has joined #openstack-infra15:28
*** pblaho_ is now known as pblaho15:28
openstackgerritgreghaynes proposed openstack/diskimage-builder: Perform a booting test for our images  https://review.openstack.org/20463915:28
*** harlowja_at_home has joined #openstack-infra15:29
*** pcrews has quit IRC15:29
*** glauco has quit IRC15:30
*** sherif has joined #openstack-infra15:31
*** hashar is now known as hasharConfcall15:31
thingeeanteaya: ping15:32
anteayathingee: hi15:32
anteayathanks15:32
*** BharatK has joined #openstack-infra15:32
anteayaso you have some cis you would like disabled15:33
anteayalet's start there15:33
*** hemnafk is now known as hemna15:33
*** jsavak has joined #openstack-infra15:34
thingeeanteaya: yes, IBM XIV CI15:34
*** spredzy is now known as spredzy|afk15:34
thingeeanteaya: and Microsoft iSCSI CI15:34
*** baoli_ has quit IRC15:35
anteayagreat thanks15:36
anteayahave we reasons why?15:36
anteayayou asking is good enough for me but when we announce to the mailing list it is nice to have something to point to15:36
*** baoli has joined #openstack-infra15:36
*** abregman has quit IRC15:37
jeblairjenkins07 is not sending zmq events, so i have put it into prepare-for-shutdown mode and will restart it laster when jobs its running are finished15:37
*** armax has joined #openstack-infra15:37
anteayajeblair: thank you15:37
*** gsagie_ has joined #openstack-infra15:38
*** kiran-r has joined #openstack-infra15:38
*** dtantsur is now known as dtantsur|afk15:38
SwansonIs the lovely logs.openstack.org log viewer something that is available?  Or whatever it is that converts the logs into that html?15:39
*** andersonvom has quit IRC15:39
*** ashleighfarnham has joined #openstack-infra15:39
anteayaI have a problem, when I run curl -X GET 'https://review.openstack.org/accounts/alonma@il.ibm.com' to get the gerrit account id for IBM XIV CI I get not found15:39
*** sergmelikyan has quit IRC15:40
anteayanot sure who is around to look at the gerrit db to find and disable IBM XIV CI but my usual method of finding their gerrit id is failing15:40
jeblairanteaya: i'll give it a shot15:40
*** _nadya_ has quit IRC15:41
anteayajeblair: thanks15:41
jeblairanteaya: that email is associated with both 10213 and 1249315:41
*** _nadya_ has joined #openstack-infra15:42
zarohasharConfcall: i think we are deadlocked on this change since pelix and i both worked on it.  no other cores to move it along.  wondering if you could help out when you have a moment? https://review.openstack.org/#/c/151472/15:42
anteayajeblair: wonderful15:42
jeblairone of them looks like an old-style manually created account (12493) and the other one looks like a launchpad account15:43
anteayathe microsoft ci gerrit id is 13394, it that also could be disabled: http://paste.openstack.org/show/400440/15:43
anteayajeblair: hmmmm15:44
*** ihrachyshka has quit IRC15:44
*** chmouel has quit IRC15:44
*** baoli has quit IRC15:45
*** andersonvom has joined #openstack-infra15:45
*** jsavak has quit IRC15:46
*** _nadya_ has quit IRC15:46
*** jsavak has joined #openstack-infra15:46
*** baoli has joined #openstack-infra15:46
jheskethokay all the queues have been restored... not a fun backlog sorry :-(15:46
thingeeanteaya: oh sorry, I sent some emails to the third party list, but I can resend them if they got stuck for some reason.15:47
jeblairi think i should delete the email address from 12493 and then disable it15:47
anteayajhesketh: thanks for all your help15:47
anteayajeblair: I support that direction15:47
*** Guest23745 has joined #openstack-infra15:47
anteayathingee: yes please resend15:47
*** scheuran has quit IRC15:47
jheskethno trouble, that's why I'm here :-)15:47
jhesketh(I hope)15:47
*** Guest23745 has quit IRC15:47
anteayajhesketh: :)15:47
anteayajhesketh: so glad you're here15:47
jheskethanyway, see y'all in the morning15:47
anteayajhesketh: night15:48
jeblairjhesketh: thanks, and sleep well! :)15:48
*** achanda has joined #openstack-infra15:48
anteayathingee: they were sent the same day as you told me to ignore your post to third-party-announce, so I rejected them15:48
anteayathingee: I didn't realize there were some you wanted posted and some you wanted ignored, my fault15:49
jeblairflushing gerrit's caches15:49
anteayathingee: so please resend now that I understand better15:49
anteayajeblair: did you disable microsoft too?15:49
anteayagerrit id 13394?15:49
jeblairanteaya: not yet, still doing db repair15:49
anteayaah k, thank you15:49
*** tsg has joined #openstack-infra15:49
thingeeanteaya: done15:50
anteayathingee: thank you15:50
anteayaonce they are disabled I will approve them15:50
anteayaapprove the ml posts15:50
*** patrickeast has quit IRC15:50
*** abregman has joined #openstack-infra15:50
*** Sukhdev has joined #openstack-infra15:50
*** dimtruck is now known as zz_dimtruck15:51
*** chmouel has joined #openstack-infra15:51
odyssey4meoh dear, we have several reports of review.o.o croaking across different regions - is anyone else seeing this?15:51
*** DericHorn-HP has quit IRC15:51
anteayaodyssey4me: we just flushed the db caches15:51
jeblairodyssey4me: can you elaborate on 'croaking'?15:51
thingeeanteaya: ah that's right. yeah it's my fault. Someone told me they were already disabled, but I kept seeing them reporting so I'm requesting again for them to be disabled15:52
*** bswartz1 has joined #openstack-infra15:52
thingeeanteaya: thank you!15:52
odyssey4meit seems to have come back to life :)15:52
*** mpaolino has quit IRC15:52
*** agentle_ has joined #openstack-infra15:52
anteayawe had a small hiccup at the nova mid-cycle too, but is likely due to the gerrit db cache jeblair performed15:52
jeblairif you mean "was unresponsive within the last 3 minutes", yes, it's because we're doing a tiny bit of maintenance15:52
odyssey4mejeblair I tried to refresh and it gave a server error.15:52
*** achanda has quit IRC15:52
jeblairif it's something more than that, it's news to us15:52
odyssey4meah, makes sense - fair enough :)15:52
anteayathingee: glad we have reconnected again15:53
thingeethorst: no reason to talk privately. IBM XIV has been failing for sometime.15:53
thingeethorst: reps from IBM are unsure why. Until then, I don't want things reporting until that's figured out.15:53
anteayathingee: I suggested he talk to you in -cinder, he just got up for break here at nova mid-cycle15:53
anteayahe will be back in a minute15:53
*** bswartz has quit IRC15:54
thingeethorst: I'm also unsure why people are leaving something running to give false reports.15:54
anteayahe told me he will try on his end to impress upon the operators of this ci the responsiblity that is theirs15:54
*** shashank_hegde has joined #openstack-infra15:54
*** mgershen has joined #openstack-infra15:54
jeblairanteaya: 12493 and 13394 are now inactive15:54
anteayathingee: thorst seems to get it15:54
anteayajeblair: thank you!15:54
* anteaya goes to approve ml posts15:54
jeblairnow we seem to be over quota in dfw, looking into that15:55
thingeeanteaya: I heard that from Jay Bryant, Alon, etc...it has been weeks now15:55
*** patrickeast has joined #openstack-infra15:55
anteayathingee: yup, got it15:55
*** achanda has joined #openstack-infra15:55
*** romcheg has quit IRC15:55
anteayathingee: you have way more patience than I do, that is for sure15:56
*** coolsvap|away is now known as coolsvap15:56
jeblairit's because we have 50 'DELETED' instances in rax15:57
nibalizermorning yall15:57
*** hasharConfcall is now known as hashar15:57
anteayajeblair: fun15:57
anteayanibalizer: morning15:57
jeblairi'll open a ticket15:57
*** tonytan4ever has quit IRC15:58
nibalizerjeblair: I'd like to approve some more puppet-httpd changes, if you're up for being infra-root-on-hand15:58
*** jamesmcarthur has quit IRC15:58
*** MarkAtwood has joined #openstack-infra15:58
*** achanda has quit IRC15:58
jeblairnibalizer: sure thing, but only do ones that have existing verified +1s, the zuul backlog is huge and it'll probably be late today before we see any check results return15:59
pabelangernibalizer: I like your moxie15:59
*** SergK has quit IRC15:59
nibalizersure, thing15:59
thorstthingee: sorry - stepped away for a bit.  Didn't see this.  Didn't mean pinging privately to be odd.  I'm just trying to track down the owner to get awareness and the owner.16:00
openstackgerritMerged openstack-infra/elastic-recheck: Add query for neutron ipv6 ping failure bug 1477192  https://review.openstack.org/20465216:00
uvirtbotLaunchpad bug 1477192 in neutron "neutron test_multi_prefix_slaac failing in the gate with ping failures starting around 7/22" [High,Confirmed] https://launchpad.net/bugs/147719216:00
openstackbug 1477192 in neutron "neutron test_multi_prefix_slaac failing in the gate with ping failures starting around 7/22" [High,Confirmed] https://launchpad.net/bugs/147719216:00
uvirtbotLaunchpad bug 1477192 in neutron "neutron test_multi_prefix_slaac failing in the gate with ping failures starting around 7/22" [High,Confirmed]16:00
thingeethorst: ok16:00
*** jlanoux has quit IRC16:00
*** deepakcs has joined #openstack-infra16:00
thorstsorry about that!16:00
openstackgerritJames E. Blair proposed openstack-infra/zuul: Specify gearman server listen address with separate option  https://review.openstack.org/20468216:01
pabelangerpleia2: when you have some time, do you mind +2 on review 179208? That's to merge our grafana.o.o node16:01
nibalizerpabelanger: I think https://review.openstack.org/#/c/202618/ is irrelevant since http://git.openstack.org/cgit/openstack-infra/puppet-dashboard/tree/README.markdown#n316:02
*** SergK has joined #openstack-infra16:03
*** agentle_ has quit IRC16:03
*** mpaolino has joined #openstack-infra16:04
*** yamamoto has quit IRC16:06
*** yfried has quit IRC16:07
*** patrickeast has quit IRC16:07
*** patrickeast has joined #openstack-infra16:07
*** ivar-lazzaro has joined #openstack-infra16:07
*** wznoinsk has quit IRC16:07
*** erikmwilson_ has joined #openstack-infra16:07
nibalizerpabelanger: on https://review.openstack.org/#/c/200204/5/manifests/site.pp the other changes seem to be direct switchout not a check/switch behaviour, why is that one special?16:07
nibalizermaybe that class is applied to a node along with other classes with apache resources?16:08
jeblairokay, rax ticket opened about the 'DELETED' servers16:09
*** geoffarnold has joined #openstack-infra16:11
pabelangernibalizer: for 200204 because we have a circular dependency on system-config.16:11
pabelangeronce merged, we can remove the system-config include apache then update this16:12
*** gyee has joined #openstack-infra16:12
*** tangchen_ has quit IRC16:12
pabelangernibalizer: as for puppet-dashboard, we should move it into attic, if we are not adding commits16:12
pabelangerI think you have something upstream16:12
*** mwagner_lap has joined #openstack-infra16:13
*** geoffarnold has quit IRC16:13
nibalizerhttp://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/eavesdrop.pp#n1816:14
nibalizeryep right there16:14
*** mjturek1 has quit IRC16:14
jeblairpabelanger, nibalizer: except instead of moving it into the attic, we should merge a change to the readme saying the project is no longer maintained and change the status to read-only in gerrit16:14
*** fhubik is now known as fhubik_afk16:15
hasharzaro: sure. Will give it a shot tomorrow :-)16:16
pabelangerI am fine with that16:16
*** arxcruz has quit IRC16:16
hasharzaro: gotta figure out some Zuul layout madness on Wikimedia setup first :D16:16
*** geoffarnold has joined #openstack-infra16:16
*** wznoinsk has joined #openstack-infra16:16
nibalizerjeblair: readme change has already happened http://git.openstack.org/cgit/openstack-infra/puppet-dashboard/tree/README.markdown#n316:17
*** harlowja_at_home has quit IRC16:17
nibalizerill work on the readonly script16:18
*** changbl has quit IRC16:18
jeblairnibalizer: oh good.  if that's not enough, we can merge a change that removes all the content and leave _only_ the readme with that paragraph.16:18
jeblairi'm not sure we've established a preference for whether to do that or not.16:18
openstackgerritSpencer Krum proposed openstack-infra/project-config: Puppet-dashboard is read only now  https://review.openstack.org/20468516:19
*** k4n0 has joined #openstack-infra16:19
* fungi is glad to see the ci crazyness has been resolved successfully16:19
*** jistr has quit IRC16:19
*** ivar-lazzaro has quit IRC16:19
nibalizerjeblair: 204685 is the readonly patch16:20
*** warpc__ has joined #openstack-infra16:20
*** ivar-lazzaro has joined #openstack-infra16:20
*** Guest3787 has quit IRC16:20
jeblairso many 500 errors from hpcloud16:20
jeblairi'm running the port cleanup script, just in case, but i ran it yesterday too16:21
pabelangerremoving all the content and leaving readme, gets my vote.  That pretty much means project is dead IMO16:21
*** fhubik_afk is now known as fhubik16:21
*** jcoufal has quit IRC16:21
jeblairfungi: have a preference for something like http://git.openstack.org/cgit/openstack-infra/puppet-dashboard/tree/README.markdown#n3  or the stronger "rm *" and leave only a 1 sentence readme?16:22
*** esmute_ has joined #openstack-infra16:23
*** achanda has joined #openstack-infra16:24
*** dprince has joined #openstack-infra16:24
jeblairi'm looking for leaked fips on hpcloud too16:24
*** ivar-lazzaro has quit IRC16:24
*** unicell has quit IRC16:24
*** alexsyip has joined #openstack-infra16:25
fungiyeah, content removal ftw. it's not like someone can't git checkout at earlier state of the frozen repository16:25
*** mrmartin has joined #openstack-infra16:25
*** signed8bit is now known as signed8bit_ZZZzz16:25
jeblairnibalizer: ^ that work for you?16:25
*** amitgandhinz has quit IRC16:25
*** kiran-r has quit IRC16:26
jeblairnibalizer: (might be good to propose content-removal then have the read-only change depends-on that since i'm sure we won't be merging either of them today)16:26
fungithe only downside i can think of is if someone is continually deploying the branch tip, then we break them... but that also has the benefit of bringing its end-of-life state to their attention16:26
jeblairfungi: yeah, i think i would be both annoyed and appreciative if i were on the receiving end of that.  :)16:27
*** signed8bit_ZZZzz is now known as signed8bit16:27
*** pal has quit IRC16:28
jeblairdeleting what i hope are leaked floating ips now16:28
*** glauco_ has joined #openstack-infra16:30
anteayajeblair: yay16:31
*** ihrachyshka has joined #openstack-infra16:31
openstackgerritgreghaynes proposed openstack/diskimage-builder: Perform a booting test for our images  https://review.openstack.org/20463916:31
*** jsavak has quit IRC16:31
*** changbl has joined #openstack-infra16:32
sc68callifeless: http://paste.openstack.org/show/400488/16:33
sc68callifeless: the cookiecutter thing we just chatted about16:33
*** jsavak has joined #openstack-infra16:33
*** achanda has quit IRC16:34
*** DericHorn-HP has joined #openstack-infra16:34
*** k4n0 has quit IRC16:34
*** DericHorn-HP has quit IRC16:35
lifelesssc68cal: so yeah, old mock16:35
lifelesssc68cal: now to find the actual minimum version16:35
*** jungleboyj has joined #openstack-infra16:35
*** dannywilson has joined #openstack-infra16:36
lifelesssc68cal: six 1.6 includes wraps16:36
*** reed_ has joined #openstack-infra16:36
*** reed_ has quit IRC16:38
lifelesssc68cal: try patching the mock.py to have a try:/ImportError: import six; print(six.__file__)16:38
*** coolsvap is now known as coolsvap|away16:38
*** kiran-r has joined #openstack-infra16:38
*** sergmelikyan has joined #openstack-infra16:38
*** pblaho has quit IRC16:38
*** laughterwym has quit IRC16:39
andrey-mpHi all. Core, please review this https://review.openstack.org/#/c/202070/3 This review is adding one more experimental job, because we need it to move futher with our functional testing in stackforge/ec2-api project16:39
*** eharney_ has joined #openstack-infra16:39
jungleboyjanteaya: Word on the street is that you are looking for me.16:39
*** eharney has quit IRC16:39
*** eharney_ is now known as eharney16:40
*** eharney has quit IRC16:40
*** eharney has joined #openstack-infra16:40
anteayajungleboyj: less so than getting the ibm folks to take care of teh currently disabled ci16:40
*** vnod has joined #openstack-infra16:41
anteayajungleboyj: if you are able to do that, thingee would be happy16:41
jeblairjenkins07 is shut down; deleting nodes attached to it16:41
anteayajeblair: ack16:41
jungleboyjanteaya: Yes, I apologize for that.  I have been pinging on them for the last week.  Escalated the yesterday.16:41
anteayajungleboyj: understood16:41
jungleboyjanteaya: Actually longer than that.16:41
anteayajungleboyj: general tiredness with ibm lack of listening to you here16:41
jungleboyjanteaya: It is being actively worked at the moment.16:41
anteayajungleboyj: I know you get it, ibm should listen to you better16:41
*** yamahata has joined #openstack-infra16:41
*** hashar has quit IRC16:42
anteayajungleboyj: I hope they start listening to you16:42
jungleboyjanteaya: :-)  Yeah.  Working on that.16:42
anteayajungleboyj: you carry a good message16:42
anteayajungleboyj: let me know if I can help16:42
openstackgerritMerged openstack-infra/project-config: VPNaaS: Experimental grenade job  https://review.openstack.org/20411416:42
openstackgerritMerged openstack-infra/project-config: VPNaaS: Experimental grenade job  https://review.openstack.org/20411416:42
mescanefhi, please review changes in project-config : https://review.openstack.org/#/c/203582/ and https://review.openstack.org/#/c/204057/16:42
mescanefSergeyLukjanov: ^^ :-)16:42
mescanefor maybe fungi16:43
mescanef:-)16:43
*** sergmelikyan has quit IRC16:43
jeblairthe floating ip delete in hpcloud is complete16:43
nibalizerjeblair: fungi see conversation here: https://review.openstack.org/#/c/189591/ I'm fine with total removal16:43
jungleboyjanteaya: Ok, just was surprised when Andrew started pinging me about it.  I wasn't aware that it had been escalated so much in the community.16:44
anteayajeblair: yay16:44
anteayajungleboyj: he just happens to be here at the nova mid-cycle16:44
*** ivar-lazzaro has joined #openstack-infra16:44
anteayaif thorst is andrew16:44
anteayawe were talking last night about ci operators taking reponsibility for their work16:44
anteayait was a heartening exchange for me16:44
anteayawhen I saw an ibm ci was about to be disabled, I asked him if he knew anything about it16:45
anteayasince he is sitting beside me16:45
thorstthorst is andrew  ;-)16:45
anteaya:)16:45
jeblairnibalizer: *nod* so it's worth a conversation about that.  but i bet we can come up with some readme text that would make jhesketh happy -- maybe somethnig like "hey we don't maintain this, but if you want to pick it up, feel free to branch it from the previous commit" or something.16:45
*** ihrachyshka has quit IRC16:45
jungleboyjOh, I was going to mention you thorst but wasn't sure.  :-)16:46
anteayajungleboyj: :)16:46
*** mpaolino has quit IRC16:46
jungleboyjanteaya: Gotcha.  I need to come over and say hi since you are in town.16:46
*** sdake has joined #openstack-infra16:47
anteayajungleboyj: you are in rochester? yes please16:47
*** ivar-lazzaro has quit IRC16:47
jungleboyjYep ... That is home base for me.16:47
anteayagood16:47
*** ivar-lazzaro has joined #openstack-infra16:47
anteayajungleboyj: dinner tonight or tomorrow?16:47
*** sdake_ has joined #openstack-infra16:48
*** bharath_ has joined #openstack-infra16:48
*** MarkAtwood has quit IRC16:48
jungleboyjanteaya: Not sure I will be able to do dinner this week.  Have family commitments in the evenings.16:49
anteayajungleboyj: fair enough, time for a hug would be great though16:49
jungleboyjanteaya: Of course.  Will stop over there today.16:50
*** maishsk has joined #openstack-infra16:50
anteayajungleboyj: grand16:50
openstackgerritMerged openstack-infra/project-config: Remove non-voting gate jobs  https://review.openstack.org/20341016:50
*** jasondotstar has quit IRC16:50
anteayajungleboyj: https://wiki.openstack.org/wiki/Sprints/NovaLibertySprint#IBM_Specifics16:50
*** dahoo has joined #openstack-infra16:50
*** dahoo is now known as jerryz16:50
*** MarkAtwood has joined #openstack-infra16:51
jeblairjenkins07 appears to be operating normally now16:51
anteayaya16:52
anteayay!16:52
*** sdake has quit IRC16:52
*** kjnelson has joined #openstack-infra16:52
*** erikmwilson_ has quit IRC16:53
*** kiran-r has quit IRC16:54
*** spredzy|afk is now known as spredzy16:54
jeblairhpcloud-b1 and hpcloud-b2 nodepool managers are stuck16:55
*** jsavak has quit IRC16:56
*** pal has joined #openstack-infra16:56
*** fhubik has quit IRC16:57
*** sputnik13 has joined #openstack-infra16:57
jeblairstuck on an ssl read in requests/urllib316:57
*** unicell has joined #openstack-infra16:58
anteayathere was a new release of urllib3 yesterday16:58
jeblairhttp://paste.openstack.org/show/400649/16:58
jeblairit's the urllib3 bundled in requests 2.7.0 that's being used here16:59
nibalizercrinkle: on https://review.openstack.org/#/c/203868/2 i think we decided the default posix provider was sufficient ya?16:59
jeblairi'm going to resatart nodepool16:59
nibalizerI keep looking at that change and reading about selinux and trying the commands out17:00
clarkbI thought stuck on ssl was why we put the api timeouts in place17:00
nibalizeri'm becomming pretty confident that the change is good17:00
jeblairhrm, let me see where we are on dib builds17:00
*** madhu_ak has joined #openstack-infra17:00
*** soren has joined #openstack-infra17:00
clarkbare they set for those clouds?17:00
nibalizerand ofc using an actual selinux module instead of execs would be better, but one thing at a time17:00
crinklenibalizer: i think we both tested it and it seems sufficient, but after reading the documentation i don't know /why/ it's sufficient17:00
jeblairclarkb: i'm pretty sure they are, i'll double check17:00
crinklenibalizer: so i'm a bit nervous about it17:01
openstackgerritlitong01 proposed openstack-infra/project-config: Add new monitor repository for OpenStack monitoring service  https://review.openstack.org/20325917:01
jeblair    api-timeout: 6017:01
jeblairclarkb: ^17:01
*** deepakcs has quit IRC17:01
*** Ala has quit IRC17:01
clarkbok so possible the client/requests/urllib3 isnt respectibg that17:01
*** jsavak has joined #openstack-infra17:01
jbernardmriedem: has multiple cinder backends in gate been discussed already?17:02
jeblairclarkb: i know we've seen it do so in the past17:02
nibalizercrinkle: im more worried that I don't really know anything about selinux network controls so that change could be bad without me knowing why17:02
*** zz_dimtruck is now known as dimtruck17:02
jeblairclarkb: like when hpcloud was taking forever to respond to api calls17:02
jeblairclarkb: so i suspect this is an edge condition it doesn't handle :/17:02
jbernardmriedem: i would really like to add coverage for the new vol migration changes17:02
jeblairclarkb: (maybe connection dropped in middle of tls packet or something)17:02
anteayajbernard: if you post in -nova more folks will see your words17:03
jbernardanteaya: thanks, will do17:03
clarkbjeblair gotcha17:03
*** mrmartin has quit IRC17:03
openstackgerritVictor Ryzhenkin proposed openstack-infra/project-config: Add opportunity to install muranoclient from change  https://review.openstack.org/20415917:03
*** derekh has quit IRC17:04
crinklenibalizer: i tested it on an hpcloud centos 6 vm, would be easy enough to reproduce my tests17:04
openstackgerritVictor Ryzhenkin proposed openstack-infra/project-config: Make job gate-murano*-requirements use job template  https://review.openstack.org/20317717:04
jeblairi suspect we are approximately 1 hour from dib image builds being complete17:04
openstackgerritMerged openstack-infra/puppet-bandersnatch: Migrate to puppet-httpd module  https://review.openstack.org/20260917:04
jeblairi think i'd rather have the extra 200 instances today, so i'm going to restart nodepool17:04
*** andrey-mp has quit IRC17:04
*** tsg has quit IRC17:04
*** soren has quit IRC17:04
jeblairstopping nodepool when it's running dib doesn't work like you expect17:06
clarkbjeblair go for it we can retrigger builds if necessary17:06
clarkbjeblair it should, the builds are killed17:07
jeblairroot@nodepool:~# /etc/init.d/nodepool restart17:07
jeblair * Restarting Nodepool nodepool                                                                                                                                                                                                                                                                                                                                 .... waiting ...............................................................................17:07
clarkbwell the build it only does one ata time17:07
clarkboh restart doesnt work at all iirc. hashar was working on it17:07
jeblairooooh17:07
crinklenibalizer: i'm going to test it again and make sure it does the right thing when behind_proxy is true/false17:07
nibalizerok17:08
jeblairi wanted to restart to minimize zmq downtime (since we leak completed nodes during a restart)17:08
mriedemjbernard: ?17:08
jeblairanyway, stop/started now17:08
*** _nadya_ has joined #openstack-infra17:09
*** gsagie_ has quit IRC17:09
jbernardmriedem: just curious if there as an existing effort (or discussion) on multipe backends for cinder in the gate17:09
jbernardmriedem: if i had it, i could add test coverage for volume migration17:10
*** Krinkle_ is now known as Krinkle17:10
jeblairso now we should just be waiting for rax to delete the 83 deleted servers17:10
clarkbjeblair https://review.openstack.org/#/c/203232/17:10
clarkbthats hashars change17:11
*** thorst has quit IRC17:11
*** shardy_ has joined #openstack-infra17:11
*** vnod has quit IRC17:12
*** puranamr has joined #openstack-infra17:12
*** jasondotstar has joined #openstack-infra17:13
*** shardy has quit IRC17:13
*** puranamr has quit IRC17:13
nibalizerhttp://puppetboard.openstack.org/report/git02.openstack.org/d0ca097885d666b64ee26ce9a44f7456563af2c1 my crystal ball says that is caused by us losing our symlink from `which pip` to /usr/local/bin/python-pip17:13
nibalizera few other git servers are affected17:13
clarkbnibalizer where did the symlink disappear to?17:14
*** Hal has joined #openstack-infra17:14
*** Hal is now known as Guest5796617:14
*** marzif has joined #openstack-infra17:15
jeblairnibalizer: 06-08 are newer; is that part of the pattern?17:15
*** puranamr has joined #openstack-infra17:15
nibalizerclarkb: I'm guessing thats the issue, can you hop on that server and confirm17:15
nibalizerif memory serves we create that symlink in install_puppet.sh but don't ensure it in puppet itself17:15
nibalizerat laest thats a problem gozer had once17:15
*** shardy_ has quit IRC17:16
jeblair[corvus@git02 ~]$ ls -ld `which pip`17:17
jeblair-rwxr-xr-x. 1 root root 281 Jul  1 09:51 /usr/bin/pip17:17
nibalizerright and puppet, on redhat, tries to use pip-python and fails when it doesn't find it17:17
*** shardy has joined #openstack-infra17:17
nibalizerwhich is fine if you're using system packages, but we use get-pip17:17
clarkbnibalizer I cannot. not on machine with keys currebtly17:17
openstackgerritDavid Cheperdak proposed openstack-infra/project-config: Add OpenStack project Fusion.  https://review.openstack.org/20443417:18
clarkbnbalizer right did we change or remove the symlink install?17:18
nibalizerclarkb: maybe the newer git machines didn't have install_puppet.sh run on them?17:18
jeblairall the git servers are like that17:19
openstackgerritSpencer Krum proposed openstack-infra/system-config: Ensure a symlink for pip-python  https://review.openstack.org/20469717:20
jeblairthe new ones were created normally so should have had install_puppet run17:20
*** MarkAtwood has quit IRC17:20
jeblairoooh17:21
nibalizerjeblair: `which pip-python` on git02?17:21
jeblair/usr/bin/pip-python exists on 1,6,7,8, but not 2,3,4,517:21
jeblairi'm unaware of a pattern that explains that17:21
funginibalizer: sounds like puppet needs fixing somewhere? red hat stopped using pip-python (after a brief stint of calling it python-pip too)17:22
*** auxian has quit IRC17:23
openstackgerritMerged openstack-infra/puppet-openstackid: Migrate to puppet-httpd module  https://review.openstack.org/20173717:23
openstackgerritMerged openstack-infra/puppet-mailman: Migrate to puppet-httpd module  https://review.openstack.org/20174317:23
funginibalizer: now the rpm only provides pip, pip2 and pip2.617:23
openstackgerritMerged openstack-infra/project-config: Remove non-voting pypy gate jobs  https://review.openstack.org/20341117:23
*** BharatK has quit IRC17:23
nibalizerfungi: ya I know where the fix to puppet-core would go17:23
nibalizerI'm supposed to be in there fixing their 'upgrade' code as well17:23
*** e0ne has quit IRC17:24
funginibalizer: you can see by running `rpm -ql python-pip-7.1.0-1.el6.noarch` on an updated centos 6.x system17:24
jeblairthe node graph looks like we're doing something now.17:24
* nibalizer -> afk for a bit17:25
*** abregman has quit IRC17:25
*** marzif has quit IRC17:25
*** gmurphy has joined #openstack-infra17:26
*** afazekas_ has quit IRC17:26
*** marzif has joined #openstack-infra17:26
openstackgerritMerged openstack-infra/project-config: remove check-gerrit-buck  https://review.openstack.org/20272717:26
funginibalizer: mainly just wondering how that module works at all on current centos 617:26
openstackgerritMerged openstack-infra/puppet-drupal: Migrate to puppet-httpd module  https://review.openstack.org/20174717:26
openstackgerritMerged openstack-infra/puppet-diskimage_builder: Add Gemfile and puppet 4 checks  https://review.openstack.org/20266117:26
fungior whether everyone who uses it puts in a similar workaround17:27
*** bswartz1 is now known as bswartz17:27
gmurphythis seems like a bit of a glitch visiting - https://tarballs.openstack.org/ presents an invalid certificate, if i choose to proceed i hit the security.openstack.org page. should i create a bug for this or is it a known issue?17:27
*** dannywilson has quit IRC17:28
clarkbgmurphy I dont think there is an https vhost for that name17:30
fungigmurphy: they're hosted on the same server17:30
fungigmurphy: security.o.o is the only https site on it17:30
gmurphyoh right. that makes sense.17:30
fungigmurphy: there is no https for tarballs.o.o17:30
clarkbso bug with our apache setup?17:30
gmurphythink so.17:31
*** marzif has quit IRC17:31
openstackgerritOleksii Zamiatin proposed openstack-infra/project-config: Make zeromq functional gate check non-voting  https://review.openstack.org/20209517:31
fungiis it? i always just assumed that's how hosting multiple sites on one server works. you have a *:443 site and then *:80 sites with name-based virtual hosting17:31
clarkbfungi apache can distinguish and should probably return an error?17:32
*** marzif has joined #openstack-infra17:32
fungican't host multiple https sites on the same ip address and port number without jumping through extra hoops17:32
fungiclarkb: well, yes and no. apache can't know that until after it's presented the cert to the browser17:32
clarkboh right17:32
fungiclarkb: ssl negotiation happens first, then the browser tells the server the hostname it's looking for17:32
jeblairwell, sni is mostly a thing in browsers now17:32
gmurphyoh fair enough. it's been way too long since i've messed with apache config to figure this one out. leave it with you guys.17:33
*** tjones1 has joined #openstack-infra17:33
*** ildikov has quit IRC17:33
jeblairso we probably could do better17:33
fungimaybe. i think sni would require us to add all the vhost names for static.o.o's apache as subject altnames for that cert?17:33
*** Sukhdev has quit IRC17:33
fungiotherwise visiting any of them with https:// would lead to an invalid cert error in the browser17:34
*** koolhead17 has joined #openstack-infra17:34
openstackgerritgreghaynes proposed openstack/diskimage-builder: Perform a booting test for our images  https://review.openstack.org/20463917:35
jeblairfungi: no, sni lets apache pick different certs based on the handshake17:35
greghaynesfungi: sni?17:35
*** madhu_ak has quit IRC17:35
greghaynesyea17:35
fungiso, yes. we could do better by either a) getting ssl cert coverage for all sites on static.o.o, or b) move security.o.o to a separate server17:35
fungiunless i'm misunderstanding17:35
greghaynesUnless you need to support letting people from nintendo wii's hit your website17:36
fungiokay, so not subjectaltname in a common cert, but multiple certs for them still17:36
jeblairfungi: i am not certain whether it would let us c) return some kind of error condition for non-security.o.o vhosts17:36
fungiwhat i'm saying is i don't think sni solves the "invalid cert for https://tarballs.openstack.org/" problem unless we actually provide ssl cert coverage for that vhost name17:37
jeblairfungi: unless c is somehow possible, i agree.17:37
jeblairi should amend (c) to say "some kind of non-confusing error condition"17:38
jeblairsince it's already returning a confusing one :)17:38
fungii've witnessed plenty of issues in the past where proxies relying on sni which neglect to add the right vhost name for the thing they've cnamed to it, and you get an invalid cert error in the browser17:38
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add a section to the readme about removing bug signatures  https://review.openstack.org/20470717:39
fungiso at least the browsers i use provide basically the same error regardless of whether the cert is for one hostname which isn't the thing you've requested, or the server isn't providing certs for the host you've requested17:39
jeblairoh good, rax deleted the 83 deleted servers17:40
jeblairso i think we're at max capacity now17:40
*** thorst has joined #openstack-infra17:40
*** changbl has quit IRC17:41
*** ericksonsantos has quit IRC17:42
*** esmute_ has quit IRC17:43
*** davideagnello has joined #openstack-infra17:43
*** ajmiller has quit IRC17:44
*** davideagnello has quit IRC17:45
*** pelix has quit IRC17:45
nibalizerfungi: https://github.com/puppetlabs/puppet/blob/3.x/lib/puppet/provider/package/pip.rb#L3817:47
*** timrc_ has joined #openstack-infra17:47
nibalizerfungi: so I would expect latest centos 6.x and latest puppet 3.x do not have a functional pip provider17:48
*** sergmelikyan has joined #openstack-infra17:48
*** maishsk has quit IRC17:48
*** maishsk has joined #openstack-infra17:49
*** davidjc has quit IRC17:49
*** kjnelson has quit IRC17:49
*** pvaneck has joined #openstack-infra17:50
*** heyongli has quit IRC17:52
*** esmute_ has joined #openstack-infra17:52
*** changbl has joined #openstack-infra17:52
*** heyongli has joined #openstack-infra17:52
*** amickus has joined #openstack-infra17:53
*** ajmiller has joined #openstack-infra17:55
*** yfried has joined #openstack-infra17:55
*** pal has quit IRC17:55
SpamapSjeblair: so, if we merge https://review.openstack.org/201666 .. I can rework the storyboard a bit and instead of making a single node ironic puppet manifest, I can start making OpenStack controllers and compute node manifests...17:55
SpamapSjeblair: also we likely want a bifrost manifest instead of the ironic puppet manifest.17:56
SpamapSwhich is just o_O .. "Yo dawg, I heard you like puppet so I wrote some puppet to deploy our ansible that deploys your puppet"17:56
*** esmute_ has quit IRC17:57
*** dimtruck is now known as zz_dimtruck17:57
*** jasondotstar has quit IRC17:57
openstackgerritBoris Pavlovic proposed openstack-infra/project-config: Add non voting fuel agent python jobs  https://review.openstack.org/20471117:58
*** ajmiller has quit IRC17:58
*** sergmelikyan has quit IRC17:59
*** baoli has quit IRC17:59
jeblairSpamapS: yeah, we told EmilienM|off that and it made him go on vacation18:00
SpamapSjeblair: doh! ;)18:01
SpamapSbut he's french, they go on vacation if you tell them there is a place that has vacations.18:01
*** baoli_ has joined #openstack-infra18:01
*** kjnelson has joined #openstack-infra18:01
*** sergmelikyan has joined #openstack-infra18:01
* SpamapS is jealous18:01
*** thorst has quit IRC18:02
*** marzif has quit IRC18:02
*** bhunter71 has joined #openstack-infra18:02
*** larainema has quit IRC18:02
jeblairSpamapS: i _think_ that change says that ansible can talk directly to ironic to launch our cloud?18:03
*** ericksonsantos has joined #openstack-infra18:03
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Set docimpact group of os-ansible-deployment  https://review.openstack.org/20471218:03
*** Kennan has joined #openstack-infra18:04
*** Kennan2 has quit IRC18:04
*** zz_dimtruck is now known as dimtruck18:04
*** marzif has joined #openstack-infra18:04
*** esmute_ has joined #openstack-infra18:04
harlowjahi guys, for anvil (stackforge package building, blah blah); the gate it has is currently testing venv building (on ubuntu); which is great, but not always sufficent, it also builds rpms and was wondering if its possible to get a custom script that gets activated on a rhel infra-vm instance (anvil needs rpmbuild installed, yum-build-dep...) so that the rpm building can also be a gate test... (vs right now its been somewhat manual18:04
harlowja pita verification...)18:04
*** davidjc has joined #openstack-infra18:05
bhunter71a question;   on my centos6 builders, I seem to get pip 1.5.4 in my py26 virtualenvs.   That is, when I activate and pip --version I see 1.5.4.  The system has pip 7.1 installed globally.   Is this normal, and if not, how do I get a recent pip into my centos6 builder?   thanks!18:05
harlowjaanvil can setup these things, but it needs sudo access (that likely requires something special ?)18:06
*** Sukhdev has joined #openstack-infra18:06
*** krtaylor has quit IRC18:06
fungiharlowja: the job just needs to not invoke the revoke-pip builder macro18:07
*** changbl has quit IRC18:07
harlowjahmmm18:07
jeblairSpamapS: sorry, let me rephrase that as a real question: once puppet has configured bifrost+ironic on the bare metal node, and we have an ansible inventory describing how to turn other bare metal instances into our cloud (by having ansible run puppet on them), ansible will talk to bifrost's ironic to launch those nodes, right?18:07
*** mgershen has quit IRC18:07
jeblairfungi, harlowja: 'revoke-sudo'18:07
*** ajmiller has joined #openstack-infra18:07
*** marzif_ has joined #openstack-infra18:07
harlowjakk, let me see if i can scrap something together that might allow this to work18:08
fungier, yes, that's what i meant ;)18:08
jeblairharlowja: and yeah, there are centos and fedora nodes available18:08
harlowja* http://paste.openstack.org/show/400732/ (the stuff that gets boostraped)18:08
fungibhunter71: yes, the version of pip in a virtualenv is determined by the version of virtualenv you installed18:08
fungivirtualenv provides its own pip, it doesn't use the pip you have installed on your system18:09
SpamapSjeblair: correct.18:09
bhunter71fungi:  ok.  Is my virtualenv likely stale?   This isn't something I've knowingly modified in my environment.   Images are up to date18:09
jeblairSpamapS: okay cool.  anyone else we should get to weigh in on that?  greghaynes?18:09
jeblairclarkb, fungi: you may be interested in https://review.openstack.org/20166618:10
fungibhunter71: if you have pip 1.5.4 in your virtualenv then that implies you've got virtualenv 11.something installed i think18:10
SpamapSjeblair: We skip the part where we put keystone, nova, glance, and neutron in place, and rely on the API being available only on 127.0.0.1.18:10
SpamapSWhich I actually don't love18:10
*** changbl has joined #openstack-infra18:10
*** marzif has quit IRC18:11
*** Sukhdev has quit IRC18:11
bhunter71fungi: yes, virtulenv --version == 1.11.418:11
jeblairSpamapS: oh, interesting... so we could not run ansible->bifrost from puppetmaster?18:11
*** alex_xu has quit IRC18:11
jeblairSpamapS: we'd have to run that ansible locally18:11
SpamapSjeblair: we could, but any node that could talk to ironic would have full control over everything on that node.18:12
harlowjajeblair fungi  does that include centos6 and centos7 envs?18:12
*** mwagner_lap has quit IRC18:12
harlowjaanvil should work with both, and having both would be sweet18:12
SpamapSjeblair: so I'm thinking we keep it 127.0.0.1 and maybe even put in place iptables owner rules18:12
jeblairSpamapS: oh, okay, so we could configure iptables to let puppetmaster ansible talk to bifrost?18:12
SpamapSjeblair: right we could also just poke a hole for the puppetmaster to do it.18:13
SpamapSjeblair: but bifrost is not being developed this way. It's entirely theoretical that these playbooks work on !localhost18:13
*** puranamr has quit IRC18:13
*** alex_xu has joined #openstack-infra18:13
SpamapSTheJulia: ^ confirm that?18:14
SpamapSNobodyCam: or you? ^18:14
*** woodster_ has joined #openstack-infra18:14
fungiharlowja: yes we have both centos 6 and 7 workers18:14
jeblairSpamapS: okay, worst case is i guess we have a playbook we run on puppetmaster that runs a playbook on bifrost-host, or tell ansible bifrost-host is an ssh jump host or something.18:15
harlowjafungi nice, let's see if i can figure out the project-config magic for this18:15
fungiharlowja: though we're likely to drop centos 6 within the year18:15
harlowjak18:15
harlowjafair enough18:15
greghaynesjeblair: The only reasoning I heard for doing a full bm cloud rather than standalone ironic was so our ansible launch node future code could talk to nova18:15
jeblairgreghaynes: yeah, i think mordred is satisfied that similar functionality can happen with launch-node talking to ironic18:16
NobodyCamSpamapS: oh I have not tested such a thing, but I think the playbooks could easily be adapted to work from !localhost18:17
SpamapSIndeed, the launching part of bifrost is actually really simple.18:17
SpamapSNobodyCam: yeah, this is one of the advantages of using Ansible. :)18:17
*** amitgandhinz has joined #openstack-infra18:17
NobodyCamTheJulia: do you know (off the top of your head) if there is something that would block ^^^^^^^^^18:18
SpamapSI'm reading the playbooks18:18
SpamapSI see nothing18:18
NobodyCamya18:18
SpamapSjust have to change localhost to a variable or a group name.18:18
TheJuliaon the phone atm, will read/context switch in a minute18:18
greghaynesSGTM, I think its preferrable to do the conservative thing in terms of effort for what were trying to do right now, and if later on it doesnt work well for launch-node we can reevaluate then18:19
*** _nadya_ has quit IRC18:19
SpamapSand move the 'connection: local' into the inventory.18:19
NobodyCamTheJulia: ack TY18:19
*** marzif_ has quit IRC18:19
SpamapSyeah18:19
*** puranamr has joined #openstack-infra18:19
*** _nadya_ has joined #openstack-infra18:19
SpamapSit's probably worth noting that this is going to jump start the cloud deployment. I can probably have all the working nodes deployed in a couple hours.18:19
greghaynesfamous last words18:20
SpamapSnote the qualification, "working nodes"18:20
*** soren has joined #openstack-infra18:20
*** changbl has quit IRC18:20
SpamapSOh! Let's talk about hostnames. Does infra have a schema that I need to adhere to?18:21
* SpamapS suspects this is covered on ci.openstack.org18:21
*** timrc_ has quit IRC18:22
*** bharath_ has quit IRC18:22
jeblairSpamapS: i don't think we've written anything about it... what do you have in mind?18:22
*** tonytan4ever has joined #openstack-infra18:23
jeblairSpamapS: i suspect we'd probably want to do something like X.<name of cloud instance>.openstack.org18:23
TheJuliaSpamapS: NobodyCam brain still occupied, wrapping up shortly18:23
*** _nadya_ has quit IRC18:24
ttxjeblair: quick question while i think about it... could we technically separate openstack and stackforge/experimental things wherever it gets presented as a catalog (think github or cgit)18:24
SpamapSI was actually just going to pick an incrementing offset. baremetal###.infracloud0.openstack.org or something like that.18:24
jeblairttx: no18:24
*** thorst has joined #openstack-infra18:24
ttxI'm trying to think of ways to reduce confusion as to what is an official thing and what is not (or not yet)18:24
jeblairSpamapS: that sounds good to me18:24
SpamapSjeblair: just need to name all the nodes.18:24
*** mwagner_lap has joined #openstack-infra18:24
jeblairSpamapS: we could shorten that a bit too if you want18:25
clarkbmaybe controllerXX and computeXX or woll they be more fluid than that?18:25
ttxjeblair: does the prefix we are using in Gerrit have to be the same in github ?18:25
clarkbbarrmetalXX doesnt say much beyond what you already know18:25
ttxjeblair: and is there a way to split things between a cgit.o.o and a other-cgit.o.o18:26
SpamapSclarkb: they're all identical except the two with access to the ilos, so it will be somewhat arbitrary how we choose.18:26
jeblairttx: yes prefixes must be the same18:26
*** agentle_ has joined #openstack-infra18:26
ttxjeblair: ok18:26
SpamapSclarkb: I'm fine with setting the roles up front like that.18:26
ttxso apatr from "forgetting to sync some" there is no real option on the guthub side18:27
jeblairttx: and the cgit split may be possible but with tremendous difficulty i think18:27
*** sergmelikyan has quit IRC18:27
clarkbSpamapS we can always update it later too18:27
clarkbso baremetalXX wfm18:27
jeblairttx: yeah, the github org moves are a big part of the driver here18:27
jeblairttx: (they are actually less automatable than gerrit)18:28
ttxjeblair: I mean, technically, we could only sync "openstack" repos18:28
SpamapSclarkb: I guarantee we'll re-deploy these nodes at _least_ once since they have network cards we don't like and RAID w/o battery backed write cache at the moment.18:28
jeblairttx: to attempt to put your mind at ease, i actually think it's been a very long time since someone may have attempted to use github.com/openstack as a catalog18:28
ttxand let stackforge/experimental things do whatever they want18:28
*** sergmelikyan has joined #openstack-infra18:28
jeblairthere's already just way too much there and it's unavigable18:28
*** davideagnello has joined #openstack-infra18:28
*** sergmelikyan has quit IRC18:29
ttxjeblair: yeah. ok, thanks for the answer18:29
*** yamamoto has joined #openstack-infra18:31
*** Sukhdev has joined #openstack-infra18:31
TheJuliaSpamapS: so enroll/deploy such playbooks can be run remotely, the module must execute locally, or concievably on a remote host where the moudle and dependencies (shade) are located18:31
jeblairSpamapS: so i'd either go with your proposal, or bm00.ic0.openstack.org if you want to type less.  :)18:32
* greghaynes nitpicks and points out that we should have at least 3 numerical digits18:32
jeblairgreghaynes: very good point18:33
*** changbl has joined #openstack-infra18:33
clarkbhundreds of machines? no way18:33
greghaynescue dr. evil pinky18:33
clarkbmwahahaha18:33
*** HeOS has quit IRC18:33
jeblairhow many digits should we have for clouds? :)18:34
greghaynes10 clouds ought to be enough for anyone ;)18:35
*** MarkAtwood has joined #openstack-infra18:35
*** yamamoto has quit IRC18:36
* jesusaurus 's clouds go to 1118:36
*** ayoung has quit IRC18:36
jeblairwow, hpcloud leaks floating ips like a sieve18:36
*** jasondotstar has joined #openstack-infra18:36
anteaya:(18:37
*** ivar-lazzaro has quit IRC18:38
asselinhow do the leaks occur? I have some too locally....18:38
jeblairand now hpcloud-b2 and -b3 have stopped working, likely also due to the requests problem18:38
*** Hal has joined #openstack-infra18:39
*** Hal is now known as Guest7505318:39
patrickeastwhen you say leaks, what does that mean? i’m seeing issues with my 3rd party ci where nodepool is not releasing floating ip’s when its done with single use nodes, is that what you guys are seeing?18:40
jeblairsigmavirus24: hey, we're regularly seeing threads get stuck here, despite having a 60 second timeout: http://paste.openstack.org/show/400649/18:40
*** achanda has joined #openstack-infra18:40
jeblairsigmavirus24: do you have any suggestions about how to avoid that?18:41
*** ZZelle_ has joined #openstack-infra18:41
jeblairpatrickeast: well, not sure whether it's nodepool or openstack at fault, but yes18:41
sigmavirus24jeblair: so timeouts are not overall timeouts18:41
sigmavirus24or what's commonly referred to as a wall timeout18:41
clarkbI think openstack18:41
*** krtaylor has joined #openstack-infra18:42
sigmavirus24They're applied to connection and read. So if it's hanging that's because data is trickling back to the client18:42
*** sergmelikyan has joined #openstack-infra18:42
sigmavirus24In other words, your timeout is long enough that just a little data comes back and resets the timeout18:42
clarkbwe say delete fip, get positive response without delete then delete the node attached to it which removes our way of knowing about the fip18:42
*** warpc__ has quit IRC18:42
asselinI wrote a script to work around the leaking floating-ips: https://git.openstack.org/cgit/stackforge/third-party-ci-tools/tree/nodepool/clean_floating_ip.sh18:42
*** dims_ has quit IRC18:42
clarkbsince in nodepool its all node oriented and we dont mamage pools of fips too18:43
patrickeastgotcha18:43
jeblairsigmavirus24: it's been an hour, so i'm almost certain no data has been received (this is an openstack api call)18:43
sigmavirus24jeblair: https://github.com/pnpnpn/timeout-decorator seems to be a good wall timeout solution18:43
sigmavirus24jeblair: well the socket thinks there's data18:43
sigmavirus24*thinks it is receiving data18:43
patrickeastasselin: do you just run that as a cron job?18:43
jeblairsigmavirus24: are you sure it isn't the ssl-wrapper waiting for a complete packet?18:44
asselinpatrickeast, yes18:44
sigmavirus24jeblair: it could be that too18:44
sigmavirus24We set the timeouts on the socket object itself18:44
*** e0ne has joined #openstack-infra18:45
asselinpatrickeast, I've been meaning to see if nodepool can do that, but didn't get a chance. And seems perhaps it's not really nodepool's issue....18:46
clarkbit would be nice if A) deleting an fip actually deleted it and B) deleting a node deleted the unneded fip (at least as an option)18:47
*** bharath_ has joined #openstack-infra18:47
*** dims_ has joined #openstack-infra18:47
openstackgerritDavid Cheperdak proposed openstack-infra/project-config: Add OpenStack project Fusion.  https://review.openstack.org/20443418:47
asselinor c) creating a node used an unallocated floating-ip18:47
clarkbbut iirc the second thing isnt atomic yet18:47
clarkbso you can get them out of sync and leak regardless18:48
fungiright, given that you can't attach metadata to a fip and it doesn't have a distinct uuid (afaik) and a fip can reappear later because it's only a limited pool of available addresses anyway, i don't think there's much hope of nodepool being able to detect that a fip it asked to have deleted wasn't actually deleted even though the api claims it would do so18:49
nibalizerasselin: can you vote/+a https://review.openstack.org/#/c/184905/21 and https://review.openstack.org/#/c/198463/ ?18:49
jeblairsigmavirus24: i think that solution will be problematic -- we have about 10 threads that are pretty much continuously making api calls; multi-threaded means we have to use the multiprocessing variant of that, and it looks like it will end up spawning a process per call...18:49
asselinnibalizer, looking18:50
jeblairsigmavirus24: we could rework nodepool to use multiprocessing internally for its managers, and then use the signal strategy in each of those (relatively long-lived) processes18:50
openstackgerritgordon chung proposed openstack/requirements: block oslo.messaging 1.17.x due to memory leak  https://review.openstack.org/20472718:50
jeblairsigmavirus24: but that will make database interaction very complicated18:51
sigmavirus24jeblair: yeah this is one of those things that causes some headaches for people (not having a wall timeout)18:51
jeblairthat's sort of at the 'rewrite nodepool level of complexity'18:51
jeblairsigmavirus24: i think this is why we were barking up the tcp keepalive tree a while a go18:51
jeblairsigmavirus24: because we're pretty sure that everyone thinks this socket is dead except our program18:52
sigmavirus24jeblair: so if you want TCP keep-alive you can use https://toolbelt.readthedocs.org/en/latest/adapters.html#tcpkeepaliveadapter18:52
asselinnibalizer, I don't know enough about ruby to approve those18:52
jeblairsigmavirus24: was this where we got as far as "we need support in the openstack clients" to use it?18:52
jeblairclarkb, fungi: ^?18:53
sigmavirus24jeblair: so keystoneclient's session should set it up by default iirc18:53
sigmavirus24Both novaclient sans keystoneclient and keystoneclient should be auto-using tcp keep-alive18:53
fungii vaguely remember that clarkb might remember this more accurately than i do ;)18:54
sigmavirus24jeblair: https://git.openstack.org/cgit/openstack/python-keystoneclient/tree/keystoneclient/session.py#n90118:54
jeblairoh interesting; so i should see if we're running a too-old novaclient.  and perhaps when we fully switch to shade, since shade is using keystone sessions, we'll get it there too18:54
clarkbat the time I dont think the clients supported it18:54
clarkbso this would be new18:54
sigmavirus24If novaclient uses a keystoneclient session, it supports it. If it doesn't use a ksc session, it supports it18:54
sigmavirus24This happened before the summit (these changes)18:55
crinklenibalizer: i'm more confident in my selinux puppet-cgit change now, i'm wondering if it would be easy to add a centos6 beaker job just for that module so we can all be as confident?18:55
*** krtaylor has quit IRC18:55
clarkbyes and when we looked at it was bwfore the summit18:55
clarkbsince after summit has been lolbusy for me18:55
sigmavirus24https://git.openstack.org/cgit/openstack/python-novaclient/tree/novaclient/client.py#n6118:55
sigmavirus24clarkb: I know what you mean18:55
*** bswartz has quit IRC18:56
nibalizercrinkle: im hoping we can land the openstackci stuff before we start copying it out to the swarm18:56
sigmavirus24Ah, we're not setting keepidle in those adapters18:56
nibalizeravoid 30-odd patches because of some trivial bug/nitpick in the spec_helper_acceptance.rb in openstackci18:56
sigmavirus24That could be what's causing problems18:56
crinklenibalizer: makes sense18:56
jeblairsigmavirus24: we are running a version of novaclient with that18:56
*** glauco__ has joined #openstack-infra18:57
anteayapaste has an inbound spike: http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=1418:57
*** baoli_ has quit IRC18:57
*** baoli has joined #openstack-infra18:59
sigmavirus24I wrote the toolbelt implementation of the TCPKeepAliveAdapter a month or two after doing it for novaclient/keystoneclient18:59
sigmavirus24I wonder if we should just depend on that instead of copying it into two clients18:59
nibalizerasselin: I'll ask for some reviews from the openstack/puppet-* folks and see if we can get some more qualified eyes on it that way18:59
*** sherif has quit IRC19:00
SpamapSsigmavirus24: very seldom is "copying it into ..." the most optimal long term solution to code issues.19:01
crinklenibalizer: asselin if it helps we have nearly the same file in all the openstack/puppet-* modules19:01
sigmavirus24SpamapS: relying on the toolbelt just for one adapter seems a bit silly though19:01
*** davideagnello has quit IRC19:01
sigmavirus24I mean there are lots of other goodies in there19:01
* asselin looks19:01
sigmavirus24But I've never seen openstack need streaming multipart/form-data uploads19:01
SpamapSsigmavirus24: could it be a standalone improvement that toolbelt relies on?19:01
SpamapSor rather, standalone library19:02
*** glauco__ has quit IRC19:02
sigmavirus24So we've actually been consolidating recipes in the toolbelt that we had previously written extemporaneously into issue comments in the toolbelt with tests19:02
sigmavirus24The idea was to keep everything in one place19:03
*** glauco__ has joined #openstack-infra19:03
*** Krinkle is now known as Krinkle_19:03
sigmavirus24I wrote the code so doing a separate package isn't a problem in reality, I just don't want to maintain two packages instead of one19:03
sigmavirus24#lazy19:03
jeblairsigmavirus24: i haven't found a definitive answer, but from my reading so far, i do strongly suspect that keepidle is required in order for the others to have an effect19:04
sigmavirus24jeblair: yeah, the other fun thing is that depending on the OS, some options may not even be available19:05
sigmavirus24So the code we have in the toolbelt does "the right thing" by checking for them before trying to set them19:05
jeblairsigmavirus24: or it may be using a very high default value, like 2 hours19:05
sigmavirus24jeblair: yeah the keep-alive documentation is pretty much non-existent last I checked19:05
anteayathe paste inbound spike seems to be decreasing19:06
openstackgerritgreghaynes proposed openstack/diskimage-builder: Perform a booting test for our images  https://review.openstack.org/20463919:06
*** maishsk_ has joined #openstack-infra19:06
*** maishsk has quit IRC19:06
*** maishsk_ is now known as maishsk19:06
jeblaircat /proc/sys/net/ipv4/tcp_keepalive_time19:06
jeblair720019:06
anteayahttp://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=152&rra_id=all19:06
sigmavirus24woah19:06
clarkbthats default iirc19:07
jeblairsigmavirus24: so yeah, that looks like the linux system-wide default19:07
*** glauc____ has joined #openstack-infra19:07
*** glauco_ has quit IRC19:07
jeblairsigmavirus24: if it would be useful, i could leave nodepool running as-is for another 40 minutes and we may be able to confirm that's what's happening (if it times out at 19:43 UTC)19:07
*** glauco_ has joined #openstack-infra19:08
*** glauco__ has quit IRC19:08
*** bharath__ has joined #openstack-infra19:09
jeblair(let me see if i can dig in the logs for an earlier instance)19:09
sigmavirus24jeblair: just for confirmation, that'd be good. in the meantime I'll write up patches for ksc and novaclient. Novaclient can lean on ksc's adapter now that the requirements should be sufficient going forward19:09
*** shardy_ has joined #openstack-infra19:10
*** koolhead17 has quit IRC19:10
*** andersonvom has quit IRC19:10
*** shardy has quit IRC19:11
openstackgerritMerged openstack/diskimage-builder: Use official mirror name for debian-minimal  https://review.openstack.org/20034519:12
*** changbl has quit IRC19:12
asselinnibalizer, done19:12
*** changbl has joined #openstack-infra19:12
*** bharath_ has quit IRC19:13
*** Krinkle_ is now known as Krinkle19:13
*** shardy_ has quit IRC19:15
*** kjnelson has quit IRC19:16
*** shardy has joined #openstack-infra19:16
jeblairi can't prove it with the logs, so we'll just wait it out19:16
asselinbtw, nova - juno doesn't delete the fip when the instance is deleted. They're left in the pool, but nodepool/nova doesn't re-use them: http://paste.openstack.org/show/400961/19:19
*** esmute_ has quit IRC19:19
*** afazekas has joined #openstack-infra19:19
clarkbasselin yes nodepool does an explicit delete first iirc19:19
clarkbso its delete fip && delete node19:19
asselinclarkb, why does it need to delte the fip? why not leave it in the pool and get it from there?19:20
clarkbasselin because it doesnt pool fips19:21
clarkbso doesnt have a way to travk them beyond node attachment19:21
clarkbit could pool them but that would require a bit of rework in nodepool19:21
*** crc32 has joined #openstack-infra19:22
ZZelle_mordred, hi19:22
asselinclarkb, but nova pools them, nodepool just needs to associate one of them http://paste.openstack.org/show/400962/19:23
patrickeastasselin: i was just looking at this to try and figure out why i’m not getting any back (like 0% success rate releasing them) it frees them here https://github.com/openstack-infra/nodepool/blob/master/nodepool/provider_manager.py#L60519:23
clarkbasselin but you have to know which arent used or bwing deleted19:23
clarkband you have to know when to get more and free unneeded IPs19:23
clarkbits not as simple as just use IP19:24
asselini see, devil in the details19:24
openstackgerritJoshua Harlow proposed openstack-infra/project-config: Adjust anvil gate/check jobs to test venv and rpm building  https://review.openstack.org/20473419:24
*** amickus has quit IRC19:24
clarkb(and this is why nwutron should just give you working networks)19:24
harlowjafungi jeblair ^ if u get a chance, think i did that right, ha19:24
*** sergmelikyan has quit IRC19:25
*** puranamr has quit IRC19:27
*** puranamr has joined #openstack-infra19:28
*** MarkAtwood has quit IRC19:29
*** ivar-lazzaro has joined #openstack-infra19:29
*** esmute_ has joined #openstack-infra19:30
*** mriedem has quit IRC19:32
sigmavirus24jeblair: I'm not sure if there's nodepool project on launchpad but you might want to mark https://bugs.launchpad.net/python-novaclient/+bug/1477275 as affecting it too19:33
openstackLaunchpad bug 1477275 in python-novaclient "KSC's TCPKeepAliveAdapter should specify keepidle to prevent connections from hanging for hours" [Undecided,New] - Assigned to Ian Cordasco (icordasc)19:33
uvirtbotLaunchpad bug 1477275 in python-novaclient "KSC's TCPKeepAliveAdapter should specify keepidle to prevent connections from hanging for hours" [Undecided,New]19:33
uvirtbotLaunchpad bug 1477275 in python-novaclient "KSC's TCPKeepAliveAdapter should specify keepidle to prevent connections from hanging for hours" [Undecided,New] https://launchpad.net/bugs/147727519:33
*** jsavak has quit IRC19:34
jeblairsigmavirus24: it's on storyboard, so can't make that link.  thanks19:34
*** esmute_ has quit IRC19:34
*** jsavak has joined #openstack-infra19:35
*** jasondotstar has quit IRC19:36
*** mriedem has joined #openstack-infra19:36
harlowjajeblair fungi in https://review.openstack.org/#/c/204734/1/jenkins/jobs/anvil-jobs.yaml does that shell script get executed in the directory of the checked out git code, or is there some other magic needed to do that?19:37
harlowjathat might be a change needed if so19:37
*** sergmelikyan has joined #openstack-infra19:37
openstackgerritBoris Pavlovic proposed openstack-infra/project-config: Add non voting fuel agent python jobs  https://review.openstack.org/20471119:38
*** sergmelikyan has quit IRC19:39
sigmavirus24jeblair: thoughts on a reasonable keepalive?19:40
sigmavirus24*keepidle19:40
mordredclarkb: https://github.com/p8952/bocker docker written in 100 lines of bash19:40
*** sergmelikyan has joined #openstack-infra19:41
*** AJaeger has joined #openstack-infra19:42
AJaegercould I get some reviews for https://review.openstack.org/204712 , please? openstack-manuals just got the third DocImpact bug today wrongly assigned by openstack-ansible, so let's fix their config...19:43
*** ayoung has joined #openstack-infra19:43
SpamapSOh nice, our brain dead mellanox firmware fix is in Ironic master already. :-D19:43
SpamapSTheJulia: nice work19:43
jeblairsigmavirus24: can you set it based on the supplied timeout value?  perhaps something like idle=timeout-(count*interval) (with some suitable range bounding) ?19:44
*** heyongli has quit IRC19:45
sigmavirus24jeblair: not quite. urllib3 didn't implement them per-request, they're per-PoolManager19:45
*** AJaeger has quit IRC19:45
*** heyongli has joined #openstack-infra19:47
*** kushal has quit IRC19:47
*** radez is now known as radez_g0n319:47
*** kjnelson has joined #openstack-infra19:47
jeblairsigmavirus24: makes sense; maybe idle=60, interval=15, count=4, to give it up to 2 minutes?  that seems reasonable to me for a cloud.19:48
sigmavirus24jeblair: sounds good19:49
openstackgerritMerged openstack-infra/elastic-recheck: Add query for oslo.db KeyError bug 1476708  https://review.openstack.org/20412319:49
uvirtbotLaunchpad bug 1476708 in oslo.db "KeyError: 'pid' during db pool connection" [High,In progress] https://launchpad.net/bugs/147670819:49
openstackbug 1476708 in oslo.db "KeyError: 'pid' during db pool connection" [High,In progress] https://launchpad.net/bugs/1476708 - Assigned to Roman Podoliaka (rpodolyaka)19:49
uvirtbotLaunchpad bug 1476708 in oslo.db "KeyError: 'pid' during db pool connection" [High,In progress]19:49
*** HeOS has joined #openstack-infra19:50
*** ZZelle_ has quit IRC19:51
*** cdent has quit IRC19:52
*** sergmelikyan has quit IRC19:53
*** sherif has joined #openstack-infra19:55
jeblairsigmavirus24: ah, the default interval and probes are also very high, so we have a total of 131.25 minutes to wait for confirmation, which has just passed19:57
jeblair2015-07-22 17:43:52,898 DEBUG nodepool.ProviderManager: Manager hpcloud-b2 running task <nodepool.provider_manager.ListServersTask object at 0x7f018004f090> (queue: 0)19:57
jeblair2015-07-22 19:55:21,985 DEBUG nodepool.ProviderManager: Manager hpcloud-b2 ran task <nodepool.provider_manager.ListServersTask object at 0x7f018004f090> in 7889.07918096s19:57
jeblairsigmavirus24: and right on time ^19:57
sigmavirus24aha19:57
sigmavirus24Well I'm just writing the commit message for ksc now19:57
*** rlandy has quit IRC19:57
sigmavirus24And then I'll send a dependent review to novaclient19:57
jeblairsigmavirus24: awesome!  and now we have a high degree of confidence that should help with this problem19:58
*** rlandy has joined #openstack-infra19:58
jeblairsigmavirus24: thank you very much!19:58
jeblairclarkb, fungi: ^19:58
jeblair  File "/usr/lib/python2.7/ssl.py", line 260, in read19:59
jeblair    return self._sslobj.read(len)19:59
jeblairerror: [Errno 110] Connection timed out19:59
clarkbjeblair: neat so things got better after 2 hours19:59
jeblairsigmavirus24: and that's the tail end of the traceback for the exception that gets thrown (which is fine for us, nodepool catches that moves on)20:00
jeblairclarkb: well 2 hours, and 11 and a half minutes :)20:00
jeblair(that extra bit almost had me thinking we were wrong)20:01
*** puranamr has quit IRC20:01
sigmavirus24This is after all my fault for not doing it right the first time =P20:02
*** MarkAtwood has joined #openstack-infra20:02
jeblairwell, things are definitely better than they used to be -- and this explains why this only happens when we're looking at it (unlike what used to happen, which is it would just get stuck and stop completely forever)20:02
*** MarkAtwood has quit IRC20:04
jeblairfungi: re presentty: doh.20:04
*** krtaylor has joined #openstack-infra20:04
*** marzif_ has joined #openstack-infra20:05
*** mrmartin has joined #openstack-infra20:07
*** sdake_ is now known as sdake20:07
*** Guest57966 has quit IRC20:07
fungijeblair: clarkb: sigmavirus24: that makes a lot more sense now20:08
*** MarkAtwood has joined #openstack-infra20:08
*** esmute_ has joined #openstack-infra20:08
clarkbso basically when we tried previously clients didn't support it. Now clients do but currently linux defaults are really terrible so overriding in clients as we speak?20:09
fungijeblair: i'm about to send you a patch for an exit key too. i was just going with q but if you have a preference, i'm happy to accommodate20:09
fungialso for a vertical version of the pan transition, what do you think of "scroll" as a keyword?20:09
*** Guest75053 has quit IRC20:10
*** jasondotstar has joined #openstack-infra20:12
*** signed8bit is now known as signed8bit_ZZZzz20:13
sigmavirus24jeblair: fungi clarkb https://review.openstack.org/#/c/204741/ https://review.openstack.org/#/c/204745/20:15
*** dannywilson has joined #openstack-infra20:16
*** pal has joined #openstack-infra20:19
sigmavirus24Also if anyone cares to nominate https://bugs.launchpad.net/python-novaclient/+bug/1477275 for backports to the stable/kilo branches of both, I'll be happy to do cherry-picks of those commits to their stable branches so nodepool can use those clients sooner rather than later20:20
openstackLaunchpad bug 1477275 in python-keystoneclient "KSC's TCPKeepAliveAdapter should specify keepidle to prevent connections from hanging for hours" [Medium,In progress] - Assigned to Ian Cordasco (icordasc)20:20
*** mrmartin has quit IRC20:20
uvirtbotLaunchpad bug 1477275 in python-keystoneclient "KSC's TCPKeepAliveAdapter should specify keepidle to prevent connections from hanging for hours" [Medium,In progress]20:20
uvirtbotLaunchpad bug 1477275 in python-keystoneclient "KSC's TCPKeepAliveAdapter should specify keepidle to prevent connections from hanging for hours" [Medium,In progress] https://launchpad.net/bugs/147727520:20
sigmavirus24(nothing like getting the bug info in triplicate =P)20:20
*** ZZelle_ has joined #openstack-infra20:20
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Remove nova ephemeral temprevert  https://review.openstack.org/20474720:21
*** signed8bit_ZZZzz is now known as signed8bit20:21
*** mrmartin has joined #openstack-infra20:22
*** pal has quit IRC20:24
clarkbwe can probably kick uvirtbot20:25
jeblairfungi: scroll works (tilt works too, to match pan)20:25
*** e0ne has quit IRC20:26
*** dannywilson has quit IRC20:26
fungihah. i like tilt20:26
fungipotentially confusing, but it's an amusing reference20:26
anteayaha ha ha20:26
*** dannywilson has joined #openstack-infra20:26
*** MarkAtwood has quit IRC20:27
* anteaya hears a dop call for the 35mm using wheels on sticks20:27
jeblairi'm having a hard time finding a definitive list of film transitions (which is what i wanted to base those on; that's why it's pan and not wipe (which would have the images stationary))20:28
jeblairi'm thinking of manually setting the kernel defaults on nodepool20:28
*** peristeri has quit IRC20:29
anteayajeblair: http://www.springhurst.org/cinemagic/glossary_terms.htm20:29
fungiyep, i've switched it to tilt. still tweaking CanvasCombine not to crash on me though20:29
jeblairurwid is fun like that20:30
*** Krinkle is now known as Krinkle_20:30
*** gildub has joined #openstack-infra20:30
jeblairokay, i manually changed the kernel keepalive values; not sure what's required for them to take effect (possibly a nodepool restart to open new sockets with them?)20:30
*** hashar has joined #openstack-infra20:31
*** Sukhdev has quit IRC20:31
jeblair(or maybe the next time they fail it will take 2 hours, but then the time after that, they will take effect)20:32
*** kjnelson has quit IRC20:33
*** MarkAtwood has joined #openstack-infra20:33
fungijeblair: _probably_ sticky from socket creation, agreed, but worth finding out20:33
*** eharney has quit IRC20:35
*** jasondotstar has quit IRC20:38
openstackgerritsebastian marcet proposed openstack-infra/openstackid: OIDC - OpenId Connect Implementation  https://review.openstack.org/19845820:38
*** soren has quit IRC20:38
*** baoli has quit IRC20:38
*** gildub has quit IRC20:40
*** mestery has quit IRC20:42
openstackgerritAntoine Musso proposed openstack-infra/zuul: Test please ignore  https://review.openstack.org/20475320:42
*** davideagnello has joined #openstack-infra20:43
*** jsavak has quit IRC20:43
*** jsavak has joined #openstack-infra20:43
hasharholly hell20:43
hasharhello :-}20:43
clarkbhashar: but now I can't ignore it :)20:44
* jeblair hands hashar some smelling salts20:44
hasharI just found out that   gerrit review --label code-review=020:44
* anteaya hopes hashar is saying hello to someone who's first name is holly20:44
greghaynesjeblair: the 11min makes sense, I think the tineout is when kernel starts sending keepaluve probes, takes about 11min for the kernel to decide its deas20:44
jeblairgreghaynes: yep20:44
anteayas/who's/whose20:44
hashardoes not work on my setup :-D  Causes some MySQL related stacktrace due to code-review being a dupe of Code-Review (note the case)20:44
greghayneswow my phone typing is bad20:45
hasharbut it works on openstack setup.  I am wondering whether you guys messed up with the SQL backend database to make it case insensitive20:45
*** jeblair is now known as CarolChanning20:45
CarolChanninghello holly!20:45
*** CarolChanning is now known as jeblair20:45
anteayaha ha ha20:46
anteayaI love her voice20:47
*** jsavak has quit IRC20:47
openstackgerritMerged openstack-dev/hacking: Updated from global requirements  https://review.openstack.org/18682920:48
ianwseems i've once again slept through a crisis :)20:49
jeblair) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin20:49
jeblairhashar: ^20:49
*** puranamr has joined #openstack-infra20:50
hasharjeblair: seems like case sensitive :D20:50
jeblairhashar: i'm pretty sure that means we're case-sensitive20:50
jeblairyeah20:50
jeblairhashar: gerrit version difference?20:50
hasharyeah potentially20:50
hasharthe root cause is definitely  gerrit review being changed to use --label ( https://review.openstack.org/#/c/138132/1/zuul/lib/gerrit.py,unified )20:51
*** kjnelson has joined #openstack-infra20:51
hasharI changed the layout.yaml file to use  Verified:  Code-Review:  everywhere, but that causes a ton of side effects in zuul which normalizes them with lower() anyway.  So approvals (verified) no more matches the submit records (Verified)20:52
jeblairugh20:52
hasharwhich mean I am going to send a lot of patches hehe20:52
jeblairhashar: yeah, so have zuul keep the correct case, and then lowercase as needed for comparisons?20:53
hasharyup20:53
jeblairhashar: or even go as far as to make zuul case-sensitive?20:53
hasharbut it only lower case the version coming from Gerrit API20:53
hasharnot the one from layout.yaml20:53
hasharso you end up with  Verified == verifie20:53
hasharI got some details at https://phabricator.wikimedia.org/T106531  if you want20:53
*** thorst has quit IRC20:54
hasharif label['label'].lower() not in allow_needs   <--- where   allow_needs has  Verified  :-)20:54
bhunter71fungi: thanks for your help earlier.  There's this thing $desired_virtualenv in system-config that's pretty important for centos6.  My value here was out of date.20:55
*** sergmelikyan has joined #openstack-infra20:56
*** agentle_ has quit IRC20:57
*** tongli has quit IRC20:57
*** tongli has joined #openstack-infra20:57
*** thorst has joined #openstack-infra20:58
hasharjeblair: dont worry too much about it. The obvious fix for me is to revert https://review.openstack.org/#/c/138132/1/zuul/lib/gerrit.py,unified  and switch back approvals to be lower cases.  This way they will matches the lower case normalized categories21:00
*** erlon has quit IRC21:00
openstackgerritMerged openstack-infra/irc-meetings: qa-team-meeting: Remove unneeded comment  https://review.openstack.org/20463621:00
*** austin81 has left #openstack-infra21:01
*** davideagnello has quit IRC21:03
*** DericHorn-HP has joined #openstack-infra21:03
*** sergmelikyan has quit IRC21:04
*** aysyd has quit IRC21:05
*** sergmelikyan has joined #openstack-infra21:06
*** krtaylor has quit IRC21:06
*** vlaza1 has joined #openstack-infra21:06
*** vlaza has quit IRC21:06
*** agentle_ has joined #openstack-infra21:08
*** timrc_ has joined #openstack-infra21:14
*** ldnunes has quit IRC21:15
fungibhunter71: right, we update that in our manifest over time. unfortunately the pip provider does unexpected things with ensure=>latest21:16
*** kgiusti has left #openstack-infra21:16
*** david-lyle has quit IRC21:17
*** timrc_ has quit IRC21:17
*** doug-fish has left #openstack-infra21:17
*** jsavak has joined #openstack-infra21:17
*** cbader has quit IRC21:18
*** vlaza1 has quit IRC21:22
*** MarkAtwood has quit IRC21:22
*** dkranz has quit IRC21:25
*** dustins has quit IRC21:25
*** marzif_ has quit IRC21:25
*** timrc has quit IRC21:26
*** timrc has joined #openstack-infra21:26
*** MarkAtwood has joined #openstack-infra21:26
*** dan_ has joined #openstack-infra21:27
*** dan_ is now known as Guest5685521:27
*** mrmartin has quit IRC21:28
*** timrc has quit IRC21:29
*** timrc has joined #openstack-infra21:30
*** Krinkle_ is now known as Krinkle21:30
*** e0ne has joined #openstack-infra21:30
*** heyongli has quit IRC21:32
*** flepied has quit IRC21:33
*** heyongli has joined #openstack-infra21:34
*** kjnelson has quit IRC21:34
*** tongli has quit IRC21:37
*** tongli has joined #openstack-infra21:37
*** notnownikki has quit IRC21:38
anteayaone more day of nova mid-cycle left tomorrow, so far it is very productive21:38
*** Sukhdev has joined #openstack-infra21:38
*** tlian has quit IRC21:39
*** sergmelikyan has quit IRC21:39
*** tonytan4ever has quit IRC21:41
*** heyongli has quit IRC21:41
*** MarkAtwood has quit IRC21:43
*** achanda has quit IRC21:43
*** heyongli has joined #openstack-infra21:43
*** agentle_ has quit IRC21:43
*** tongli has quit IRC21:46
*** annegentle has joined #openstack-infra21:46
*** tongli has joined #openstack-infra21:46
*** jaypipes has quit IRC21:46
*** amitgandhinz has quit IRC21:47
*** tongli has quit IRC21:48
*** mriedem has quit IRC21:49
*** timrc has quit IRC21:51
*** timrc has joined #openstack-infra21:52
*** dizquierdo has joined #openstack-infra21:52
*** sergmelikyan has joined #openstack-infra21:53
openstackgerritlitong01 proposed openstack-infra/project-config: Add new monitor repository for OpenStack monitoring service  https://review.openstack.org/20325921:53
*** dims_ has quit IRC21:53
*** markus_z has quit IRC21:54
*** signed8bit has quit IRC21:54
*** dizquierdo has quit IRC21:55
openstackgerritgreghaynes proposed openstack/diskimage-builder: Perform a booting test for our images  https://review.openstack.org/20463921:56
*** hashar has quit IRC21:57
*** e0ne has quit IRC21:58
*** claudiub has quit IRC21:58
*** jungleboyj has quit IRC22:00
*** patrickeast has quit IRC22:00
*** jasondotstar has joined #openstack-infra22:01
*** Kennan has quit IRC22:02
*** sigmavirus24 is now known as sigmavirus24_awa22:03
*** Kennan has joined #openstack-infra22:03
*** heyongli has quit IRC22:04
*** heyongli has joined #openstack-infra22:05
*** gordc has quit IRC22:06
timrcjeblair: Hey so re: https://review.openstack.org/#/c/204682 - So does setting "host = None" tell gear to listen to localhost?22:07
*** sergmelikyan has quit IRC22:07
*** bknudson has quit IRC22:08
*** bnemec has quit IRC22:09
*** thorst has quit IRC22:10
*** sergmelikyan has joined #openstack-infra22:10
openstackgerritlitong01 proposed openstack-infra/project-config: Add new kiloeyes repository for OpenStack monitoring service  https://review.openstack.org/20325922:11
*** jsavak has quit IRC22:12
*** earlephilhower has quit IRC22:13
openstackgerritMerged openstack-infra/release-tools: Prototype script to release from the new YAML files  https://review.openstack.org/20224022:15
openstackgerritMerged openstack-infra/release-tools: translate glance-store to glance_store  https://review.openstack.org/20278922:15
openstackgerritMerged openstack-infra/release-tools: Move release_notes.py into releasetools package  https://review.openstack.org/20225722:15
*** tvanderwiel has quit IRC22:15
*** achanda has joined #openstack-infra22:15
openstackgerritMerged openstack-infra/release-tools: move list-repos into the releasetools project  https://review.openstack.org/20232022:15
openstackgerritlitong01 proposed openstack-infra/project-config: Add new kiloeyes repository for OpenStack monitoring service  https://review.openstack.org/20325922:16
openstackgerritMerged openstack-infra/release-tools: move commands into their own package  https://review.openstack.org/20232122:16
greghaynestimrc: None makes it default to whatever getaddrinfo() returns22:16
greghaynestimrc: so, should be something localhost-like22:16
*** jasondotstar has quit IRC22:18
*** heyongli has quit IRC22:19
*** glauc____ has quit IRC22:21
openstackgerritKyle Partridge proposed openstack-infra/jenkins-job-builder: Add "commit-status-context" to the build option supporting the most recent version of GHPRB  https://review.openstack.org/20477922:22
openstackgerritgreghaynes proposed openstack/diskimage-builder: Perform a booting test for our images  https://review.openstack.org/20463922:24
*** glauco_ has quit IRC22:24
*** heyongli has joined #openstack-infra22:24
*** jasondotstar has joined #openstack-infra22:26
*** dprince has quit IRC22:31
*** rlandy has quit IRC22:31
*** jasondotstar has quit IRC22:33
*** rossella_s has quit IRC22:37
*** rossella_s has joined #openstack-infra22:37
jeblairshould be something like 'all addresses'22:38
jeblairtimrc, greghaynes: ^22:38
openstackgerritgreghaynes proposed openstack/diskimage-builder: Only warn about missing map files with debug  https://review.openstack.org/20478322:39
*** ashleighfarnham has quit IRC22:43
*** sergmelikyan has quit IRC22:43
*** DericHorn-HP has quit IRC22:45
*** sergmelikyan has joined #openstack-infra22:51
*** DericHorn-HP has joined #openstack-infra22:51
*** DericHorn-HP has quit IRC22:53
*** prad has quit IRC22:53
openstackgerritgreghaynes proposed openstack/diskimage-builder: Perform a booting test for our images  https://review.openstack.org/20463922:53
jheskethMorning22:54
*** ivar-lazzaro has quit IRC22:55
*** dimtruck is now known as zz_dimtruck22:59
pvaneckHi, can this review https://review.openstack.org/#/c/198910/ be approved? It has two +2s and mainly just adds a .gitreview file to a new puppet project23:03
*** krtaylor has joined #openstack-infra23:04
*** ddieterly has joined #openstack-infra23:07
*** camunoz_gone is now known as camunoz23:08
*** sergmelikyan has quit IRC23:08
*** davideagnello has joined #openstack-infra23:09
jeblairpvaneck: done23:11
*** larainema has joined #openstack-infra23:11
pvaneckjeblair: Thanks :)23:12
*** shardy_ has joined #openstack-infra23:12
*** shardy has quit IRC23:14
*** Krinkle has left #openstack-infra23:14
*** shardy_ has quit IRC23:17
*** shardy has joined #openstack-infra23:18
*** jungleboyj has joined #openstack-infra23:21
*** sherif has quit IRC23:22
openstackgerritgreghaynes proposed openstack/diskimage-builder: Perform a booting test for our images  https://review.openstack.org/20463923:23
*** jasondotstar has joined #openstack-infra23:25
*** esmute_ has quit IRC23:25
openstackgerritMerged openstack-infra/puppet-refstack: Add .gitreview & fix puppet-lint errors/warnings  https://review.openstack.org/19891023:27
*** esmute_ has joined #openstack-infra23:27
*** sergmelikyan has joined #openstack-infra23:28
*** Guest56855 has quit IRC23:29
*** sergmelikyan has quit IRC23:30
*** gildub has joined #openstack-infra23:35
*** Sukhdev has quit IRC23:38
greghaynesclarkb: https://jenkins07.openstack.org/job/gate-dib-dsvm-functests-devstack-f21/115/console !23:43
greghaynesSpamapS: ^23:43
greghaynesSpamapS: That is a test that qemu boots a dib built image and verifies we can ping23:44
*** crc32 has quit IRC23:45
greghaynesonly took about 2mins to boot too23:46
clarkbnice23:46
*** vlaza has joined #openstack-infra23:46
*** ajmiller has quit IRC23:47
*** sabeen3 has quit IRC23:48
*** xyang1 has quit IRC23:48
*** achanda has quit IRC23:49
*** davideagnello has quit IRC23:53
openstackgerritGhanshyam Mann proposed openstack-infra/project-config: Move multinode job from experimental to non-voting for Tempest  https://review.openstack.org/20392323:53
*** rm_work is now known as rm_work|away23:53
*** gildub has quit IRC23:54
*** FallenPegasus has joined #openstack-infra23:56
*** hemna is now known as hemnafk23:57
*** dingyichen has joined #openstack-infra23:57
SpamapSgreghaynes: I like it!23:58
*** sergmelikyan has joined #openstack-infra23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!