Thursday, 2016-04-21

melwittwell, that's not entirely true. when new tests are added in a patch I look there to look at them00:00
mtreinishmelwitt: I think it'll be a change in this function:
mtreinishbut I'm not sure what n, t, o, and e are there00:01
*** baoli has joined #openstack-infra00:01
melwittmtreinish: thanks, that's a big help00:01
*** mylu has quit IRC00:02
*** sdake_ has quit IRC00:02
*** mylu has joined #openstack-infra00:03
*** gyee has joined #openstack-infra00:04
*** mylu has quit IRC00:05
*** xarses has joined #openstack-infra00:05
mtreinishmelwitt: once this project config patch lands: we'll also be able to add another visualization for the results to log output00:05
mtreinishmelwitt: it will look something like:
mtreinishit's still kinda rough though00:06
*** Swami has quit IRC00:06
melwittwhoa, pretty cool00:06
*** mylu has joined #openstack-infra00:07
*** xarses_ has joined #openstack-infra00:08
*** mylu has quit IRC00:09
*** Qiming has quit IRC00:09
*** xarses has quit IRC00:10
*** asettle has quit IRC00:11
*** sdake__ has quit IRC00:13
*** MarkAtwood has quit IRC00:14
*** gyee has quit IRC00:14
*** asettle has joined #openstack-infra00:16
dmsimardIs there an ongoing gate issue relative to puppetlabs-release ?00:20
dmsimardpabelanger, EmilienM ^ ?00:20
*** kengo_sakai has joined #openstack-infra00:21
crinkledmsimard: yes there is a fix in progress for infra and EmilienM has a workaround
*** ybathia has quit IRC00:21
crinkledmsimard: so try recheck00:21
dmsimardcrinkle: hmm, I did a recheck after that merged I think, I guess it hasn't propagated yet ?
*** xarses_ has quit IRC00:23
dmsimardah, Jeremy said in the comments it requires a new image to be spun out00:23
dmsimardAt what frequency are the images rebuilt ? I can try later tonight ..00:24
crinkledmsimard: oh that change is for system-config, EmilienM's change was only for puppet-openstack00:24
* crinkle not sure about image build frequency00:24
*** ddieterly[away] is now known as ddieterly00:25
*** kengo_sakai has quit IRC00:25
*** doug-fish has joined #openstack-infra00:25
*** salv-orlando has joined #openstack-infra00:26
*** jamielennox|away is now known as jamielennox00:28
dmsimardoh, so, tomorrow then T_T00:29
dmsimardunless they forcefully updated it or something00:29
dmsimardthanks mgagne :D00:29
*** ddieterly is now known as ddieterly[away]00:30
*** ddieterly[away] is now known as ddieterly00:30
*** salv-orlando has quit IRC00:33
*** asettle has quit IRC00:35
*** avarner__ has joined #openstack-infra00:37
*** julim_ has joined #openstack-infra00:37
*** tphummel has quit IRC00:37
*** kjorgens_ has joined #openstack-infra00:37
*** lakshmiS_ has joined #openstack-infra00:37
*** ajmiller_ has joined #openstack-infra00:37
*** jklare has quit IRC00:38
*** wznoinsk_ has joined #openstack-infra00:38
*** taron2 has joined #openstack-infra00:38
*** kjorgensen has quit IRC00:38
*** taron1 has quit IRC00:38
*** andymccr has quit IRC00:38
*** lakshmiS has quit IRC00:39
*** ajmiller has quit IRC00:39
*** wznoinsk has quit IRC00:39
*** eliqiao_ has quit IRC00:39
*** trey has quit IRC00:39
*** adreznec has quit IRC00:39
*** julim has quit IRC00:39
*** jesusaur has quit IRC00:39
*** thingee has quit IRC00:39
*** tqtran has quit IRC00:39
*** Na3iL has quit IRC00:39
*** avarner_ has quit IRC00:39
*** amrith has quit IRC00:39
*** eliqiao has joined #openstack-infra00:41
*** esker has joined #openstack-infra00:43
*** amrith has joined #openstack-infra00:44
*** jklare has joined #openstack-infra00:44
*** andymccr has joined #openstack-infra00:44
openstackgerritElizabeth K. Joseph proposed openstack-infra/publications: Update slide deck for 2016 Newton Summit
*** trey has joined #openstack-infra00:46
*** stewie925_ has quit IRC00:46
*** adreznec has joined #openstack-infra00:47
*** jamielennox is now known as jamielennox|away00:49
*** jamielennox|away is now known as jamielennox00:51
pleia2pabelanger & docaedo ^^00:51
*** jesusaur has joined #openstack-infra00:52
*** kjorgens_ has quit IRC00:53
*** madhu_ak has quit IRC00:55
*** apoorvad has quit IRC00:55
clarkbpleia2: re your comments on the doc, while great because they point out real issues I juts don't know if I have the steam to address them. Those directions are copy pasta'd from the readme due to jeblair's earlier -100:56
clarkbunfortunately this change has gone around in circles when it started with a very simple direct goal and got pulled in several different other directions and just never seemed to be good enough :(00:57
clarkbI might repropose it as "run nodepool devstack plugin" and only do that00:57
*** tlian2 has joined #openstack-infra00:58
*** mtanino has quit IRC00:58
clarkbwhat do people think about just having a "run nodepool devstack plugin" doc?00:59
clarkboh thta reminds me nibalizer wanted to get his talk reviewed01:00
* clarkb does this before he forgets01:00
clarkbpleia2: should I be reviewing 308700 yet? or do you expect to make more revision?01:01
*** Na3iL has joined #openstack-infra01:01
*** tlian has quit IRC01:01
nibalizerclarkb: i ike docs01:01
clarkbnibalizer: for your talk, how do I render it01:03
*** rossella_s has quit IRC01:03
clarkbsphinx based on requirements.txt01:04
*** rossella_s has joined #openstack-infra01:04
*** bpokorny_ has joined #openstack-infra01:05
clarkbthat seems to have sort of worked with make html01:06
nibalizerya make slides01:07
*** kjorgensen has joined #openstack-infra01:07
pleia2clarkb: it does present a framework that people can nudge into working so I think it's valuable, maybe we just land this and fix it up?01:08
openstackgerritThanh Ha proposed openstack-infra/jenkins-job-builder: Fix CFP configuration for >=2.10.0
pleia2clarkb: 308700 is only WIP because I know I need to update screenshots, feel free go to town on it :)01:09
*** zhurong has joined #openstack-infra01:09
pleia2it likely needs other updates I didn't catch in my first pass too01:09
*** bpokorny has quit IRC01:10
*** bpokorny_ has quit IRC01:10
clarkbok, mostly figured I would offer since I was reviewing. I should pour a beer and review it01:12
*** kjorgensen has quit IRC01:12
*** chuck__ is now known as zul01:15
*** pvaneck has quit IRC01:16
*** nwkarsten has joined #openstack-infra01:17
clarkbpleia2: I would maybe drop the what is openstack slide? perhaps we can assume attendees know what it is at the summit? maybe that is a bad assumption01:17
*** gongysh has joined #openstack-infra01:18
clarkbslides 2 and 3 maybe go away because summit?01:18
*** Qiming has joined #openstack-infra01:19
openstackgerritFei Long Wang proposed openstack-infra/project-config: Support multi backends for Zaqar tempest job
pleia2clarkb: heh, the way we maintain this repo gets a bit in the way here, it's still a generic slide deck but we tag it for a presentation, I guess after the next person could restore those files if they want them back in their next presentation?01:21
*** mylu has joined #openstack-infra01:21
clarkbpleia2: ya, maybe pull them out in a separate commit so a revert is easy?01:21
clarkbor just comment them out day of talk locally01:22
pleia2clarkb: I was going to swap out names as well on day-of01:22
*** nwkarsten has quit IRC01:22
pleia2but I never quite new what was appropriate to do with that every growing list of names01:22
pleia2er, s/new/sure01:22
pleia2multi-tasking \o/01:22
clarkbI think just do whatever is easy01:23
*** doug-fish has quit IRC01:24
openstackgerritDan Prince proposed openstack-infra/project-config: Create an os-net-config ACL
clarkbpleia2: otherwise looks good01:27
pleia2clarkb: great, thanks :)01:27
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: Switch website to use
*** mylu has quit IRC01:30
*** Na3iL has quit IRC01:32
*** ddieterly has quit IRC01:32
*** mylu has joined #openstack-infra01:33
*** adreznec has quit IRC01:39
*** gildub has joined #openstack-infra01:39
*** adreznec has joined #openstack-infra01:41
*** sridhar_ram has quit IRC01:41
*** ddieterly has joined #openstack-infra01:42
*** thingee has joined #openstack-infra01:42
*** mylu has quit IRC01:43
*** mylu has joined #openstack-infra01:43
*** gongysh has quit IRC01:46
openstackgerritIan Wienand proposed openstack-infra/project-config: Fix comment on requirements job
ianw_jeblair: ^ so, yeah01:48
*** salv-orlando has joined #openstack-infra01:49
EmilienMnibalizer: can you look please? to make sure we're doing well01:50
*** zz_dimtruck is now known as dimtruck01:52
*** esker has quit IRC01:52
*** amrith is now known as _amrith_01:55
*** yamamot__ has joined #openstack-infra01:56
*** elo has quit IRC01:57
openstackgerritIan Wienand proposed openstack-infra/project-config: Fix comment on requirements job
*** kengo_sakai has joined #openstack-infra01:58
*** esker has joined #openstack-infra02:01
*** salv-orlando has quit IRC02:05
jheskethinfra-root: the ovh images appear to be outdated. Is there currently an issue uploading?02:06
clarkbjhesketh: no the issue is mordred restarts image builds often enough that we don't really get images updated02:07
*** julim_ has quit IRC02:07
clarkb(well that and ovh upload success rate is not always great)02:07
jheskethovh-bhs1 is 7 days old02:07
jheskethclarkb: should I kick off an upload?02:07
clarkbjhesketh: how old are the images?02:07
clarkbmordred has restarted the image builds almost every day the last week I think02:08
jhesketh9 hours02:08
clarkboh cool we got images built today02:08
* jhesketh is just looking at trusty02:08
clarkbthen ya either there was a restart during uploads after build or we just failed to ovh02:08
clarkbuplaods to ovh do seem to fail relatively frequently, I would try a new upload to get that updated02:08
jheskethalthough looks like most if not all got built today02:09
jheskethclarkb: okay, I'll kick that off in a screen02:09
*** EricGonc_ has quit IRC02:10
lifelessmordred: fungi: your thoughts on
*** jrmyjohn has joined #openstack-infra02:15
*** vikram has joined #openstack-infra02:15
vikramNeed one more +2 for Can someone plz help ;)02:16
jheskethclarkb: ERROR shade.exc: Image creation failed: Error finding address for Unable to establish connection to
*** thorst has joined #openstack-infra02:18
*** cody-somerville has quit IRC02:19
*** nwkarsten has joined #openstack-infra02:19
jheskethclarkb: is that a transient thing we've seen before, or should we ping somebody at ovh?02:19
*** gongysh has joined #openstack-infra02:19
*** jrmyjohn has quit IRC02:19
*** lakshmiS_ has quit IRC02:21
clarkbya thats transient02:21
clarkbit seems to affect bhs1 more than gra102:21
clarkblifeless: +202:21
clarkbTIL that Fuel in addition to everyone else has reimplemented everything02:22
clarkbbut on top of that are -1'ing our changes because we don't do things the way they hav edone them02:22
*** Goneri has joined #openstack-infra02:22
clarkb(also gerrits comment quoting is broken)02:23
*** asettle has joined #openstack-infra02:23
*** nwkarsten has quit IRC02:24
*** tlian2 has quit IRC02:24
jheskethclarkb: so... keep retrying until success?02:26
clarkbjhesketh: ya it seems to succeed eventually02:26
*** esker has quit IRC02:26
jheskethnot ideal, but okay02:26
mordredclarkb: like, what changes?02:27
clarkbmordred: lifeless' from above02:27
mordredclarkb: am I going to be happy about this?02:27
*** ddieterly is now known as ddieterly[away]02:28
*** r-mibu has quit IRC02:29
lifelessmordred: you're goint to +2A it for us all :)02:31
*** tlian has joined #openstack-infra02:32
*** baoli has quit IRC02:32
mordredlifeless: I have done so02:33
mordredlifeless: I have also left a not very nice comment02:33
mordredjust for the record02:33
*** doug-fish has joined #openstack-infra02:34
lifelessmordred: thans :)02:34
*** r-mibu has joined #openstack-infra02:40
*** esker has joined #openstack-infra02:40
*** asettle has quit IRC02:40
*** brad_behle has quit IRC02:42
*** kengo_sa_ has joined #openstack-infra02:42
*** vikram has quit IRC02:42
*** yamahata has joined #openstack-infra02:43
*** baoli has joined #openstack-infra02:44
*** asettle has joined #openstack-infra02:44
*** kengo_sakai has quit IRC02:45
*** resker has joined #openstack-infra02:46
*** esker has quit IRC02:48
*** Goneri has quit IRC02:49
*** baoli has quit IRC02:50
*** EricGonczer_ has joined #openstack-infra02:57
*** resker has quit IRC02:58
*** yamahata has quit IRC03:00
*** EricGonczer_ has quit IRC03:02
*** salv-orlando has joined #openstack-infra03:06
*** sridhar_ram has joined #openstack-infra03:09
*** sridhar_ram has quit IRC03:09
*** salv-orlando has quit IRC03:10
*** esker has joined #openstack-infra03:11
*** thorst has quit IRC03:14
*** thorst has joined #openstack-infra03:14
*** yamamot__ has quit IRC03:15
*** esker has quit IRC03:15
*** asettle has quit IRC03:16
*** Nikolay_St has joined #openstack-infra03:16
*** asettle has joined #openstack-infra03:16
openstackgerritTony Breeds proposed openstack/requirements: Blacklist broken Babel releases
*** Rockyg has quit IRC03:18
tonybAJaeger: when you're back,n,z03:18
openstackgerritIan Wienand proposed openstack/diskimage-builder: Add releasenotes
*** jamielennox is now known as jamielennox|away03:21
*** thorst has quit IRC03:23
*** sdake has joined #openstack-infra03:23
*** gongysh has quit IRC03:23
*** salv-orlando has joined #openstack-infra03:23
*** sdake_ has joined #openstack-infra03:24
*** gongysh has joined #openstack-infra03:25
*** doug-fish has quit IRC03:25
*** mylu has quit IRC03:27
*** sdake has quit IRC03:27
*** xarses_ has joined #openstack-infra03:27
*** doug-fish has joined #openstack-infra03:27
*** esker has joined #openstack-infra03:28
*** ddieterly[away] has quit IRC03:28
*** salv-orlando has quit IRC03:29
*** links has joined #openstack-infra03:30
*** sridhar_ram has joined #openstack-infra03:32
*** sridhar_ram1 has joined #openstack-infra03:34
*** zaro has joined #openstack-infra03:34
*** andreww has joined #openstack-infra03:36
*** sridhar_ram has quit IRC03:36
*** cody-somerville has joined #openstack-infra03:39
*** xarses_ has quit IRC03:39
*** tlian has quit IRC03:41
*** amotoki has joined #openstack-infra03:42
*** mylu has joined #openstack-infra03:43
*** esker has quit IRC03:45
*** asettle has quit IRC03:48
*** asettle has joined #openstack-infra03:48
*** esker has joined #openstack-infra03:55
openstackgerritIan Wienand proposed openstack-infra/system-config: Fix URL detection in testresult match
*** keedya has quit IRC03:56
*** harlowja_at_home has joined #openstack-infra03:57
*** yamamot__ has joined #openstack-infra03:59
*** nwkarsten has joined #openstack-infra03:59
*** Nikolay_St has quit IRC04:00
openstackgerritIan Wienand proposed openstack-infra/project-config: Fix comment on requirements job
*** amotoki has quit IRC04:01
openstackgerritRyo Tagami proposed openstack-infra/jenkins-job-builder: Adding support for Jenkins OpenStack Cloud Plugin
*** nwkarsten has quit IRC04:04
*** esker has quit IRC04:04
*** maishsk has quit IRC04:05
*** bapalm has quit IRC04:10
*** esker has joined #openstack-infra04:10
*** amotoki has joined #openstack-infra04:11
*** bapalm has joined #openstack-infra04:12
*** sdake_ has quit IRC04:13
*** esker has quit IRC04:15
*** esker has joined #openstack-infra04:17
*** fedexo has joined #openstack-infra04:20
*** thorst has joined #openstack-infra04:20
*** maishsk has joined #openstack-infra04:22
*** kzaitsev_mb has joined #openstack-infra04:24
*** esker has quit IRC04:25
openstackgerritClark Boylan proposed openstack-infra/project-config: Cross test constraints and project unittests.
*** doug-fish has quit IRC04:26
*** asettle has quit IRC04:27
*** maishsk has quit IRC04:27
*** asettle has joined #openstack-infra04:27
*** salv-orlando has joined #openstack-infra04:28
*** thorst has quit IRC04:28
*** kzaitsev_mb has quit IRC04:30
openstackgerritMerged openstack/diskimage-builder: Move selinux restore to end of finalise
*** vhosakot has quit IRC04:30
*** mylu has quit IRC04:32
*** rkukura has joined #openstack-infra04:32
*** dahoo has quit IRC04:32
*** mylu has joined #openstack-infra04:33
*** maishsk has joined #openstack-infra04:33
*** sdake has joined #openstack-infra04:36
tonybclarkb: Thanks for that's awesome!04:37
*** salv-orlando has quit IRC04:37
*** rlandy has quit IRC04:37
*** harlowja_at_home has quit IRC04:38
*** maishsk has quit IRC04:38
*** esker has joined #openstack-infra04:40
*** hdd has quit IRC04:40
*** zhenguo__ has joined #openstack-infra04:40
*** zhenguo_ has quit IRC04:43
*** zhenguo__ is now known as zhenguo_04:43
*** psachin has joined #openstack-infra04:44
prometheanfireianw_: when merges I'll rebase my gentoo patch for that element04:48
prometheanfireianw_: also, did you just +2 your own patch? :P04:48
*** rcernin has quit IRC04:48
prometheanfireianw_: not that it seems there are enough cores to get +2 on stuff in DIB04:48
prometheanfireor active ones at lest04:48
*** doug-fish has joined #openstack-infra04:50
*** sridhar_ram1 has quit IRC04:52
*** doug-fish has quit IRC04:54
ianw_clarkb: if around; did you test with the fix to the EPEL element ( or just manually fix up your wrongly installed epel6 issue?04:55
*** maestro1 has joined #openstack-infra04:57
*** salv-orlando has joined #openstack-infra04:58
openstackgerritIan Wienand proposed openstack/diskimage-builder: yum-minimal: strip locale archive
*** sdake_ has joined #openstack-infra05:01
*** salv-orlando has quit IRC05:02
*** kengo_sakai has joined #openstack-infra05:03
*** sdake has quit IRC05:03
*** rossella_s has quit IRC05:03
openstackgerritMerged openstack/diskimage-builder: Add releasenotes
*** rossella_s has joined #openstack-infra05:04
*** maestro1 has quit IRC05:05
openstackgerritvinay kumar muddu proposed openstack/diskimage-builder: Included "sudo" for devuser element
*** maestro1 has joined #openstack-infra05:06
*** salv-orlando has joined #openstack-infra05:07
*** jaosorior has joined #openstack-infra05:07
*** mylu has quit IRC05:09
*** ayoung has quit IRC05:10
*** maestro1 has quit IRC05:11
*** cody-somerville has quit IRC05:13
*** rcernin has joined #openstack-infra05:17
openstackgerritRyo Tagami proposed openstack-infra/jenkins-job-builder: Adding support for Jenkins OpenStack Cloud Plugin
*** psachin has quit IRC05:21
*** oanson has joined #openstack-infra05:22
ianw_pabelanger: when around, gate-openstackci-beaker-centos-7 failing -> file /etc/pki/rpm-gpg/RPM-GPG-KEY-puppetlabs from install of puppetlabs-release-7-12.noarch conflicts with file from package puppetlabs-release-7-10.noarch05:23
*** e0ne has joined #openstack-infra05:23
ianw_guess new packages are broken or something?05:23
openstackgerritMerged openstack/diskimage-builder: Really remove all interfaces in dhcp-all-ifaces
openstackgerritMerged openstack/diskimage-builder: Handle unconfigured interfaces for dhcp-all-ifaces
openstackgerritMatthew Thode proposed openstack-infra/bindep: Add Gentoo support to bindep
prometheanfirefungi: now with more tests
*** thorst has joined #openstack-infra05:26
*** ianw_ has quit IRC05:28
*** ramishra has quit IRC05:28
*** ianw_ has joined #openstack-infra05:29
*** ianw_ has quit IRC05:30
*** ramishra has joined #openstack-infra05:31
*** ianw has quit IRC05:31
*** asettle has quit IRC05:31
*** ianw has joined #openstack-infra05:31
*** psachin has joined #openstack-infra05:33
*** thorst has quit IRC05:33
*** ianw has quit IRC05:34
*** ianw has joined #openstack-infra05:34
*** esker has quit IRC05:35
*** sridhar_ram has joined #openstack-infra05:36
*** sridhar_ram has quit IRC05:37
ianwgreghaynes: i've cleared out a bunch of stuff in dib; it would be great if you could get a release out tomorrow your time before the build.  i don't want to do it now because i'm about to disappear05:40
*** esker has joined #openstack-infra05:40
*** asettle has joined #openstack-infra05:41
*** sridhar_ram has joined #openstack-infra05:44
*** sridhar_ram1 has joined #openstack-infra05:50
*** roxanagh_ has joined #openstack-infra05:50
*** sridhar_ram has quit IRC05:50
*** tphummel has joined #openstack-infra05:51
*** roxanagh_ has quit IRC05:52
*** andymaier has joined #openstack-infra05:52
*** roxanagh_ has joined #openstack-infra05:53
*** kengo_sakai has quit IRC05:53
*** esker has quit IRC05:55
openstackgerritTony Breeds proposed openstack-infra/release-tools: Hide the zuul-cloner output unless there is an error
*** esker has joined #openstack-infra05:56
prometheanfireianw: thanks, are you and greghaynes the only active cores in DIB?05:57
*** roxanagh_ has quit IRC05:58
*** maishsk has joined #openstack-infra05:59
*** maishsk has quit IRC06:01
*** maishsk has joined #openstack-infra06:05
AJaegerianw: that puppetlabs change is fixed already (we hope) but needs new images...06:08
AJaegerianw: see change Idc9f40cdc8a9e47bdf9baa141c09e97f7884d3cf for system-config06:08
*** fitoduarte has joined #openstack-infra06:09
*** Yarboa has joined #openstack-infra06:09
*** aeng has quit IRC06:11
*** kushal has quit IRC06:12
*** dimtruck is now known as zz_dimtruck06:14
*** esker has quit IRC06:15
openstackgerritAkihiro Motoki proposed openstack-infra/project-config: Skip networking-nec python34 job for liberty and kilo
*** yamahata has joined #openstack-infra06:16
*** andreykurilin has quit IRC06:17
*** esker has joined #openstack-infra06:17
*** asselin__ has joined #openstack-infra06:18
*** asselin_ has quit IRC06:20
*** andymaier_ has joined #openstack-infra06:21
*** mrmartin has joined #openstack-infra06:23
openstackgerritMehdi Abaakouk (sileht) proposed openstack-infra/project-config: gnocchi: Add grenade jobs
*** scheuran has joined #openstack-infra06:24
*** andymaier has quit IRC06:25
*** ihrachys has joined #openstack-infra06:26
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints
*** mikelk has joined #openstack-infra06:27
*** asettle has quit IRC06:29
*** asettle has joined #openstack-infra06:29
*** elo has joined #openstack-infra06:31
*** angdraug has joined #openstack-infra06:32
*** Nikolay_St has joined #openstack-infra06:33
*** jrmyjohn has joined #openstack-infra06:34
*** aarefiev has quit IRC06:35
*** yolanda has quit IRC06:37
*** armax has quit IRC06:37
masayukigmtreinish: timothyb89: there's something wrong about : "The last data in the subunit2sql database, from "2016-04-18T19:38:09.000Z", is >1 day old. There might be an issue with result collection."06:38
*** peterlisak has quit IRC06:38
*** jtomasek has quit IRC06:38
*** jrmyjohn has quit IRC06:39
*** peterlisak has joined #openstack-infra06:39
*** angdraug has quit IRC06:39
*** nadya has joined #openstack-infra06:40
*** ihrachys has quit IRC06:40
*** nadya has quit IRC06:41
*** mylu has joined #openstack-infra06:43
*** asettle has quit IRC06:43
*** asettle has joined #openstack-infra06:47
*** yolanda has joined #openstack-infra06:47
*** mylu has quit IRC06:48
*** fitoduarte has quit IRC06:48
*** kushal has joined #openstack-infra06:50
*** nadya has joined #openstack-infra06:50
*** roxanagh_ has joined #openstack-infra06:54
*** thorst has joined #openstack-infra06:54
*** tphummel has quit IRC06:55
*** gildub_ has joined #openstack-infra06:56
*** roxanagh_ has quit IRC06:58
*** thorst has quit IRC06:59
*** aarefiev has joined #openstack-infra07:00
*** e0ne has quit IRC07:01
*** nadya has quit IRC07:02
*** HeOS has joined #openstack-infra07:03
*** mrmartin has quit IRC07:03
*** tesseract has joined #openstack-infra07:03
*** gildub has quit IRC07:03
*** gildub_ has quit IRC07:04
*** tesseract is now known as Guest1450907:04
*** gildub has joined #openstack-infra07:04
*** e0ne has joined #openstack-infra07:07
*** arxcruz has joined #openstack-infra07:09
*** matrohon has joined #openstack-infra07:13
openstackgerritYAMAMOTO Takashi proposed openstack-infra/project-config: taas: Skip unit tests for doc-only changes
*** ekonsdo_ has joined #openstack-infra07:15
*** fedexo has quit IRC07:18
*** woodster_ has quit IRC07:18
*** permalac has quit IRC07:19
*** flepied1 has quit IRC07:21
*** vincentll has joined #openstack-infra07:21
*** zeih has joined #openstack-infra07:21
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/shade: Add Designate recordsets support
*** schipiga has joined #openstack-infra07:23
*** nmagnezi has joined #openstack-infra07:23
*** cathy__ has joined #openstack-infra07:24
schipigaHi guys! I have a very helpful patch to horizon config Whom may I request to review that to be sure it will be merged as quick as possible?07:25
cathy__hi, i want to delete branches that no longer in use. Who can help me?07:25
*** Aegil_ is now known as Aegil07:26
Aegilcathy__: do you mean local git branches? for that you can use git branch -d mybranchname07:27
cathy__not in local, branches have been committed and merged07:28
cathy__i need to delete branches named 6.1, 7.0 and kilo( not Kilo )07:31
*** shardy has joined #openstack-infra07:32
*** e0ne has quit IRC07:32
*** infra-red has joined #openstack-infra07:33
*** asettle has quit IRC07:33
*** dizquierdo has joined #openstack-infra07:33
*** infra-re_ has joined #openstack-infra07:34
*** ekonsdo_ has quit IRC07:34
*** ihrachys has joined #openstack-infra07:34
*** ihrachys has quit IRC07:35
*** infra-red has quit IRC07:37
AJaegerschipiga: there're not only guys in this channel, we value our female team members as well07:39
AJaegercathy__: any infra root can - but it's a manual step. Best to send an email to openstack-infra mailing list asking for it.07:39
schipigaoh my appoligies, I didn't want to offend nobody07:40
vincentllis there a way to see the status of the CI runners? like how many are there, which one is currently running my review, etc.07:41
AJaegerschipiga: I commented on your review.07:42
cathy__AJaeger, thanks.07:43
AJaegervincentll: gives some stats as well as grafana.openstack.org07:43
vincentllthanks AJaeger07:43
AJaegervincentll: we start a new node for each job, so no need to know the status of them ;)07:44
*** jlanoux has joined #openstack-infra07:46
openstackgerritMerged openstack-infra/tripleo-ci: Fix REPO_PREFIX for stable releases in
*** asettle has joined #openstack-infra07:48
*** cathy__ has quit IRC07:50
*** hashar has joined #openstack-infra07:50
*** andymaier_ has quit IRC07:50
*** mrmartin has joined #openstack-infra07:52
*** flepied has joined #openstack-infra07:52
*** hogepodge has quit IRC07:52
*** ifarkas has joined #openstack-infra07:54
*** hogepodge has joined #openstack-infra07:54
*** abregman has joined #openstack-infra07:56
schipigaAJaeger: I attached a link about separated logs to requested patch07:56
*** esker has quit IRC07:56
*** thorst has joined #openstack-infra07:56
*** esikachev has joined #openstack-infra07:57
*** vikram_ has joined #openstack-infra07:57
vikram_Need one more +2 for Can someone plz help ;)07:58
openstackgerritMichele Paolino proposed openstack-infra/project-config: New project devstack-plugin-tar-installer
openstackgerritSergei Chipiga proposed openstack-infra/project-config: Publish test_reports dir
*** zzzeek has quit IRC08:00
*** zzzeek has joined #openstack-infra08:00
AJaegervikram_: what kind of change is it? Please explain, I don't want to open a review for one of the 70+ infra repos that I don't review...08:00
*** Na3iL has joined #openstack-infra08:01
vikram_AJaeger: This change already has your approval08:01
vikram_AJaeger: It's about creating "neutron-dynamic-routing" project08:01
*** jed56 has joined #openstack-infra08:02
vikram_AJaeger: Sorry for not mentioning08:02
*** asettle has quit IRC08:02
AJaegervikram_: Ah, that one. Finally ready ;)08:02
vikram_AJaeger: Yes, after a long tidy wait :)08:02
*** thorst has quit IRC08:03
*** HeOS has quit IRC08:04
*** esker has joined #openstack-infra08:04
*** ramishra has quit IRC08:08
*** ramishra has joined #openstack-infra08:09
*** esker has quit IRC08:10
schipigaAJaeger, should I ask to review my config patch somebody from horizon core team, for ex. robcresswell, to be sure that horizon infra needs it ?08:10
*** Na3iL has quit IRC08:12
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/shade: Add Designate recordsets support
*** jistr has joined #openstack-infra08:13
*** ociuhandu has quit IRC08:14
*** jlanoux has quit IRC08:16
*** jlanoux has joined #openstack-infra08:17
*** esker has joined #openstack-infra08:20
*** abregman is now known as abregman|pto08:21
*** jaosorior has quit IRC08:22
*** nadya has joined #openstack-infra08:25
*** esker has quit IRC08:25
vikram_ajo: ping08:26
vikram_sorry pinged on wrong window08:26
*** markvoelker has quit IRC08:27
*** jordanP has joined #openstack-infra08:30
matttis it no longer possible to view jenkins log as a job is running ?08:30
*** jaosorior has joined #openstack-infra08:30
*** jrmyjohn has joined #openstack-infra08:30
*** jtomasek has joined #openstack-infra08:31
*** infra-red has joined #openstack-infra08:33
*** infra-re_ has quit IRC08:34
*** derekh has joined #openstack-infra08:34
*** dtardivel has joined #openstack-infra08:35
*** jrmyjohn has quit IRC08:35
*** dtantsur|afk is now known as dtantsur08:36
*** esker has joined #openstack-infra08:40
*** mylu has joined #openstack-infra08:44
*** electrofelix has joined #openstack-infra08:44
*** dingyichen has quit IRC08:44
*** infra-red has quit IRC08:45
jamespageyolanda, sorry - mean't to ping here not in -dev...08:47
*** infra-red has joined #openstack-infra08:47
*** mrmartin has quit IRC08:48
*** infra-red has quit IRC08:48
lifelessmattt: not at the moment - security issue08:49
lifelessmattt: see the mail to openstack-dev08:49
*** mylu has quit IRC08:49
*** infra-red has joined #openstack-infra08:50
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/shade: Add Designate recordsets support
jamespageyolanda, so I'm trying to push new stable branchs across the charm-* repositories08:50
jamespageyolanda, failed on my first08:50
jamespage! [remote rejected] stable/16.04 -> stable/16.04 (failed to lock)08:50
openstackjamespage: Error: "remote" is not a valid command.08:50
jamespagethats for charm-ceilometer...08:50
jamespageany ideas? (or anyone else for that matter)08:50
*** schipiga_ has joined #openstack-infra08:51
*** infra-red has quit IRC08:53
AJaegerschipiga: yes, please get some feedback from other horizon cores on that - I don't want that to merge and the other never having a chance to go in.08:54
AJaegerjamespage: what command did you push?08:54
AJaegerdid you use, I mean08:54
*** esker has quit IRC08:55
*** ihrachys has joined #openstack-infra08:55
*** maishsk_ has joined #openstack-infra08:56
*** maishsk has quit IRC08:56
*** maishsk_ is now known as maishsk08:56
*** asettle has joined #openstack-infra08:57
jamespageAJaeger, git push gerrit stable/16.0408:57
AJaegerjamespage: that should work  in general08:57
*** infra-red has joined #openstack-infra08:58
jamespageAJaeger, the repositories do have an existing 'stable' branch - might that be impacting things?08:58
AJaegerjamespage: I assume you're in charm-release group and you signed your commit?08:58
jamespageAJaeger, ah wait...08:58
*** niska has quit IRC08:58
yolandahi, am back... i got internet problems this morning again08:59
*** imcsk8 is now known as imcsk8|zZz08:59
jamespageAJaeger, so I am in the charm-release group; there is no commit to sign - this is just a branch - I've not tagged yet...08:59
AJaegerjamespage: is reference for steps08:59
*** infra-red has quit IRC09:00
*** infra-re_ has joined #openstack-infra09:00
AJaegerjamespage: explains how to create a branch - use the webui09:00
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: Run tempest tests with periodic jobs
jamespageAJaeger, I was hoping not to have to use the webui for all 23 repositors09:00
jamespageAJaeger, tried with charm-ceilometer09:01
AJaegerjamespage: I'm not aware of any alternative ;(09:01
jamespageCannot create branch "refs/heads/stable/16.04" since it conflicts with branch "refs/heads/stable".09:01
*** thorst has joined #openstack-infra09:01
*** esker has joined #openstack-infra09:01
jamespageAJaeger, pushing a branch from the CLI is in that document09:01
*** asettle has quit IRC09:01
jamespageAJaeger, so it is the existing stable branch that is causing the problem...09:01
AJaegerjamespage: you might want to talk with the release team (channel #openstack-release) whether they have a way to create branches for several projects09:02
*** amotoki has quit IRC09:02
AJaegerjamespage: Ah, see now. Didn't know about the push there as well...09:02
AJaegerjamespage: Then I can't help further - best ask one of our infra-roots later (US time)09:02
*** infra-re_ has quit IRC09:03
*** rossella_s has quit IRC09:03
*** esker has quit IRC09:06
*** vikram_ has quit IRC09:06
*** rossella_s has joined #openstack-infra09:06
*** zeih has quit IRC09:06
*** niska has joined #openstack-infra09:06
*** esker has joined #openstack-infra09:07
*** mpaolino has joined #openstack-infra09:07
*** thorst has quit IRC09:09
*** zeih has joined #openstack-infra09:13
yolandaso beaker tests still not passing. My change was merged, but we'll need to refresh images09:15
yolandagoing to check09:15
matttlifeless: will do, thanks09:15
*** sridhar_ram has joined #openstack-infra09:15
*** esker has quit IRC09:16
aarefievhi! could you guys help09:17
aarefievExperimental job doesn't triggered anymore:
*** mpaolino has quit IRC09:17
*** gildub has quit IRC09:17
aarefievWhat goes wrong?09:17
AJaegeraarefiev: did you check status.openstack/zuul already?09:17
aarefievAJaeger: yep, nothing09:18
*** sridhar_ram has quit IRC09:18
*** sridhar_ram1 has quit IRC09:19
*** HeOS has joined #openstack-infra09:19
AJaegerno idea - everything else looks fine09:20
*** ociuhandu has joined #openstack-infra09:20
jamespageAJaeger, yolanda: do either of you guys have the ability to delete branches in gerrit?09:20
yolandajamespage, i can09:20
yolandaaarefiev, let me take a look at logs09:21
* jamespage thinks up a plan09:21
jamespageyolanda, ok - so how does this sound09:22
yolandaaarefiev, i sent a check experimental to one of my jobs, and worked09:22
openstackgerritSergei Chipiga proposed openstack-infra/project-config: Publish test_reports dir
jamespagethe current stable branch in the charm-* repositories is blocking creation of any stable/XXX branches for new stables...09:22
jamespageI'll tag the latest commit on the existing stable branches; we can then delete them and I'll re-create as stable/16.01 allows us to create stable/16.04 as well09:23
jamespageyolanda, does that sound reasonable?09:23
*** zeih has quit IRC09:23
aarefievyolanda: hmm, it for sure worked for me on last week on another patch, what I can do here?09:23
*** wznoinsk_ has quit IRC09:24
openstackgerritMerged openstack-infra/tripleo-ci: Switch website to use
yolandajamespage, i think so09:24
*** wznoinsk_ has joined #openstack-infra09:24
yolandajamespage, so let me know if you are ready, and let me know the list of projects affected, so i can delete stable branch09:25
yolandaaarefiev, going to take a look at zuul logs, to see if that's ignoring for some reason09:25
aarefievyolanda: thanks09:26
*** markvoelker has joined #openstack-infra09:27
yolandaaarefiev, No jobs for change <Change 0x7fd25109e510 293926,4>09:27
*** wznoinsk_ is now known as wznoinsk09:28
yolandafor some reason it's just ignoring the experimental jobs there. Is something that may have changed in terms of your job config in last week?09:28
frickleraarefiev: yolanda: I think this is skipped due to
yolandafrickler, ah, you are right09:29
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Add temporary script to collect the status of periodic jobs
aarefievhmm,  how does this work before09:30
yolandatests folder is totally ignored on this job09:30
yolandaaarefiev, is there something that may have changed in that config?09:30
*** tosky_ has joined #openstack-infra09:30
aarefievyolanda: we changed only tempest regex09:31
*** mpaolino has joined #openstack-infra09:31
AJaegeraarefiev: Add a file *outside* of your tests directory to your change so that the conditions there do not match anymore and then run check experimental again09:31
*** doug-fish has joined #openstack-infra09:31
AJaegerfrickler: great find, thanks!09:31
aarefievAJaeger: already doing09:32
*** esker has joined #openstack-infra09:32
aarefievthanks frickler09:32
dtantsurgood find, yeah09:32
AJaegeraarefiev: just touching "do-not-merge" would be enough ;)09:32
dtantsuraarefiev, seems like we should change this "skip-if" regex09:32
*** markvoelker has quit IRC09:32
jamespageyolanda, ok tags created and pushed - this is the list of repos I need the stable branch deleted from  -
AJaegerdtantsur: regex or exclusion list09:32
dtantsurwe should ignore ironic_inspector/test/unit, not just ironic_inspector/test09:33
*** sambetts|afk is now known as sambetts09:33
yolandajamespage, ack. Going to delete them09:33
jamespageyolanda, thankyou :-)09:34
*** zeih has joined #openstack-infra09:35
aarefievdtantsur: yep, looks like I didn't run this job after moved tempest plugin (09:35
aarefievmy fail09:35
aarefievyolanda, AJaeger, frickler: thanks for help09:36
*** doug-fish has quit IRC09:36
*** yamahata has quit IRC09:37
*** Guest15985 has quit IRC09:37
*** tosky_ is now known as tosky09:38
yolandajamespage, done09:40
jamespageyolanda, thanks!09:41
yolandajamespage, aarefiev , glad to help09:42
*** hashar has quit IRC09:43
electrofelixanyone around to review a chance for git-review? I've hit an issue with running the tests where high pids are allowed, which is also fixed by
*** zeih has quit IRC09:45
*** esker has quit IRC09:45
AJaegeryolanda, jamespage, infra-root: Will that deletion of stable cause any problems for our infra systems?09:46
AJaegerI get now an error updating the charm repos like
yolandaAJaeger, i didn't think about that, let me double check09:48
yolandai've had quite problems in the past with non-existing master branches, but i wasn't expecting stable to give problems09:48
AJaegergit remote prune origin helps with that...09:49
openstackgerritMerged openstack-infra/project-config: New project devstack-plugin-tar-installer
*** zhurong has quit IRC09:49
*** _degorenko|afk is now known as degorenko09:49
*** vgridnev has quit IRC09:49
yolandaAJaeger, so when i clone that now, stable is not existing anymore. Maybe you had an old copy of that repo where stable still existed?09:50
AJaegeryolanda: yes, that was an old copy.09:50
AJaegerSo, might not hit us - let's see...09:50
*** zhurong has joined #openstack-infra09:51
yolandareplication correctly removed stable branch, as far as i can see09:51
jamespageyolanda, -> ?09:51
AJaegerbut might hit community members - so jamespage best sends out an email09:51
jamespageyeah I see that as well..09:51
jamespageAJaeger, will do...09:51
yolandajamespage, it should be replicated to github as well09:51
*** zhurong has quit IRC09:51
jamespageyolanda, yah it is09:52
jamespageditto the new stable branch I just created...09:52
*** maishsk has quit IRC09:52
*** vgridnev has joined #openstack-infra09:53
*** vgridnev has quit IRC09:54
openstackgerritMerged openstack-infra/project-config: taas: Skip unit tests for doc-only changes
openstackgerritMerged openstack-infra/project-config: Fix comment on requirements job
openstackgerritMerged openstack-infra/project-config: Correct function name to run the right script
*** amotoki has joined #openstack-infra09:59
*** vgridnev has joined #openstack-infra10:00
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/shade: Add Designate recordsets support
*** avarner_ has joined #openstack-infra10:02
*** avarner__ has quit IRC10:02
*** esker has joined #openstack-infra10:05
*** thorst has joined #openstack-infra10:06
*** zeih has joined #openstack-infra10:08
*** _amrith_ is now known as amrith10:10
*** e0ne has joined #openstack-infra10:12
*** yamamot__ has quit IRC10:12
*** asselin__ has quit IRC10:12
*** asselin__ has joined #openstack-infra10:13
*** thorst has quit IRC10:13
*** nikhil has quit IRC10:14
*** nikhil has joined #openstack-infra10:14
*** esikachev has quit IRC10:14
*** esker has quit IRC10:15
*** asselin_ has joined #openstack-infra10:15
*** esker has joined #openstack-infra10:17
*** salv-orl_ has joined #openstack-infra10:19
*** asselin__ has quit IRC10:19
penguinologHi/ I see job fails on zuul: Unable to connect to -> Some index files failed to download. They have been ignored, or old ones used instead. -> package install fails10:21
*** Guest14509 has quit IRC10:22
*** salv-orlando has quit IRC10:22
*** sdague has joined #openstack-infra10:24
rcarrillocruzi wonder if the docs job on my change failed due to ^10:24
rcarrillocruzcan't see the reason (yet)10:24
openstackgerritAnton Arefiev proposed openstack-infra/project-config: Don't skip expensive jobs for Inspector tempest tests
rcarrillocruztill other jobs bail out10:24
*** Guest14509 has joined #openstack-infra10:24
*** zeih has quit IRC10:25
*** esker has quit IRC10:25
yolandamirrors failing again?10:25
penguinologyolanda, looks like yes :( , as I see in logs10:26
yolandapenguinolog, ok, let me check mirror status10:27
yolandayep, mirror not responsive again10:28
*** esker has joined #openstack-infra10:31
yolandaso our openstackci account is failing again. I'm going to remove ovh servers from nodepool until we fix it10:32
*** asselin__ has joined #openstack-infra10:33
*** kzaitsev_mb has joined #openstack-infra10:34
yolanda#status notice OVH servers are down, we are working to solve it. This will cause that jobs queue is processed slowly, please have patience.10:35
*** maishsk has joined #openstack-infra10:35
yolanda#status notice OVH servers are down, we are working to solve it. This will cause that jobs queue is processed slowly, please have patience.10:35
openstackstatusyolanda: sending notice10:35
*** asselin_ has quit IRC10:36
-openstackstatus- NOTICE: OVH servers are down, we are working to solve it. This will cause that jobs queue is processed slowly, please have patience.10:37
rcarrillocruzregardless of ovh, my func tests are still wrong :/10:38
*** ociuhandu has quit IRC10:38
yolandasame problem with ovh, i'm checking with Jean-Daniel10:38
openstackstatusyolanda: finished sending notice10:38
schipigaAJaeger: could you please review again I had got approve of two core developers10:39
*** baoli has joined #openstack-infra10:41
*** Qiming has quit IRC10:41
*** baoli_ has joined #openstack-infra10:44
*** mylu has joined #openstack-infra10:45
*** esker has quit IRC10:46
*** yamamoto_ has joined #openstack-infra10:46
*** baoli has quit IRC10:47
*** esker has joined #openstack-infra10:48
openstackgerritAndrey Pavlov proposed openstack-infra/project-config: ACL: allow to create branches for gce-api project
*** e0ne has quit IRC10:49
*** asselin_ has joined #openstack-infra10:49
*** esikachev has joined #openstack-infra10:49
*** mylu has quit IRC10:50
*** yamamoto_ has quit IRC10:51
*** asselin__ has quit IRC10:51
*** jaosorior has quit IRC10:52
*** jaosorior has joined #openstack-infra10:53
*** yamamoto_ has joined #openstack-infra10:54
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/shade: Add Designate recordsets support
*** imcsk8|z1z has joined #openstack-infra10:55
openstackgerrityolanda.robla proposed openstack-infra/project-config: Revert "Revert "Temporarily disable OVH GRA, as we are seeing failures""
openstackgerrityolanda.robla proposed openstack-infra/project-config: Revert "Revert "Temporarily disable ovh-bhs provider""
*** imcsk8|zZz has quit IRC10:56
yolandaAJaeger ^10:56
*** ociuhandu has joined #openstack-infra10:57
*** kushal has quit IRC10:58
*** asettle has joined #openstack-infra10:59
*** asettle has quit IRC10:59
*** mpaolino has quit IRC11:01
*** esikachev has quit IRC11:01
*** esikachev has joined #openstack-infra11:02
sdagueI'm trying to figure out why some of the nova post jobs didn't seem to work11:02
*** zeih has joined #openstack-infra11:02
sdagueis there a good place to look for logs there? this is nova-api-ref11:03
*** lucasagomes is now known as lucas-hungry11:05
*** icey_ has joined #openstack-infra11:05
*** amotoki has quit IRC11:07
*** icey has quit IRC11:08
*** sdake_ has quit IRC11:08
*** kzaitsev_mb has quit IRC11:08
*** sdake has joined #openstack-infra11:08
dims_sdague do you have a specific review? we'll need the change-id to go hunt for the logs11:09
sdaguedims_: yeh, though it's the post job11:09
*** thorst has joined #openstack-infra11:11
*** kzaitsev_mb has joined #openstack-infra11:11
dims_sdague : this one?
dims_sdague : we use the last 2 digits "64" and then "/" then the review number to formulate the url11:12
*** yamamoto_ has quit IRC11:12
*** Hal has joined #openstack-infra11:16
*** yamamoto_ has joined #openstack-infra11:16
*** Hal is now known as Guest6478111:16
*** doug-fish has joined #openstack-infra11:17
*** mpaolino has joined #openstack-infra11:17
*** thorst has quit IRC11:19
*** Guest64781 has quit IRC11:20
sdaguedims_: I think so11:21
sdaguethat should have also uploaded content11:21
dims_sdague : job thinks it did11:22
dims_dims_ : [SCP] uploading file: '/srv/static/docs-draft/64/307164/7/gate/gate-nova-api-ref/36e6a79/api-ref/build/html/searchindex.js'11:22
sdagueyeh, the content on the target site is definitely not right though11:22
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Don't try to update tasks if they don't have an ID
*** schipiga_ has quit IRC11:23
*** schipiga_ has joined #openstack-infra11:23
*** asettle has joined #openstack-infra11:24
AJaegeryolanda: thanks, approved11:24
sdagueAJaeger: any ideas on this content publish issue11:25
*** esker has quit IRC11:26
*** esker has joined #openstack-infra11:27
*** kzaitsev_mb has quit IRC11:27
schipigayolanda: I suppose, may I ask you to set workflow to ? :)11:27
yolandaschipiga, I can review, yes11:28
schipigaplease, would be very grateful11:29
AJaegersdague: let me read backscroll...11:29
schipigathank you11:29
AJaegersdague: post job:11:29
sdagueAJaeger: the issue is is definitely not up to date with the content11:29
AJaegerLet me check 30716411:30
sdagueoh, wait a second11:30
sdaguemaybe I'm wrong,11:30
*** e0ne has joined #openstack-infra11:30
sdagueI guess it just took forever to flush all the post jobs yesterday11:30
*** zhurong has joined #openstack-infra11:30
AJaegersdague: yes, that took ages11:30
*** yamamoto_ has quit IRC11:30
AJaegersdague: ?11:31
*** Qiming has joined #openstack-infra11:32
AJaegersdague: has as footer:11:32
*** Goneri has joined #openstack-infra11:32
AJaeger" Last updated on 'Wed Apr 22 09:17:25 2015, commit 73d5d1f'." - *2015*11:32
*** kzaitsev_mb has joined #openstack-infra11:32
*** yamamoto_ has joined #openstack-infra11:33
*** salv-orl_ has quit IRC11:33
AJaegersdague: so, publishing looks fine in the post job IMHO11:33
sdagueok, great, thanks11:34
*** rhallisey has joined #openstack-infra11:34
openstackgerritMerged openstack-infra/project-config: Skip networking-nec python34 job for liberty and kilo
*** Goneri has quit IRC11:38
*** MarkAtwood has joined #openstack-infra11:38
*** resker has joined #openstack-infra11:38
*** gordc has joined #openstack-infra11:38
openstackgerritMerged openstack-infra/project-config: Publish test_reports dir
*** nadya has quit IRC11:39
openstackgerritDavid Moreau Simard proposed openstack-infra/project-config: Replace the packstack-dev channel by rdo
dmsimardyolanda: ^ they were sorted, in my heart :P11:40
yolandadmsimard, cool. It was just a nit11:41
*** esker has quit IRC11:41
dmsimardyolanda: no worries, my brain went top (most recent) to bottom (oldest) so that's why I said they were sorted in my heart :p11:42
openstackgerritMerged openstack-infra/release-tools: add
*** kzaitsev_mb has quit IRC11:43
openstackgerritMerged openstack-infra/release-tools: make quieter
*** schipiga_ has quit IRC11:46
*** yamamoto_ has quit IRC11:46
*** asettle has quit IRC11:47
*** MarkAtwood has quit IRC11:48
*** kengo_sa_ has quit IRC11:48
*** thorst has joined #openstack-infra11:49
*** lucas-hungry is now known as lucasagomes11:49
*** yamamoto_ has joined #openstack-infra11:50
*** nadya has joined #openstack-infra11:54
*** resker has quit IRC11:55
*** icey_ is now known as icey11:55
*** amotoki has joined #openstack-infra11:58
*** yamamoto_ has quit IRC11:58
*** kushal has joined #openstack-infra11:59
*** Qiming has quit IRC12:00
*** mylu has joined #openstack-infra12:00
*** dtantsur is now known as dtantsur|brb12:05
*** mylu has quit IRC12:06
openstackgerritAndrey Pavlov proposed openstack-infra/project-config: ACL: allow to create branches for gce-api project
*** sdague has quit IRC12:06
*** esker has joined #openstack-infra12:07
*** amotoki_ has joined #openstack-infra12:09
*** zul has quit IRC12:11
*** amotoki has quit IRC12:11
*** chuck__ has joined #openstack-infra12:11
*** amotoki_ has quit IRC12:11
*** chuck__ is now known as zul12:12
EmilienMfungi, pabelanger, AJaeger: just an update about our Puppet OpenStack periodic jobs -> they work fine now. Thanks again for your help this week12:13
*** EricGonczer_ has joined #openstack-infra12:15
krotscheck_dcmcrinkle: OMG yes I need that. In fact, I think I have one!12:15
*** krotscheck_dcm is now known as krotscheck12:15
*** jlanoux has quit IRC12:17
*** markvoelker has joined #openstack-infra12:17
*** Yarboa has quit IRC12:18
AJaegerEmilienM: woot!12:18
EmilienMAJaeger: the tarballs are fine also
EmilienMAJaeger: though I haven't tested the tarballs for stable branches12:19
openstackgerritMichael Krotscheck proposed openstack-infra/system-config: Added clean exit hooks to npm registry script
yolandaoh, beaker finally passing! i launched the image rebuilds and updates this morning12:19
EmilienMyolanda: images are uploaded to all clouds?12:20
yolandayes, they finished 30 minutes ago12:20
EmilienMexcellent news12:20
yolandaand i saw my failing test passing now12:21
*** amotoki has joined #openstack-infra12:21
EmilienMyolanda: FYI we had a workaround
EmilienMmaybe infra could also have a openstack-infra/puppet-infra_spec_helper12:21
EmilienMit's a lot of work to put in place, but afterward everything is centralized12:22
AJaegergreat, yolanda ! I'll recheck now - reviews are welcome (puppet-reviewday logging)12:23
*** salv-orlando has joined #openstack-infra12:28
*** xwizard___ has quit IRC12:28
yolandai'll do12:28
*** asettle has joined #openstack-infra12:29
*** esikachev has quit IRC12:30
openstackgerritMerged openstack-infra/tripleo-ci: Update deprecated commands in (Old heat * calls)
*** obondarev has quit IRC12:31
*** jordanP has quit IRC12:32
*** sdague has joined #openstack-infra12:32
*** Yarboa has joined #openstack-infra12:32
*** obondarev has joined #openstack-infra12:33
*** dtantsur|brb is now known as dtantsur12:33
*** EricGonczer_ has quit IRC12:34
*** hughhalf has quit IRC12:34
*** burgerk has joined #openstack-infra12:35
*** annegentle has joined #openstack-infra12:36
AJaegerthanks, yolanda12:37
*** rfolco has joined #openstack-infra12:37
rcarrillocruzany news from OVH folks?12:37
*** Hal has joined #openstack-infra12:37
yolandai just got updates from Jean-Daniel, i'm restarting the suspended mirrors now12:37
yolandathings are going back to life12:38
*** Hal is now known as Guest8383612:38
amrithI asked this earlier on openstack-dev, maybe someone here knows the answer .. does anyone know of a simple way to export your summit schedule into ICS or calendaring software or are you stuck having to use the app?12:38
AJaegeryolanda: plesae abandon your changes to take these down...12:38
AJaegeryolanda: Great, thanks!12:38
*** sdague has quit IRC12:38
AJaegeramrith: there were some emails on openstack-dev mailing list for this ...12:39
*** sdague has joined #openstack-infra12:39
*** sdague has quit IRC12:39
*** yamamoto_ has joined #openstack-infra12:39
*** gongysh has quit IRC12:39
*** e0ne has quit IRC12:39
amrithAJaeger, hmmm. didn't see any, let me go look. was there a solution or discussion? I recall something about futures for cheddar, nothing about the final schedule.12:40
*** Yarboa has quit IRC12:40
*** asettle has quit IRC12:40
*** hughhalf has joined #openstack-infra12:40
yolandamirrors are alive again, going to restore manually in nodepool12:40
*** gongysh has joined #openstack-infra12:42
*** e0ne has joined #openstack-infra12:42
AJaegeramrith: subject "summit tools" - the mail from dstanek12:42
* amrith looks for the mail12:43
amrithone second12:43
annegentleamrith: Oh I was looking for that myself. Post the link when / if you find it12:43
annegentle(though I already made my calendar ha)12:43
amriththx, got it.
*** jlanoux has joined #openstack-infra12:43
*** jaypipes has joined #openstack-infra12:44
annegentleamrith hehe well if Tony (who made all the ical files automation for meetings) doesn't need it then...12:44
annegentleamrith ah there we go!12:44
annegentleamrith thanks!12:45
amrithannegentle, just QA'ing the tool. Will give it a +1 if it works :)12:45
*** zeih has quit IRC12:45
* mordred hands dstanek a giant box of kittens12:46
mordredannegentle, amrith: you may also be interested in
annegentleI gave dstanek a star, is that enough? Seems less than kittens.12:47
yolanda#status notice OVH provider is enabled again, please wait for the job queue to be processed12:47
openstackstatusyolanda: sending notice12:47
*** sdague has joined #openstack-infra12:47
*** sdague has quit IRC12:47
*** sdague has joined #openstack-infra12:48
mordredpabelanger: any chance you know how we're doing on image uploads?12:48
-openstackstatus- NOTICE: OVH provider is enabled again, please wait for the job queue to be processed12:48
AJaegermordred: yolanda regenerated images this European morning, all updated aFAIU12:48
mordredyolanda: so I should be safe to restart nodepool again?12:49
yolandamordred, no image builds at the moment from my side12:49
*** weshay has joined #openstack-infra12:49
openstackstatusyolanda: finished sending notice12:49
*** rlandy has joined #openstack-infra12:50
*** rlandy has quit IRC12:50
*** rlandy has joined #openstack-infra12:52
*** amotoki has quit IRC12:52
annegentleamrith works for me, sweet! Thanks dstanek12:52
*** Yarboa has joined #openstack-infra12:53
*** zeih has joined #openstack-infra12:53
dstanekannegentle: i sent you an email with the link :-(12:53
*** asettle has joined #openstack-infra12:53
*** degorenko is now known as _degorenko|afk12:53
annegentledstanek aww I missed that! looking12:53
*** kushal has quit IRC12:54
*** keedya has joined #openstack-infra12:54
*** gongysh has quit IRC12:56
amrithannegentle, didn't work for me. something about _strptime() ValueError: 'z' is a bad directive in format '%Y-%m-%d %I:%M%p %z'. Which reminds me, I have to send an email to the ML about the dreaded isotime() thing :(12:56
annegentleamrith anything special about the sessions you chose? :)12:57
amrithannegentle, not that I can tell12:57
*** claudiub has joined #openstack-infra12:58
openstackgerritMerged openstack-infra/storyboard-webclient: Improve 'Tasks Assigned to Me'
*** esker has quit IRC12:58
*** esker has joined #openstack-infra12:58
amrithannegentle, how did you save your schedule as mine.html? I tried to do it with wget but couldn't figure out how to give it my username/password. so I opened it in firefox and saved as html-only.12:59
*** links has quit IRC12:59
amrithwhich, as it turns out stuck a bunch of javascript and stuff in there12:59
annegentleI was in Chrome and did html-only (though my first mistake was saving it as "web page complete" or some such12:59
annegentlewonder if the html is different from chrome12:59
*** e0ne has quit IRC13:01
dstanekamrith: the site is pretty terrible from a data point of view. i have to parse javascript to build the calendar.13:01
dstanekamrith: did you same the print view of your calendar?13:02
annegentledstanek dannng I'm glad I didn't try to tackle this then13:02
amrithdstanek, hi ... I navigated to in firefox13:02
amrithdid a 'save-as'13:02
amrithweb page html only13:02
amrithit has a crap load of javascript in there13:02
amrithbut all the start and end times13:02
amrithhave no 'z'13:02
dstanekamrith: there shouldn't be a z in the times13:03
*** kgiusti has joined #openstack-infra13:03
dstanekthey should be something like 12:25pm13:03
*** maishsk has quit IRC13:03
amrithlet me just strip the javascript out and try13:03
dstanekamrith: that's the only part i need :-)13:03
dstaneki'm actually parsing the javascript13:04
amrithok, let me not do that then :)13:04
*** matt-borland has joined #openstack-infra13:04
amrithstart and end times are like13:05
amrith                   start_time: "12:05pm",13:05
amrith                    end_time: "12:45pm",13:05
*** maishsk has joined #openstack-infra13:05
dstanekamrith: can you paste your file somewhere so i can take a look?13:05
amrithsure, one second13:05
*** nwkarsten has joined #openstack-infra13:06
*** dizquierdo has quit IRC13:06
*** esker has quit IRC13:07
KiallHey infra folks.. I've noticed some projects putting optional / feature requirements into setup.cfg.. My Q is, is that supported by things like the requirements tooling etc?13:07
*** mylu has joined #openstack-infra13:07
*** esker has joined #openstack-infra13:07
*** esker has quit IRC13:07
openstackgerrityolanda.robla proposed openstack-infra/project-config: Add job validation and latest link to magnum image upload
*** jrmyjohn has joined #openstack-infra13:08
amrithdstanek, sent you the link in PM13:09
*** esikachev has joined #openstack-infra13:09
*** jrmyjohn has quit IRC13:09
*** jrmyjohn has joined #openstack-infra13:09
*** mtanino has joined #openstack-infra13:09
amrithso, get_dt wants to parse 2016-04-25 9:00am -0500 with the format %Y-%m-%d %I:%M%p %z13:10
amrithis this a python3/python2.7 thing?13:11
dstanekamrith: oh, it could be. i'm using py313:12
amrithtrying that13:12
amrithone second13:12
*** andymaier_ has joined #openstack-infra13:12
*** doug-fish has quit IRC13:13
*** asettle has quit IRC13:13
*** thiagop has joined #openstack-infra13:13
*** doug-fish has joined #openstack-infra13:13
*** asettle has joined #openstack-infra13:13
*** amotoki has joined #openstack-infra13:16
*** hughhalf has quit IRC13:16
*** doug-fish has quit IRC13:17
*** annegentle has quit IRC13:20
*** mtanino has quit IRC13:20
*** nmagnezi has quit IRC13:21
*** Goneri has joined #openstack-infra13:23
*** mriedem has joined #openstack-infra13:23
*** hughhalf has joined #openstack-infra13:23
*** _ari_ has joined #openstack-infra13:25
*** doug-fish has joined #openstack-infra13:27
* dims_ peeks to see if we are past the ovh network problems13:28
*** ddieterly has joined #openstack-infra13:28
amriththanks much dstanek ... I knew I wouldn't have had these problems with Verizon :)13:28
AJaegerdims_: yes, we are - still some backlog to go through...13:28
dims_thanks AJaeger !13:28
amrithit was only that one session13:29
*** trown|outtypewww is now known as trown13:29
*** doug-fis_ has joined #openstack-infra13:29
dstanekamrith: :-)13:30
*** sdake_ has joined #openstack-infra13:30
*** mylu has quit IRC13:30
*** mpaolino has quit IRC13:31
EmilienMinfra-root: can someone remove files in ? we did some cleanup in scripts that generate the tarballs, and those ones have wrong name. Thanks13:31
*** vhosakot has joined #openstack-infra13:32
*** sdake has quit IRC13:32
*** aimeeU has joined #openstack-infra13:33
*** doug-fish has quit IRC13:33
*** AJaeger has quit IRC13:33
*** AJaeger has joined #openstack-infra13:36
*** jheroux has joined #openstack-infra13:36
*** nwkarsten has quit IRC13:37
*** nwkarsten has joined #openstack-infra13:38
*** HeOS_ has quit IRC13:38
*** xyang1 has joined #openstack-infra13:39
*** nwkarsten has quit IRC13:42
*** AJaeger has quit IRC13:42
*** AJaeger has joined #openstack-infra13:43
*** mpaolino has joined #openstack-infra13:43
fungiyolanda: so what was the deal with ovh? did they take our openstackci tenant offline again already?13:44
yolandaagain yes13:44
fungii skimmed scrollback but didn't spot much detail about it13:44
fungithat's unfortunate13:44
yolandaour account was still showing a billing due with them, so the account was blocked13:45
yolandait is fixed now13:45
openstackgerritKen Dreyer proposed openstack/python-jenkins: add get_whoami()
*** inc0 has joined #openstack-infra13:45
fungilooks like it only lasted from 08:40 to 12:40 utc this time, so not as painful as before13:46
AJaegeryolanda: have they fixed things so that it won't break again next time?13:46
fungier, 9:40 to 12:4013:46
AJaegerfungi: this time yolanda knew whom to reach I think ;)13:46
fungiso 3 hours13:46
yolandafungi, as soon as I saw it, i contacted Jean-Daniel and he fixed13:46
fungithanks yolanda!!!13:46
yolandahe confirmed the account problems are solved now13:46
AJaegerthanks, yolanda !13:46
*** nwkarsten has joined #openstack-infra13:47
*** nwkarsten has quit IRC13:47
*** nwkarsten has joined #openstack-infra13:47
openstackgerrityolanda.robla proposed openstack-infra/project-config: Enable magnum plugin on shade and use our atomic generated images
*** EricGonczer_ has joined #openstack-infra13:49
openstackgerrityolanda.robla proposed openstack-infra/project-config: Enable magnum plugin on shade and use our atomic generated images
openstackgerritMatthew Treinish proposed openstack-infra/puppet-openstack_health: Add packages needed to build lxml
*** yamamoto_ has quit IRC13:54
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/shade: Add Designate recordsets support
openstackgerritMerged openstack-infra/project-config: Add new virtualbmc project under ironic's umbrella
openstackgerrityolanda.robla proposed openstack-infra/project-config: Enable magnum plugin on shade and use our atomic generated images
*** mylu has joined #openstack-infra13:57
*** EricGonczer_ has quit IRC13:57
openstackgerritMerged openstack-infra/project-config: Add ServerGet task to nodepool graphs
openstackgerritMerged openstack-infra/project-config: Adding a new project called "neutron-dynamic-routing"
*** EricGonczer_ has joined #openstack-infra13:59
*** sdake_ has quit IRC14:00
*** sdake has joined #openstack-infra14:02
*** BigWillie has joined #openstack-infra14:02
*** yamamoto has joined #openstack-infra14:04
*** zhurong has quit IRC14:05
prometheanfirecan someone re-workflow this dib job?
*** _degorenko|afk is now known as degorenko14:06
fungiprometheanfire: how will that help?14:07
*** burgerk has quit IRC14:07
fungiit depends on a change that depends on a change that isn't approved (and is marked wip)14:07
*** zz_dimtruck is now known as dimtruck14:08
*** mtanino has joined #openstack-infra14:08
prometheanfirewhwhich change?14:08
fungiprometheanfire: 300256 depends on 296626, which depends on 299694 which is wip14:08
AJaegerprometheanfire: in the web ui, see "Related changes" in the upper right corner14:09
fungithat's why the other two aren't enqueying14:09
fungier, enqueuing14:09
prometheanfireAJaeger: ya, it's never obvious whether related is related or dependant14:09
*** lezbar__ has quit IRC14:10
prometheanfireI don't know why those are dependant on mine :|14:10
fungii agree, the order in that list in the new change screen is confusing. gertty displays it more like the old change screen does where you get a "depends on" list and a "needed by" list14:10
*** lezbar has joined #openstack-infra14:11
prometheanfireI thought mine was waiting on those14:11
fungiprometheanfire: they were pushed to gerrit as a dependent change series (the git parent of a change, if also an open change, is seen by gerrit as a dependency)14:11
fungiif you don't want 296626 to depend on 299694 or don't want 300256 to depend on 296626 then you should rebase them directly onto the tip of master and push new patchsets14:12
*** e0ne has joined #openstack-infra14:12
prometheanfirefungi: 300256 and 296626 seem to do a bunch of work that I was also doing in 29969414:13
*** gongysh has joined #openstack-infra14:13
prometheanfirefungi: at this point it'd probably be better to just have them vote on 29969414:14
fungiprometheanfire: yeah, coordinate with greghaynes and ianw on that14:14
prometheanfiregreghaynes: ianw: both of your DIB changes depend on mine as the root, mind voting? :P
fungii'm not a regular reviewer on dib (much less a core reviewer for it) so lack insight into what their intent was there14:14
prometheanfirefungi: done, you had a chance to look at the bindep changes?14:15
prometheanfirefungi: fair enough14:15
mtreinishfungi, yolanda: is there a reason tests didn't run against:
prometheanfirefungi: I honestly think dib needs at least one more core, atm it only seems to be them reviewing14:15
*** daemontool has joined #openstack-infra14:15
mtreinishI would have expected at least a docs job to get triggered14:15
AJaegerwhy do we set up repositories if nobody ever submits code to them?,members is still empty - the repo was created a month ago ;(14:15
fungiprometheanfire: no, not yet. still caffeinating for the morning and trying to get through overnight e-mail14:16
prometheanfirefungi: sure, I'm still catching up too :D14:16
*** hashar has joined #openstack-infra14:16
*** lezbar__ has joined #openstack-infra14:17
prometheanfirefungi: so, which view do you use to see the dependant chanes more clearly?14:17
bdemersAre there any restrictions (other then no TC voting) for unofficial (i.e. recent neutron stadium) projects ?14:17
*** lezbar has quit IRC14:18
AJaegerbdemers: translators will only look at official projects, is only for official projects14:18
openstackgerritIan Cordasco proposed openstack-dev/hacking: Add reno and release notes tracking
yolandamtreinish, i triggered a recheck and worked now14:20
fungiprometheanfire: i actually use and eschew the gerrit webui for anything besides group management et cetera14:20
mtreinishyolanda: cool, thanks14:20
fungibut i'm not a big gui fan to start with14:21
*** claudiub has quit IRC14:21
mtreinishyolanda: although looking at it now it looks like there weren't any results on the previous rev either14:21
mtreinishI wonder what caused that14:21
fungianything i can do in a terminal i can usually do faster and more effectively than stuff which requires me to touch a mouse14:21
jeblairfungi: we are running *without* the connections revert since yolanda restarted zuul14:21
mordredmice suck14:21
fungijeblair: okay, that makes sense on the leak behavior then14:21
fungiso it probably still is the connections work which is leaking14:22
openstackgerritJames E. Blair proposed openstack-infra/project-config: Revert "Fix comment on requirements job"
*** rbrndt has joined #openstack-infra14:22
jeblairianw: ^ i'm sorry, i thought i explained that the problem was in the javascript+regex, *not* in the message itself14:23
jeblairyolanda, AJaeger: ^ please see this revert -- i do not believe the original patch should have been approved14:23
AJaegerjeblair: reading...14:23
jeblairfungi, mordred: would you please instapprove 309018?14:23
fungidone and done14:24
fungiinteresting that the removal also included removing the error message from the job script14:24
jeblairfungi: it's a straight revert14:24
*** Goneri has quit IRC14:24
jeblairit was added in the "fix"14:24
fungier, right, i was looking at the revert backwards on that file14:25
AJaegerfungi: it's indeed a straight revert. The error message could be added in IMHO.14:25
yolandajeblair, checking14:25
*** maishsk has quit IRC14:25
bdemersAJaeger docs.o.o completely or just the translation bit? if if completely what happens/should happen to current pages on docs.o.o ?14:26
fungiyeah, i'd gladly approve a change which just added the error message to the job script, but also i agree the hideci-js needs proper fixing for that14:26
AJaegerbdemers: completely14:26
bdemerssay for example:
*** dizquierdo has joined #openstack-infra14:27
bdemerswill a redirect be setup? or i'm unclear how dead links or out dated doc would be handled14:27
jamespageyolanda, hey - I think this might be side effect of the branch dance we did this morning:14:28
AJaegerbdemers: we could add a redirect14:28
jamespageyolanda, what do you think?14:28
bdemersAJaeger ok, is there an unofficial docs / wiki space we should be using  ?14:28
AJaegerbdemers: rtfm is used for unofficial docs - but is fine14:29
*** roxanagh_ has joined #openstack-infra14:29
*** asettle has quit IRC14:29
yolandajamespage, nice... looking, i tried a rebase from ui and didn't work14:29
bdemersAJaeger ok, cool, that works14:29
AJaegerbdemers: for a redirect, send patch to openstack-manuals/www/static/.htaccess14:30
bdemersok, thanks!14:30
*** sputnik13 has quit IRC14:30
fungiianw: probably somewhere in ci_parse_comments() or a new function called from it
*** ayoung has joined #openstack-infra14:31
jamespageyolanda, yeah - ditto in the ui - if I rebase locally, there is no change to rebase on...14:32
jamespagewhich is kinda what I expected...14:32
yolandayep, same here, i cannot send any update because rebase is doing nothing14:32
fungiianw: or like jeblair suggested, the regular expressions at the top may be sufficiently tweakable to match on it14:32
AJaegerfungi: ianw sent - I think that should fix the issue as well14:32
AJaegerfungi: but I didn't dig further14:33
*** roxanagh_ has quit IRC14:33
*** vikram has joined #openstack-infra14:34
yolandajamespage, i tried to send just a small change on the commit message, and it failed14:34
fungiahh, yep, that looks sane though would want to locally test out that patch with a javascript console14:34
yolandalet me dig deeper14:34
vikramfungi: Can you please help modifying the ACL for the newly created project @
jamespageyolanda, I would suspect that whatever git repo is being used to check merges, it needs a prune...14:35
yolandai was checking now zuul git repos, yes14:35
jamespageyolanda, thanks for your help today - much appreciated...14:35
yolandajamespage, anytime14:35
fungijamespage: yolanda: the jgit library (java-based git implementation) used by gerrit can be sort of obtuse when it comes to merge conflict resolution. cgit has a lot more effective resolution methods it tries so can make it seem like there is no conflict when there actually is for the methods jgit is limited to. no idea if that's the situation here, just something to keep in mind when troubleshooting14:37
yolandafungi, the problem is that i removed a bunch of stable branches today from that charms repos. I think zuul-mergers may be confused14:38
yolandai'm checking one of the zuul mergers repo now14:38
yolandaso that's it... stable branch already exists on the zuul-mergers repo14:39
yolandafungi, what shall be the best way to clean up those?14:39
openstackgerritCarlos Camacho proposed openstack-infra/tripleo-ci: Finish: Update deprecated commands in (Old heat stack-* calls)
fungiyolanda: oh, yes replacing branches with non-ff-able content will do that14:39
yolandaon downstream, i normally just removed the repo14:39
fungiyolanda: that's one of the reasons we heavily discourage it14:39
fungiit's also pretty user-hostile since anyone who cloned that branch will now need to manually delete or reset it14:40
yolandayes, problem is that those stable branches were blocking the creation of 16.04 ones14:40
jeblairi'm not sure we've ever done this.14:41
*** Goneri has joined #openstack-infra14:41
fungiwait, what was the name of the branch you deleted?14:41
*** oanson has quit IRC14:42
fungii see. for example zm01 thinks there is a remotes/origin/stable and it has a local tracking branch called stable14:43
*** Nikolay_St has quit IRC14:43
fungiin /var/lib/zuul/git/openstack/charm-swift-proxy14:43
yolandai've done that branches deletion in downstream several times, and hitting similar problems.. what we did normally is to just remove the repos from the mergers, so zm recreated again14:43
openstackgerritChris Denneen proposed openstack-infra/jenkins-job-builder: Add BuildNameUpdater to Builders
fungias far as i know that's safe, but it's a lot of added manual effort14:44
*** roxanagh_ has joined #openstack-infra14:44
yolandai know, but i'm happy to take the responsability as I did the branch deletion14:44
yolandai cannot thing on a better way14:44
jeblairthe branches should not have been imported in the first place14:45
yolandain zuul mergers?14:45
fungiin new project creation14:45
fungii suspect nobody expected having a branch named "stable" would prevent later creation of branches named like "stable/foo" and didn't question it14:46
*** Goneri has quit IRC14:46
AJaegerfungi: in I asked whether *all* branches should be imported - but didn't notice the relevance of "stable"14:46
prometheanfirefungi: I found that out the hard way by having a branch named bug and not be able to make one named bug/foo/bar :(14:46
yolandai'm just learning about that naming problem as i see14:47
jamespagejeblair, apologies - that was my action - I was unaware that having a stable branch would block having a stable/XXX branch later...14:47
AJaegershould we blacklist such a branch? we can add a check for the future...14:47
fungi"stable" might be a special case we'd look out for since projects commonly use it as a branch namespace14:47
fungii'm on the fence about introducing ever-growing blacklists which might occasionally prevent legitimate use though14:48
jeblairstable, feature14:48
jamespageagreed - would have used a namespaced stable/XXX if I'd have known...14:48
mordredjeblair: proposed is another one, right? (is that where we're putting pre-release stuff now?)14:48
yolandaso i can purge that repos from the zuul mergers as a workaround14:48
jeblairfungi: blacklisting only the ones that have meaning in infra might be reasonable though?14:49
jeblairmordred: ya; so stable, feature, proposed14:49
jeblairhopefully no one uses 'feature' as a branch name -- it's a pretty terrible one :)14:50
jeblair"oh, yeah, that's where we added that feature"14:50
fungiyeah, if we can keep it limited that seems fine14:50
jeblair"which one?"  "the one that wasn't a bug"14:50
*** sdague_ has joined #openstack-infra14:50
fungiwe fixed the bug in that other branch. the one named "bug"14:51
openstackgerritMerged openstack-infra/puppet-reviewday: Create logs for update_review
AJaegermordred: proposed is not used anymore14:51
fungiright, we dropped proposed/ and switched to just creating stable/ during rc instead14:52
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Block "stable" branches for new project imports
AJaegerteam, here's a suggested change just blacklisting "stable" - I can abandon or enhance to check for "feature" as well. ^14:52
openstackgerritJames E. Blair proposed openstack-infra/project-config: Don't check for proposed branches in propose_update
jeblairfungi, AJaeger: in that case ... ^ :)14:53
*** sdague_ has quit IRC14:53
*** sdague has quit IRC14:53
nibalizerEmilienM: yea that change looks fine14:54
*** mrmartin has joined #openstack-infra14:54
fungiAJaeger: let's add feature too, yeah14:54
*** sdague has joined #openstack-infra14:54
AJaegerwill do, fungi14:55
dansmithpabelanger: clarkb: fungi: by the way, there are not hammer xenial packages yet, but the next build will generate them, which will be in about a week, reportedly, so we should be good I think14:55
pabelangerdansmith: great!14:55
dansmithpabelanger: it also sounds like jewel will be a thing we'll want to migrate to (from hammer) at some point just to stay current14:56
pabelangermordred: sorry, out for a walk. Looks like yolanda helped!14:56
*** jaosorior has quit IRC14:56
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Block stable/feature branch for new project imports
AJaegerfungi, jeblair ^14:57
openstackgerritJames E. Blair proposed openstack-infra/system-config: Remove proposed/ acls
pabelangerdansmith: Ya, clarkb and I were talking about that yesterday. It shouldn't be hard to get the next mirror up now14:57
jeblairAJaeger, fungi: ^ and some more proposed proposed cleanup14:57
dansmithpabelanger: cool14:57
yolandajamespage, i cleaned repos. Seems better now14:57
*** tlian has joined #openstack-infra14:57
jamespageyolanda, thankyou!14:57
AJaegerjeblair: does the infra manual still mention "proposed/"? That's one more place that might include it.14:58
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Revert "Test with latest master heat"
AJaegerjeblair: thanks for the cleanup!14:58
jeblairAJaeger: yep14:58
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Block stable/feature branch for new project imports
AJaegerjeblair: fixed the double of - thanks for review14:59
mordredjeblair: I have a hunch on another thing to do to make less fip leaks15:00
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: TEST: Delete the overcloud when finished
* AJaeger needs a spell checker. Thanks fungi15:00
fungiAJaeger: one of those typos was a real word, so a spell checker would only have half helped15:00
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Block stable/feature branch for new project imports
*** kushal has joined #openstack-infra15:01
jeblairAJaeger: the infra-manual is up to date15:01
AJaegerfungi: I know.15:01
*** Goneri has joined #openstack-infra15:01
AJaegerjeblair: cool!15:01
mordredShrews, jeblair: when I do a neutron floatingip-show on a leaked FIP - it shows "state: DOWN"15:02
AJaegerwow, we're very fast right now in proposing and reviewing changes ;)15:03
mordredI'm tinking that perhaps in the create_floating_ip routine - we check for getting a FIP with state: DOWN. if we do, delete it and try again15:03
* AJaeger has to leave the party now, will be back later...15:03
pabelangerAJaeger: o/15:04
*** psilvad_ has joined #openstack-infra15:04
*** psilvad_ has quit IRC15:04
Shrewsmordred: does it end up in the DOWN state on creation or deletion?15:04
mordredShrews, jeblair: I need to do some testing to see if in a normal case where it's working the fip returns as down then flips to up - or if it returns as up when it works15:04
mordredShrews: that's the question. but all of the leaked ones are definitely in down state15:05
Shrewsmordred: creating a new FIP that starts in the DOWN state makes no sense to me. does that mean it's in an error state?15:05
* Shrews offers no help15:05
mordredit's the next thing I'm going to explore15:05
Shrewswe should poke neutron'ers15:06
jeblair++to the source15:06
mordredjeblair: got a sec to help me look at logs?15:07
mordredjeblair: node15:07
mordredjeblair: node 44204715:07
jeblair(it may be really hard to thoroughly evaluate this in practice since, if being in 'down' is every a legit part of a successful operation, it may be very hard to catch)15:07
jeblairmordred: ya15:07
mordredis a node that had a leaked fip15:07
*** roxanagh_ has quit IRC15:07
mordredjeblair: but I can't see where that node either succeeded or failed15:07
mordredjeblair: also, d9657654-cf92-4442-9247-767ad97a32f2 is a newish leaked fip15:08
jeblairmordred: it's still building15:08
mordredjeblair: ah. SO - that means that we're waiting for it to show up on the server15:08
jeblairmordred: yeah15:08
jeblairmordred: 2016-04-21 14:25:21,481 DEBUG shade: Created FIP for server e8e54f61-44e0-4125-801e-ba2768b5d65415:08
mordredbut it's showing down and no port id in neutron15:08
jeblairmordred: that's the last thing for it i think15:09
openstackgerritMerged openstack-infra/tripleo-ci: Use the cached ironic-python-agent images
vikraminfra_core: Can someone please help modifying the ACL for the newly created project via
mordredjeblair: maybe step one is "check to see that the returned fip has port_id == requested_port_id15:09
jeblairmordred: yeah -- another good question for neutron devs -- is that a transient state?15:10
*** burgerk has joined #openstack-infra15:10
jeblairmordred: (ie, if you request a fip on a port, is it ever valid for the port to be null?)15:10
jeblair(on the fip)15:10
*** andymaier_ has quit IRC15:10
pabelangeryolanda: when we disabled OVH, did we also clean up nodepool?  I see a lot of ready nodes in OVH15:11
*** Goneri has quit IRC15:11
pabelangeryolanda: 2.5+ hours15:11
*** tpsilva has joined #openstack-infra15:11
pabelangeryolanda: think we might need to delete them from nodepool15:12
openstackgerritMonty Taylor proposed openstack-infra/shade: Fail if FIP doens't have the requested port_id
mordredjeblair, Shrews: ^^ something like this15:12
mordredmestery: any chance you know the answer here?15:13
*** mylu has quit IRC15:14
*** schipiga_ has joined #openstack-infra15:15
*** e0ne has quit IRC15:15
*** salv-orlando has quit IRC15:16
*** jaosorior has joined #openstack-infra15:16
openstackgerritChris Denneen proposed openstack-infra/jenkins-job-builder: Add BuildNameUpdater to Builders
*** e0ne has joined #openstack-infra15:17
fungion a cleanup note, should we keep the old gerrit server around any longer? should i snapshot its rootfs and delete the instance?15:18
pabelangerinfra-root: I just cleaned up the stale ready nodes from nodepool, FYI.15:19
fungithanks pabelanger15:19
mordredI have asked about the port id thing in the neutron channel15:21
*** inc0 has quit IRC15:22
pabelangerAny objections on adding an --age argument to filter on nodepool list ?15:22
openstackgerritMerged openstack-infra/project-config: Replace the packstack-dev channel by rdo
*** BigWillie has quit IRC15:23
clarkbgood morning15:23
clarkblooks like fip leaks are being debugged and ceph will get both xenial and new release in near future.15:24
pabelangerclarkb: morning15:24
mordredclarkb: yup. we have new theories on the FIP leaks :15:24
jeblairclarkb: what nodepool docs change were you talking about with pleia2?15:26
*** Goneri has joined #openstack-infra15:26
*** andreww has quit IRC15:27
*** scheuran has quit IRC15:27
clarkbnibalizer: looks like ES is still unhappy. I have no idea why this is happening15:27
clarkbnibalizer: as an improvement we seem to be indexing stuff unlike before but the cluster is refusing to finish up shard recovery15:27
yolandapabelanger, no , i just set the max-servers to 0, then re-enabled again15:27
nibalizerclarkb: bah!15:27
*** BigWillie has joined #openstack-infra15:28
pabelangeryolanda: Thanks15:28
clarkbnibalizer: it seems that the issue is when recoverying a shard from the master node everything times out15:28
clarkbnibalizer: but there is network connectivity between them15:29
*** e0ne has quit IRC15:29
*** Swami has joined #openstack-infra15:30
*** mhickey has joined #openstack-infra15:30
*** zeih has quit IRC15:30
*** Na3iL has joined #openstack-infra15:30
*** mylu has joined #openstack-infra15:31
yolandamordred, jeblair, can you take a look to the glean resolv.conf fix when you have a sec?
*** sigmavirus24_awa is now known as sigmavirus2415:33
clarkbnibalizer: those seem to be the relevant logs15:33
*** esikachev has quit IRC15:34
* nibalizer looks at bigdesk15:34
pabelangerpleia2: docaedo: Did we maybe want to get onto the asterisk server this afternoon / tomorrow and have a quick chat about presentation. See if we need to make any other changes to overview and / or review the slides?  Doing it on site at austin is another option15:35
*** e0ne has joined #openstack-infra15:36
clarkbnibalizer: I am going to delete the .kibana index because queries against it are creating noise in the logs (and since we aren't kibana4ing yet it isn't necessary)15:36
nibalizerit says out of memory in the logs but bigdesk thinks 05 is doing fine on memory15:37
nibalizeri guess its now 30 minutes from those log events15:37
*** jordanP has joined #openstack-infra15:37
*** yamahata has joined #openstack-infra15:38
clarkbnibalizer: yup, but if you read further it says "actual exception" then "org.elasticsearch.indices.recovery.DelayRecoveryException: source node does not have the shard listed in its state as allocated on the node" is the error there15:38
*** fedexo has joined #openstack-infra15:38
clarkbI wonder if the heap space is just noise in some error handling stack. And ^ is the real issue15:39
nibalizerit labeled the actual exception15:39
mordredclarkb, Shrews, jeblair: ok. I have tested (once) a valid server creation on osic and it returned the port_id in the return dict from the floating ip create15:41
*** armax has joined #openstack-infra15:42
*** hdd has joined #openstack-infra15:43
*** Nikolay_St has joined #openstack-infra15:43
* jeblair installs an ftp server15:43
mordredaha. SO - it comes with state DOWN and the requested port_id15:44
jeblairmordred: but if it doesn't have a port, it's definitely an error?15:45
mordredjeblair: well, that I don't know - but the successes have ports15:46
*** vincentll has quit IRC15:46
mordredjeblair: so I'm willing to take a plunge and treat no-port as error for now15:46
*** hdd has quit IRC15:46
mordredand also perhaps put in a wait for the fip to go from DOWN to ACTIVE15:46
clarkbnibalizer: I am still reading that but I think it may be our problem15:46
mordredbefore we return success15:46
*** roxanagh_ has joined #openstack-infra15:47
*** openstackgerrit has quit IRC15:48
*** cloudtrainme has joined #openstack-infra15:48
*** julim has joined #openstack-infra15:48
*** jaosorior has quit IRC15:48
*** openstackgerrit has joined #openstack-infra15:48
docaedopabelanger: I could hop on a call this afternoon to discuss, or else Sunday in AUS or Monday morning15:48
pabelangerdocaedo: sure, lets see what pleia2 says15:49
docaedopabelanger: works for me :)15:50
clarkbnibalizer: tl;dr is "we had some deadlocks that prevented shards from being freed. Should be fixed in 2.1" :(15:51
*** xyang1 has quit IRC15:51
clarkbnibalizer: no mention of whether or not such deadlocks would servive a full cluster restart but that may be our next step15:51
clarkbnibalizer: I am willing to run it in this state for now though since it is functioning, then if we get back to unfunctioning state try full cluster restart15:52
nibalizerdid you run that sketchy curl to look at 'hot_threads' ?15:52
clarkbjesusaur: ^ FYI this may be interesting info to you15:52
*** mikelk has quit IRC15:52
clarkbnibalizer: I have not, but maybe we should15:52
clarkbnibalizer: I usually read the es docs on sketchy queries first :)15:52
nibalizerdo we have 'failed to lock' in our  logs?15:53
*** trown is now known as trown|lunch15:53
clarkbwe did yesterday when I was first debugging this, not sure if it has persisted15:53
openstackgerritDean Troyer proposed openstack-infra/irc-meetings: Move OSC team meeting even weeks to 1300 UTC Thursday
* clarkb uses grep15:53
* nibalizer not see it on 0515:53
clarkb07 was where I saw it yesterday let me see if I can pull it up there15:54
*** matrohon has quit IRC15:54
*** mrmartin has quit IRC15:54
clarkbnibalizer: found it15:54
nibalizerclarkb: clearly we need a log aggregation and indexing service to scan elasticsearch logs15:55
clarkbnibalizer: I have thought about that, but things get weird when you put elsaticsearch in elasticsearch15:55
*** andreww has joined #openstack-infra15:55
clarkbnibalizer: like you generate logs by indexing logs, and all of a sudden the snake is eating its tail15:55
*** psilvad has joined #openstack-infra15:57
nibalizerclarkb: haha yes im totally just joking around15:57
nibalizersecond ES cluster for system logs15:58
mordredclarkb, Shrews:
nibalizerthen if the systemlog ES cluster needs log analysis we just do it like this probably15:58
mordredShrews: so two things- create and list have completely different FIP structures, so we need a normalize15:58
mordredand that is the sequence I get on a successful FIP creation15:58
mordredso I believe create_floating_ip wants a wait/timeout15:58
EmilienMfungi: can you please clean files in
mordredso that it can wait for status==ACTIVE before returning15:59
mordredand we can short circuit the wait loop if we see port_id not populated15:59
fungiEmilienM: you just want all of those removed?15:59
EmilienMfungi: yes, please15:59
EmilienMfungi: just for puppet-aodh15:59
*** baoli_ has quit IRC16:00
fungiEmilienM: so puppet-aodh-7.0.0.tar.gz and puppet-aodh-tarball-master.tar.gz16:00
*** baoli has joined #openstack-infra16:00
EmilienMfungi: another question: would it be possible to generate tarballs for all our modules in master / stable/mitaka and stable/liberty ?16:00
*** esikachev has joined #openstack-infra16:00
fungiEmilienM: rather than waiting for new changes to merge for each of those branches?16:01
EmilienMfungi: right16:01
EmilienMfungi: specially for stable branches, where we have rare commits16:01
fungiEmilienM: it's possible but would be a fair amount of manual work to research and trigger all those jobs with the appropriate parameters16:01
*** claudiub has joined #openstack-infra16:02
*** lezbar__ has quit IRC16:02
EmilienMfungi: ok I see, so I better to commit on each repo and each branch16:02
fungiEmilienM: yeah, or just wait for them to appear over time, unless you urgently need them16:02
clarkbjeblair: mordred are you aware that we are failing to deliver root mail to you?16:03
*** dalgaaf has joined #openstack-infra16:03
*** andreww is now known as xarses16:03
mordredclarkb: nope16:03
EmilienMfungi: some of our operators need it16:03
mordredclarkb: I imagine our mail server thinks openstack is spam16:03
clarkbmordred: yes16:03
fungiclarkb: which messages? i get a lot of them that bounce for you, mordred, jeblair, nibalizer and yolanda16:03
EmilienMfungi: can't we trigger the post jobs manually?16:03
*** lezbar__ has joined #openstack-infra16:04
clarkbfungi: from graphite.o.o, looks like possibly related to not having AAAA and reverse ptr records?16:04
fungiclarkb: a major cause seems to be that teh from address doesn't actually pass sender callback tests16:04
mordredthat would certainly do it16:04
mordredthe reverse ptr record16:04
clarkbreverse ptr is there16:04
nibalizeri also get a lot of bounces16:04
clarkbbut AAAA isn't16:05
*** lezbar__ has quit IRC16:05
mordredI doubt we'd reject from our email server due to lack of AAAA - but I could be wrong16:05
*** lezbar__ has joined #openstack-infra16:05
mordredmaybe if it's coming in on IPV6 and we try to reverse then forward the ipv6 addr16:05
zxiirozaro: electrofelix : can you guys review this patch?  I'd like to get it merged before we cut a release.16:05
nibalizermordred: that would be pretty ambitious of you16:05
fungiEmilienM: an infra root admin can reenqueue refs or manually inject job work requests, but would need to loop over a list of project names, branch names and resolved branch tip git shas for all of them16:05
mordrednibalizer: well, we do the reverse check16:05
nibalizerso mail bounces trying to reach me?16:06
clarkbmordred:  550-2001:4800:7810:512:3bc3:d7f6:ff04:8201 is listed in the Spamhaus DNSBL. that is the real reason I think16:06
nibalizermy email is hosted at fastmail so there is limited control I have over that16:06
*** tphummel has joined #openstack-infra16:06
nibalizerbut if the mail gets to clark (who is on fastmail too I believe) then it should come to me fine16:06
*** inc0 has joined #openstack-infra16:06
clarkbnibalizer: correct fastmail here16:07
mordredclarkb: oh. yah16:07
*** fhubik has quit IRC16:07
*** lakshmiS has joined #openstack-infra16:07
*** xyang1 has joined #openstack-infra16:08
*** fhubik has joined #openstack-infra16:08
*** imcsk8|z1z is now known as imcsk816:08
*** arxcruz has quit IRC16:09
electrofelixzxiiro: have you seen  shows how to do version specific behaviour for the maven plugin16:09
krotscheckAJaeger: I'm not certain I understand your comment on - the idea with the patch is to use one core group - infra-javascript-core - for the various javascript projects that are generically infra. Removing the old ACL and replacing it with a new one. I'm guessing that editing this particular file is the wrong one, yes?16:09
*** hashar has quit IRC16:10
*** schipiga_ has quit IRC16:10
zxiiroelectrofelix: ah interseting... I'll bookmark it and see if i can implement that tonight16:10
*** brad_behle has joined #openstack-infra16:11
*** mhickey has quit IRC16:11
electrofelixwaynr zaro: I've got a few patches needing the final +2:,,,,,
zxiiroelectrofelix: when are we cutting the release?16:12
*** esikachev has quit IRC16:12
pleia2pabelanger, docaedo: any time today after 10:30am pacific works for me16:12
electrofelixzxiiro: I think Monday/Tuesday would be the sensible time just in case something sneaks in16:12
*** kjorgensen has joined #openstack-infra16:13
*** inc0 has quit IRC16:13
zxiiroelectrofelix: ok great, I really want that CFP patch in so I'll ping you tomorrow about getting it reviewed16:13
clarkbnibalizer: I don't really see anything in hot_threads that looks terrible, but I also do not know what I am looking for16:13
electrofelixwaynr: meant to ask, for the 2.0 work, with the patches targeted against a feature branch, what was the plan for merging in? Or if we cut a final 1.x release do you want to retarget them against master?16:13
openstackgerritIvan Kolodyazhny proposed openstack/requirements: Add global requirement update on python-brick-cinderclient-ext project
zxiirosadly i'm stuck in meetings all day today so can't work on it during the day16:13
fungikrotscheck: if it's going to be a lot, probably needs to be done in the same way as infra-puppet-core. we have a script in system-config to manage it
fungikrotscheck: if it's only going to be a few repos, we can just do it with repo-specific groups16:14
*** ddieterly is now known as ddieterly[away]16:15
*** ddieterly[away] is now known as ddieterly16:15
krotscheckfungi: I anticipate... 3, maybe 4 projects? Right now the only thing on the roadmap are the two mirror repos, a project generator (js-version of cookiecutter), and maybe moving eslit-config-openstack under the infra banner.16:15
docaedopleia2 pabelanger - any time today after 10:30am PST works for me too16:15
krotscheckIt may involve npm-publish-xstatic and some kind of a docs shim.16:15
*** brad_behle has left #openstack-infra16:15
fungikrotscheck: so by default we just create a specific core group for each repo, our script adds infra-core to the groups for it but we can also add other core reviewers individually to those repo-specific core groups16:16
fungifour doesn't sound like a lot to deal with manually16:17
openstackgerrityolanda.robla proposed openstack-infra/shade: Add initial setup for magnum in shade
krotscheckfungi: That's fair. Just stick to the normal groups naming convention then?16:17
fungikrotscheck: yeah, i vote for that16:17
clarkbnibalizer: I think we stick to what I originally proposed, let it run as is, if it breaks harder do a full cluster restart to hopefully free up the locks16:17
krotscheckfungi: Gotcha. I'll abandon that patch.16:18
fungikrotscheck: as soon as the governance changes for those repos merge i'll run the maintenance script to make sure the groups are done up correctly16:18
*** inc0 has joined #openstack-infra16:18
clarkbnibalizer: I tried to do hot threads based on cpu, block, and wait, don't really see anything glaring16:18
amrithsdague, would you help with this odd issue please; not sure if it is related to gerrit dash creator in some way but, see gist there are two queries. You'll notice that they differ in only one way, the second has a final expression "NOT Code-Review<=-2". The latter search unfortunately drops a lot of reviews which it should not; i.e. a bunch of reviews16:18
amrithwhich have no -2's are being ommitted.16:18
fungijust give me a heads up in case i miss that happening16:18
amrithsdague ^^16:18
*** dtardivel has quit IRC16:18
amriththis makes the dashboard suppress some reviews that it shouldn't16:18
krotscheckfungi; GOvernance for the first one landed. Not the second one.16:18
pabelangerdocaedo: pleia2: Ack, 10:30am PST then16:19
fungikrotscheck: also let me know if there are any outstanding additions to infra in governance lacking my +1 so i can take care of it16:19
nibalizerclarkb: wfm16:19
*** roxanag__ has joined #openstack-infra16:19
mordredsdague: is this: true simply for hysterical raisins?16:19
mtreinishclarkb, fungi, nibalizer, pleia2: quick patch to unblock puppet on health.o.o: should be good to go now16:20
krotscheckfungi: Urm... #308669, #308662, #30866316:20
*** nwkarsten has quit IRC16:20
docaedopabelanger pleia2 sounds good, let me know where we're calling in to :) Have used my own asterisk but not the openstack asterisk16:21
openstackgerrityolanda.robla proposed openstack-infra/shade: Add initial setup for magnum in shade
pleia2docaedo: info at
*** jistr has quit IRC16:21
*** roxanagh_ has quit IRC16:22
pleia2docaedo, pabelanger: say, bridge number 600216:22
*** bpokorny has joined #openstack-infra16:22
amrithsdague, a couple of google results indicate that rebuilding indexes on the database fixed the problem16:23
docaedopleia2: thanks, bridge 6002 in a touch over an hour16:23
*** madhu_ak has joined #openstack-infra16:23
fungikrotscheck: what's the corresponding governance patch for 308669?16:24
*** hdd has joined #openstack-infra16:24
*** flepied has quit IRC16:24
pleia2yay, new ubuntu lts :)
krotscheckfungi: urrr.. I don't think that one requires a governance patch.16:24
krotscheckpleia2: YAY!16:24
fungikrotscheck: oh, nevermind. i skimmed the commit message16:25
openstackgerritMerged openstack-infra/puppet-subunit2sql: Make main thread non-daemon
*** lucasagomes is now known as lucas-dinner16:26
Shrewsmordred: that is amazing16:26
fungipleia2: wow! i thought it wasn't due out for 5 more days16:26
pabelangerdocaedo: pleia2: WFM16:26
pleia2fungi: I thought about saying something when you said next week in the email yesterday, but then I forgot16:26
amrithanyone else in infra ...  see gist there are two queries. You'll notice that they differ in only one way, the second has a final expression "NOT Code-Review<=-2". The latter search unfortunately drops a lot of reviews which it should not; i.e. a bunch of reviews which have no -2's are being ommitted. a quick google search indicates that bad indexes may be16:27
amriththe problem.16:27
pabelangerpleia2: fungi: \o/16:27
pabelangerwonder if rax has it yet, we can bring our xenial wheel mirror online :)16:27
fungipleia2: yep, my dyslexia strikes again. i transposed the week number with the date there16:27
fungithought it said april 2616:28
yottatsaclarkb mordred speaking about spamhaus ipv6 dnsbl
prometheanfiregreghaynes: ianw: ya has my commit sitting in front of it, so will need to merge first16:29
*** ifarkas has quit IRC16:29
*** johnny has joined #openstack-infra16:30
yottatsaand it really listed in sbl
greghaynesprometheanfire: it's just waiting on my -1...16:30
*** johnny is now known as Guest8242316:31
prometheanfiregreghaynes: -1?16:31
openstackgerritMerged openstack-infra/puppet-subunit2sql: Stop using a seperate retreiver thread
greghaynesprometheanfire: yes -
mtreinishinfra-root: ^^^ we should be good to restart the subunit gearman workers now16:31
prometheanfiregreghaynes: ok, all the dependant commits will need rebasing then16:32
*** jlanoux has quit IRC16:32
greghaynesI think they do already16:32
prometheanfireok, didn't want to ruin all those +w16:32
mtreinishwell after puppet applies it on subunit-worker01.o.o and subunit-worker02.o.o16:32
greghaynesIt shouldn't, the +2's stay so you just need a code to +w what already has the +2's it needs16:33
greghaynesEr, a core16:33
*** mylu has quit IRC16:33
* greghaynes -> landing16:33
mordredShrews: I have a fun patch coming for you16:33
prometheanfiregreghaynes: k16:33
openstackgerritMerged openstack-infra/jenkins-job-builder: gerrit trigger: Fix doc on topics.
openstackgerritMatthew Thode proposed openstack/diskimage-builder: Add Gentoo to the dhcp-all-interfaces element
openstackgerritMerged openstack-infra/jenkins-job-builder: Added possibility to exclude project combinations
prometheanfiregreghaynes: removed the installtype: package16:36
AJaegerpabelanger: do you have time to check reviewday cron job again, please? The log file change should be active now16:36
clarkbamrith: just reading through that you have a redundant set of project clauses16:36
amrithI have that on purpose16:37
amrithbecause I want to restrict (in foreach) to only the trove projects16:37
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Make the delorean-ci repository available when build overcloud images
*** inc0 has quit IRC16:37
amrithbut some of the queries are for a subset16:37
amrithbut ...16:37
amrithI found the problem16:37
amrithit was an error in the query16:37
pabelangerAJaeger: sure in 15mins16:37
amriththe clause that gave trouble16:37
clarkbamrith: removing that fixes it16:37
AJaegerthanks, pabelanger16:37
amrithshould have been NOT label:CodeReview<=-216:38
*** baoli_ has joined #openstack-infra16:38
amrithI missed the label:16:38
openstackgerritMatthew Thode proposed openstack/diskimage-builder: Add Gentoo to the dhcp-all-interfaces element
pabelangermordred: q: what does No valid host was found error mean in nodepool?
amrithclarkb, removing what?16:38
pabelangerseeing that alot in ovh-gra116:38
clarkbamrith: if I remove the redundant clause I get the same results in both16:38
amriththat's right16:38
clarkbamrith: in any case I do not understand why you would have the redundant claause16:38
amriththat particular query in the dashboard16:38
amriththis came from a dashboard16:38
amrithtake a look at the review I just linked to16:39
amrithyou'll see the error I made16:39
clarkbys I understand the issue16:39
*** nadya has quit IRC16:39
clarkbI am saying that you shouldn't need to restrict projects twice16:39
openstackgerritMerged openstack-dev/hacking: Add support for PEP257
clarkbin addition to the issue you fixed16:39
amrithclarkb, how do I get around it?16:39
amrithI must have a foreach16:39
amrithso I could have status:open there16:39
clarkbthere is no foreach in the gits you linked?16:40
amrithsorry, please see
AJaegerpabelanger: quick 15mins ;) Thanks16:40
clarkbah I see16:40
pabelangerAJaeger: :)16:40
amrithI thought the issue was with the query and how gerrit-dash-creator did something16:40
amrithI presumed (incorrectly)16:40
amriththerefore I took just the query16:40
pabelangerAJaeger: I'm like scotty on the enterprise!16:40
*** baoli has quit IRC16:40
*** Guest14509 has quit IRC16:40
pabelangerAJaeger: guess we need a debug flag16:41
amrithIn fact the error was in the queries ... why it didn't generate an error saying CodeReview<=-2 makes no sense, I know not16:41
amrithbut ...16:41
AJaegerpabelanger: let me see what I can do with debug flag...16:41
amriththanks AJaeger16:41
amrithclarkb, does the fix appear correct to you?16:42
openstackgerritMonty Taylor proposed openstack-infra/shade: Fail if FIP doens't have the requested port_id
mordredShrews: ^^16:42
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: support ftp publisher
clarkbamrith: ya  Ithink you need the label there16:42
clarkbnot sure why it isn't erroring on the other behavior but yay gerrir16:42
clarkber gerrit16:42
amrithand oddly enough it only suppressed some reviews :)16:43
*** HeOS_ has joined #openstack-infra16:43
AJaegerpabelanger: seems it cannot find gerrit-dash-creator command.16:43
AJaegerpabelanger: when you run it manually, did you run it as reviewday user with same arguments as the cronjob16:44
pabelangerAJaeger: Ya, see that now16:44
pabelangerAJaeger: it is likely a PATH issue16:44
pabelangerAJaeger: I did16:44
openstackgerritMonty Taylor proposed openstack-infra/shade: Fail if FIP doens't have the requested port_id
openstackgerritValeriy Ponomaryov proposed openstack-infra/project-config: [Rally] Fix job for Manila project
mordredShrews: (updated to make sure we deduct fip wait time from the total timeout)16:45
openstackgerritMerged openstack-infra/jenkins-job-builder: Add support for disable-strict-forbidden-file-verification
pabelangerAJaeger: I suspect crontab doesn't know about /usr/local/bin16:45
pabelangerAJaeger: and we need to update the PATH variable16:45
*** schipiga has quit IRC16:45
*** ddieterly is now known as ddieterly[away]16:46
jeblairclarkb, fungi: can you send me a bounce message?16:47
*** inc0 has joined #openstack-infra16:47
AJaegerpabelanger: let me prepare a patch16:47
*** amitgandhinz has joined #openstack-infra16:47
clarkbjeblair: ya just forward you what was sent?16:48
*** baoli_ has quit IRC16:48
*** amitgandhinz has quit IRC16:48
*** baoli has joined #openstack-infra16:48
fungijeblair: i've been deleting the ones i receive but can bounce you the next one i get16:48
*** ayoung has quit IRC16:48
pabelangerAJaeger: we can use the environment param for cron in puppet16:48
clarkbI just fowarded one of the ones I got16:48
pabelangerAJaeger: just looking at syntax16:49
jeblairclarkb, fungi: yeah, whatever the backscatter message you get is16:49
*** amitgandhinz has joined #openstack-infra16:49
clarkbjeblair: mordred looks like other hosts are doing it too, not just graphite16:49
jeblairclarkb: perfect, thanks16:49
fungioh, actually i may have some in my cronspam dumping ground. i'll check16:49
*** alexpilotti has joined #openstack-infra16:49
*** arxcruz has joined #openstack-infra16:49
clarkbjeblair: wiki, logstash, vexxhost mirror, stakckalytics16:49
openstackgerritAndreas Jaeger proposed openstack-infra/puppet-reviewday: Add /usr/local/bin to path for cron job
pabelangerAJaeger: should be able to do environment => ['TMPDIR=/var/tmp', 'TMP=/var/tmp'],16:50
Shrewsmordred: fyi, there are comments for you to address on
Shrewsmordred: not from me, this time16:50
mordredShrews: woot!16:50
AJaegerpabelanger: 309100 does it differently but I can update16:50
jeblairclarkb: i have removed graphite from the sblcss16:50
pabelangerAJaeger: let me check16:50
*** ddieterly[away] is now known as ddieterly16:50
jeblairclarkb: can you send all of those to me?16:51
pabelangerAJaeger: okay, let me manually test that16:51
clarkbjeblair: yes16:51
fungiclarkb: jeblair: those are the same ones i'm finding in my cronspam inbox, yeah. i'll see if i spot any for other hosts16:51
openstackgerritMerged openstack-infra/jenkins-job-builder: slack notifications on repeated failure
*** apoorvad has joined #openstack-infra16:52
*** Nikolay_St has quit IRC16:52
AJaegerpabelanger: so, alternative should be  environment => 'PATH=/bin:/usr/bin:/usr/local/bin', - I'm no puppet expert, so need your advise on what's better16:53
pabelangerAJaeger: okay, testing in 8mins16:53
AJaegerpabelanger: thanks!16:53
pabelangerAJaeger: either should work16:53
*** roxanag__ has quit IRC16:53
openstackgerritMerged openstack-infra/jenkins-job-builder: multijob: add support for "ALWAYS" continuation condition
pabelangerAJaeger: question, should it be \$PATH?16:53
clarkbjeblair: there is a different one for nibalizer yolanda and my self complaining about root@jeblair-nodepool16:54
sdaguemordred: you are going to need to explain context16:54
*** sambetts is now known as sambetts|afk16:54
clarkbjeblair: I am guessing that is from when you made the new host off the nodepool snapshot16:54
*** HeOS_ has quit IRC16:54
jeblairclarkb: yep; fungi is that host still up (from the second ressurection?)16:55
AJaegerpabelanger: let me use environment - we do this elsewhere as well...16:55
mordredsdague: that's a snippet from a nova server dict, and from neutron net-list on the same cloud (I was tracking down another thing and it looked funny to me so I thought I'd capture it for discussion)16:55
*** ihrachys has quit IRC16:55
*** Goneri has quit IRC16:55
pabelangerAJaeger: WFM16:55
*** baoli has quit IRC16:55
mordredsdague: essentially, the nova addresses dict is showing a list of two addresses from the openstackci-network1 network16:55
*** alexpilotti has quit IRC16:55
sdaguemordred: I would not entirely be surprised if nova proxying neutron information about multiple networks is wrong16:55
mordredsdague: but the floating ip actually did not come from that network16:56
mordredsdague: ok. cool. I was mostly wondering if there was an interface intention there that I should wrap my head around16:56
mordredsdague: or if I should file a bug or something16:56
sdaguefiling a bug would be good16:56
*** psachin has quit IRC16:56
mordredsdague: also - related to that and your thread about killing the proxy things16:56
*** fedexo has quit IRC16:56
openstackgerritAndreas Jaeger proposed openstack-infra/puppet-reviewday: Add /usr/local/bin to path for cron job
openstackgerritMerged openstack-infra/project-config: Revert "Fix comment on requirements job"
AJaegerpabelanger: updated ^16:57
mordredsdague: are you planning on just killing proxy apis? or also killing the proxied data inside of the server dict (like that addresse dict)16:57
sdaguemordred: honestly, just the pure proxy apis16:57
mordredthat's what would make me happy16:57
mordredthe data in the server dict is actually invaluable16:57
pabelangerAJaeger: okay, updated local cron too16:57
sdaguemordred: agreed16:57
pabelangerAJaeger: 2 mins16:57
sdaguewhen nova needs to provide a composite view of resources, that's fine16:58
*** doug-fish has joined #openstack-infra16:58
sdagueI just want to get out of the business of middleman for interfaces that you really should be going to the main interface for16:58
clarkbfungi: I am not seeing rax-ord git fails in logstash (assuming I clicked on the right bug search link)16:58
*** Na3iL has quit IRC16:58
*** ociuhandu has quit IRC16:59
*** flepied has joined #openstack-infra16:59
sdaguenova servers not telling you about their networks and interfaces would just be crazy :)16:59
*** thiagop has quit IRC16:59
*** dalgaaf has quit IRC17:00
*** dalgaaf has joined #openstack-infra17:00
*** claudiub has quit IRC17:00
*** tosky_ has joined #openstack-infra17:01
*** yonglihe has quit IRC17:01
*** sputnik13 has joined #openstack-infra17:01
mordredsdague: indeed... but crazier things have happened before :)17:01
openstackLaunchpad bug 1573142 in OpenStack Compute (nova) "proxied neutron network information is incorrect" [Undecided,New]17:01
*** mahatic_ has quit IRC17:01
*** tosky has quit IRC17:01
mordredsdague: btw - this is not urgent or anyting - I mostly just happened upon it17:01
*** nwkarsten has joined #openstack-infra17:02
*** nwkarsten has quit IRC17:02
sdaguemordred: any chance you have a reproduce for a devstack env on that?17:02
*** trown|lunch is now known as trown17:02
*** doug-fis_ has quit IRC17:02
*** ddieterly has quit IRC17:02
*** abregman|pto has quit IRC17:02
*** pradk has quit IRC17:02
*** yarkot has quit IRC17:02
*** kevinbenton has quit IRC17:02
sdagueit would be handy to make it easier to chase down17:02
*** nwkarsten has joined #openstack-infra17:02
mordredsdague: I don't - but it should be super easy - make a vm with a floating ip, then grab the /servers/{id} for it - and also do a neutron net-list17:02
*** mahatic has joined #openstack-infra17:02
mordredlemme say that on the bug17:03
*** fungi has quit IRC17:03
*** msuriar has quit IRC17:03
*** kevinshort has quit IRC17:03
*** jgriffith has quit IRC17:03
*** serverascode has quit IRC17:03
*** yonglihe has joined #openstack-infra17:03
*** sridhar_ram has joined #openstack-infra17:03
*** dahoo has joined #openstack-infra17:03
*** kevinshort has joined #openstack-infra17:03
*** jgriffith has joined #openstack-infra17:03
*** rossella_s has quit IRC17:03
*** fungi has joined #openstack-infra17:04
*** lindsayk_ has quit IRC17:04
*** sc68cal has quit IRC17:04
fungijeblair: oh, yeah i mentioned an hour or so later that i was done with it but didn't know if i should delete it myself or if you wanted anything from it too17:04
*** jgriffith is now known as Guest389417:04
fungiclarkb: i've been checking for them too, so far no hits in ord17:04
*** lindsayk_ has joined #openstack-infra17:04
*** cbader has quit IRC17:04
*** rossella_s has joined #openstack-infra17:04
openstackgerritMerged openstack-infra/grafyaml: Enable no fill in seriesOverrides
*** msuriar has joined #openstack-infra17:04
*** cbader has joined #openstack-infra17:04
*** bookwar has quit IRC17:04
*** sc68cal has joined #openstack-infra17:04
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/shade: Add Designate recordsets support
*** derekh has quit IRC17:05
*** kevinbenton has joined #openstack-infra17:05
*** arif-ali has quit IRC17:05
*** gongysh has quit IRC17:05
*** yarkot has joined #openstack-infra17:05
fungiclarkb: if this continues, i'll push a change later to go back to our full quota there17:06
*** serverascode has joined #openstack-infra17:06
*** sdake_ has joined #openstack-infra17:06
*** stevemar has quit IRC17:06
*** mnencia has quit IRC17:06
fungiclarkb: most recent hit for the patterns associated with that bug were in dfw once late yesterday17:07
*** sdake has quit IRC17:07
*** sshnaidm has quit IRC17:07
*** Kiall has quit IRC17:07
*** gabriel-bezerra has quit IRC17:07
fungiand that's the only one in 6 days17:07
*** arif-ali has joined #openstack-infra17:07
openstackgerritMonty Taylor proposed openstack-infra/shade: Add public helper method for cleaning floating ips
openstackgerritMonty Taylor proposed openstack-infra/shade: Fail if FIP doens't have the requested port_id
fungiclarkb: but not sure what the state of elasticsearch data is right now, so wasn't putting a lot of stock in it yet17:08
*** Yarboa is now known as Yarboa_Zzzzz17:08
*** rfolco has quit IRC17:08
*** stevemar has joined #openstack-infra17:08
clarkbfungi: maybe you can confirm that git is looking happy and we can use all of rax-ord again?17:08
clarkbalso what was the deal with ovh earlier today? I thought we had already reenabled them?17:08
clarkbjust a recurrence I guess?17:08
*** gabriel-bezerra has joined #openstack-infra17:08
openstackgerritMerged openstack-infra/grafyaml: Add support for time range
*** dalgaaf has quit IRC17:08
*** niska has quit IRC17:08
*** cschwede has quit IRC17:08
*** njohnston_afk has quit IRC17:08
*** ekarlso has quit IRC17:08
*** rfolco has joined #openstack-infra17:08
*** mnencia_ has joined #openstack-infra17:09
*** sdake has joined #openstack-infra17:09
*** cschwede has joined #openstack-infra17:09
*** mnencia_ is now known as mnencia17:09
*** ramishra has quit IRC17:09
AJaegermordred: could you review yolanda's change to update the shade jobs, please?
*** bookwar has joined #openstack-infra17:09
*** niska has joined #openstack-infra17:09
*** rkukura_ has joined #openstack-infra17:09
openstackgerritClark Boylan proposed openstack-infra/elastic-recheck: Remove query about yum missing on ubuntu
*** Kiall has joined #openstack-infra17:10
*** ramishra has joined #openstack-infra17:10
clarkbfungi: the data from yesterday forward should be good17:10
*** sridhar_ram1 has joined #openstack-infra17:10
clarkbthen there is a gap with the ~3 days before yesterday17:10
*** sdake_ has quit IRC17:11
fungiand nodepool shows we're definitely running jobs in ord17:11
mordredAJaeger: lgtm - maybe lets check wih Shrews too17:11
*** pcm_ has joined #openstack-infra17:11
AJaegerShrews: please review as well..17:11
*** zz_ja has joined #openstack-infra17:12
openstackgerritMerged openstack-infra/project-config: Don't check for proposed branches in propose_update
*** dalgaaf has joined #openstack-infra17:12
*** fernnest_ has joined #openstack-infra17:12
*** sshnaidm has joined #openstack-infra17:12
*** armax_ has joined #openstack-infra17:13
Shrewsmordred: does that style of plugin enablement mean the latest version from git is used?17:13
Shrewsmordred: b/c if so, couldn't that be a problem with both magnum AND designate being used in our stable/kilo keystonev2 job?17:14
*** zhiyan_ has joined #openstack-infra17:14
*** abregman|pto has joined #openstack-infra17:14
*** tristanC_ has joined #openstack-infra17:14
pabelangerAJaeger: okay, check reviewday. I think it looks right17:14
clarkbShrews: devstack-gate will checkout the appropriate ref for each job. So it will be from git but if testing stable/kilo you should get latest commit from kilo and so on17:14
*** sc68cal_ has joined #openstack-infra17:14
*** njohnston has joined #openstack-infra17:14
Shrewsclarkb: ah, excellent. thanks17:15
*** vdrok_ has joined #openstack-infra17:15
*** pradk has joined #openstack-infra17:15
*** inc0 has quit IRC17:15
*** nelsnels_ has joined #openstack-infra17:15
clarkbmordred: re nodepool did sdake's IP thing ever get sorted?17:15
*** pleia2_ has joined #openstack-infra17:15
*** yamamoto has quit IRC17:15
*** Vivek has joined #openstack-infra17:15
*** inc0 has joined #openstack-infra17:15
clarkbthere is far too much nodepool stuff in flux right now ;17:15
sdakeclarkb we worked around it17:15
clarkbsdake: thanks, I assume that means the issue generally persists17:16
*** Vivek is now known as Guest5444817:16
*** shardy has quit IRC17:16
sdakeright same issue17:16
*** arif-ali has quit IRC17:16
*** abregman|pto has quit IRC17:17
clarkbmordred: maybe we can do a state of the union today before we all get on airplanes?17:17
*** mrodden_ has joined #openstack-infra17:17
*** jlk` has joined #openstack-infra17:18
openstackgerritMerged openstack-infra/project-config: Block stable/feature branch for new project imports
*** jmccrory- has joined #openstack-infra17:18
*** electrofelix1 has joined #openstack-infra17:18
*** electrofelix has quit IRC17:18
vikramCan someone plz approve this governance patch ""17:18
*** uberjay_ has joined #openstack-infra17:18
*** Ng_ has joined #openstack-infra17:18
*** aorourke_ has joined #openstack-infra17:19
*** sc68cal has quit IRC17:19
*** rossella_s has quit IRC17:19
*** lindsayk_ has quit IRC17:19
*** sridhar_ram has quit IRC17:19
*** armax has quit IRC17:19
*** yamahata has quit IRC17:19
*** kushal has quit IRC17:19
*** dizquierdo has quit IRC17:19
*** Yarboa_Zzzzz has quit IRC17:19
*** rkukura has quit IRC17:19
*** r-mibu has quit IRC17:19
*** Adri2000 has quit IRC17:19
*** pfallenop has quit IRC17:19
*** nt has quit IRC17:19
*** fernnest has quit IRC17:19
*** amit213 has quit IRC17:19
*** nelsnelson has quit IRC17:19
*** jmccrory has quit IRC17:19
*** pleia2 has quit IRC17:19
*** tchaypo has quit IRC17:19
*** Guest32495 has quit IRC17:19
*** zz_zz_ja has quit IRC17:19
*** jlk has quit IRC17:19
*** uberjay has quit IRC17:19
*** pc_m has quit IRC17:19
*** davideagnello has quit IRC17:19
*** mrodden has quit IRC17:19
*** dmsimard has quit IRC17:19
*** sweston has quit IRC17:19
*** Ng has quit IRC17:19
*** jasondotstar has quit IRC17:19
*** a2hill has quit IRC17:19
*** tristanC has quit IRC17:19
*** lbragstad has quit IRC17:19
*** vdrok has quit IRC17:19
*** rguillebert has quit IRC17:19
*** etoews has quit IRC17:19
*** zhiyan has quit IRC17:19
*** aorourke__ has quit IRC17:19
*** armax_ is now known as armax17:19
*** electrofelix1 is now known as help17:19
*** mrodden_ is now known as mrodden17:19
*** rkukura_ is now known as rkukura17:19
*** yamahata_ has joined #openstack-infra17:19
*** jmccrory- is now known as jmccrory17:19
*** dmsimard has joined #openstack-infra17:19
*** help is now known as electrofelix17:19
*** ptoohill has joined #openstack-infra17:20
*** tchaypo has joined #openstack-infra17:20
*** rossella_s has joined #openstack-infra17:20
*** vdrok_ is now known as vdrok17:20
*** pfallenop has joined #openstack-infra17:20
*** ekarlso has joined #openstack-infra17:20
*** pleia2_ is now known as pleia217:20
*** sweston has joined #openstack-infra17:21
*** weshay is now known as weshay_food17:21
*** Adri2000 has joined #openstack-infra17:21
*** lbragstad has joined #openstack-infra17:22
*** alexpilotti has joined #openstack-infra17:22
*** amit213 has joined #openstack-infra17:22
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/shade: Add Designate recordsets support
*** daemontool has quit IRC17:22
pabelangervikram: you'll have to wait for the TC to finish the rollcall17:22
mordredclarkb: yes - although I'm not getting on a plane today :)17:23
*** alexpilotti has quit IRC17:23
vikraminfra_core: Can someone please help modifying the ACL for the newly created project via
mordredclarkb: howeer, yes, I think that's a good idea17:23
mordredclarkb: have you looked at the most recent crazy patch btw?17:23
*** etoews has joined #openstack-infra17:23
clarkbmordred: no I am just trying to figure out what the current state of things is17:23
*** zhiyan_ is now known as zhiyan17:23
*** sc68cal_ has quit IRC17:23
AJaegerpabelanger: woot! Yeah!17:23
clarkbseems like the private/public ip behavior is still not addressed and we leak floating IPs due to the port not showing up on the ip after you attach it to the port?17:23
*** lindsayk_ has joined #openstack-infra17:23
AJaegerpabelanger: thanks!17:24
vikrampabelanger: currently, it has a rollcall-vote by Thierry Carrez17:24
*** sc68cal has joined #openstack-infra17:24
AJaegerany other infra-root to review for me, please?17:24
mordredclarkb: yes. or, we're still leaking fips but we think we have found a way to detect the leak early and abort17:24
*** jasondotstar has joined #openstack-infra17:24
*** abregman has joined #openstack-infra17:24
AJaegervikram: governance is approved by ttx - and only by him (ttx=Thierry)17:25
waynrelectrofelix: i'll look at those patches today. as for the 2.0 work, we went with a separate branch originally so 1.x and 2.x reviews could happen in tandem but as it turns out we are waiting for a final 1.x release anyway...i'll just retarget against master eventually as long as after the final release we only take 1.x branch patches on a nonmaster 1.x-specific branch17:25
mordredclarkb: needs to be approved so that we can get the nodepool fip cleanup in17:25
AJaegervikram: ttx will approve after waiting for a week, no worries, this is pure housekeeping now17:25
*** kushal has joined #openstack-infra17:25
mordredclarkb: I need to put one more catch/delete into the latest patch to detect the lack of port_id17:25
mordredclarkb: and then add tests17:25
*** ayoung has joined #openstack-infra17:26
vikramAJaeger: ok.. got it .. thanks17:26
*** nt has joined #openstack-infra17:26
*** MarkAtwood has joined #openstack-infra17:26
*** davideagnello has joined #openstack-infra17:26
*** psilvad has quit IRC17:26
*** jordanP has quit IRC17:27
pabelangersdake: you still on track to release kolla today?17:27
*** alex_xu has quit IRC17:27
sdakepabelanger yes17:27
sdakeboth 1.1.0 and 2.0.017:27
*** Na3iL has joined #openstack-infra17:27
AJaegerthanks, fungi!17:28
openstackgerritMerged openstack-infra/jenkins-job-builder: Add user/password command line argument
*** fawadkhaliq has joined #openstack-infra17:29
clarkbmordred: are the fips that don't have the correct port functional? or do they also fail to work17:30
openstackgerritMerged openstack-infra/system-config: Remove proposed/ acls
*** rguillebert has joined #openstack-infra17:30
mordredclarkb: althugh I have tested it against osic and it does work (I have not run it enough times to see it leak)17:30
mordredclarkb: I have not been able to capture that state yet17:30
pabelangersdake: great, please let me know when you've finished and re-review 301320.  Thanks again17:30
*** dtantsur is now known as dtantsur|afk17:30
*** kushal has quit IRC17:30
sdakewill do17:30
sdakewe are trgeting 5pm pst17:31
mordredclarkb: the one thing I have been able to capture is the transition from DOWN to ACTIVE as normally occuring in all cases, and the leaked fips having no port_id17:31
*** salv-orlando has joined #openstack-infra17:31
pabelangersdake: wfm17:31
*** tqtran has joined #openstack-infra17:31
mordredclarkb: leaked fips returning with no port_id is conjecture17:31
*** alex_xu has joined #openstack-infra17:31
pabelangerclarkb: ianw: looks like we should be able to finish our devstack-centos7 migration later tonight17:31
openstackgerritMonty Taylor proposed openstack-infra/shade: Fail if FIP doens't have the requested port_id
*** maestro has joined #openstack-infra17:32
*** maestro has left #openstack-infra17:32
docaedopabelanger pleia2 - I'll be a minute late, dial in number for pbx is busy for me so will try jitsi17:32
*** maestro has joined #openstack-infra17:32
fungivikram: i have added armax as the initial member of the neutron-dynamic-routing-core group in gerrit since it's proposed for inclusion in neutron. he can redelegate it as needed17:32
pabelangerdocaedo: np, dialing in now17:32
armaxfungi: ack17:32
pleia2docaedo: it's busy for me too17:33
mordredclarkb: but with the new code, if we do keep leaking and see that tey have no port_id when leaked, we'll at least know they _had_ a port id at at least somepoint in their past17:33
vikramfungi: thanks17:33
pleia2I don't know what that means :)17:33
*** dizquierdo has joined #openstack-infra17:33
pabelangerpleia2: we might be out for credits17:33
docaedopleia2: ah well glad I have a little company there at least17:33
*** jed56 has quit IRC17:33
openstackgerritMerged openstack-infra/project-config: Enable magnum plugin on shade and use our atomic generated images
maestroHi all.. I have cloned Disk Image builder in my devstack environment and started to prepare ubuntu image by "disk-image-create -a amd64 -o ubuntu-amd64 vm ubuntu" while doing getting error as "+ /tmp/image.gu56sfk7/hooks/bin/cache-url /home/pandy/.cache/image-create/SHA256SUMS.ubuntu.trusty.amd6417:34
maestro* Hostname was NOT found in DNS cache17:34
maestro  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current17:34
maestro                                 Dload  Upload   Total   Spent    Left  Speed17:34
maestro  0     0    0     0    0     0      0      0 --:--:--  0:00:19 --:--:--     0* Could not resolve host: cloud-images.ubuntu.com17:34
maestro* Closing connection 017:34
maestrocurl: (6) Could not resolve host: cloud-images.ubuntu.com17:34
maestroUnmount /tmp/image.gu56sfk7/mnt/tmp/ccache17:34
armaxvikram: I had a first set:,members17:34
*** r-mibu has joined #openstack-infra17:34
*** salv-orlando has quit IRC17:34
armaxvikram: please refresh this page:
fungimaestro: please use a paste site like in the future17:34
*** salv-orlando has joined #openstack-infra17:34
*** jlk` is now known as jlk17:35
electrofelixwaynr: yeah, I think that's the best approach. I'm going to continue working on review stuff tomorrow, and I'll probably do a pass on Sat to tidy up any patches that have a -1 on them for minor tweaks to see if we can't close out on most things.17:35
fungimaestro: resolves in dns for me17:35
maestrosure, getting error while preparing ubuntu image via disk image builder
vikramarmax: what I should do?17:36
pabelangerdocaedo: pleia2: restarting asterisk now17:36
*** maestro has left #openstack-infra17:36
clarkbmordred: commented on (note it is previous patchset, you keep beating my comments to gerrit with new patchsets)17:36
fungimaestro: are you able to resolve any dns names from that machine? i think your dns resolution is broken17:36
pleia2pabelanger: thanks17:36
*** vikram has left #openstack-infra17:37
pabelangerdocaedo: pleia2: okay, better17:37
*** vikram has joined #openstack-infra17:37
*** kurtmartin has joined #openstack-infra17:37
*** tpsilva has quit IRC17:37
pabelangerpossible deadlock on asterisk17:37
*** ybathia has joined #openstack-infra17:37
pabelangerwe'll need to enable debug mode to see17:37
*** kurtmartin has quit IRC17:37
*** dizquierdo has quit IRC17:38
mordredclarkb: I do that. sorry :)17:38
waynrelectrofelix: cool17:39
pabelangerpleia2: docaedo:
openstackgerritDarragh Bailey proposed openstack-infra/jenkins-job-builder: Tidy up True/False and use better exceptions
*** vikram has quit IRC17:40
Shrewsclarkb: i let mordred get through at least 5 patchsets before i even try to comment  :)17:40
mordredclarkb: the second call for attach is because the floating ip create might have used nova. the attach wait skips the actual attach call if it's not needed - but that code is what gets the updated server dict (and waits for nova to indicate that the fip has attached- the new wait in create only waits to see that neutron is now satisfied that this ip is happy)17:41
mordredclarkb: to be fair, the flow here could certainly be refactored - but I'd rather not do that surgery at the moment17:41
clarkbmordred: maybe we can at least point that out then? That seems desrving of a comment becuase the flow isn't really intuitive17:42
*** vikram has joined #openstack-infra17:42
mordredI agree17:43
openstackgerritDarragh Bailey proposed openstack-infra/jenkins-job-builder: Allow using lockfile per jenkins master
electrofelixyolanda: I've restored your lockfile work and combined it with a change to ensure unlocking only occurs when the cache object is finally deleted during a run. I've yet to review use of pylockfile as a cross platform solution though.17:46
*** mpaolino has quit IRC17:46
clarkbelectrofelix: I just left a comment on it, I am confused about how a per master lockfile helps17:46
clarkbI think the lock needs to be per cache17:46
vikramarmax: will refresh it17:46
electrofelixclarkb: the cache is per master17:46
clarkbit is?17:46
*** ayoung has quit IRC17:47
*** flepied has quit IRC17:47
electrofelixyes, uses the master url as the cache file name17:47
clarkbelectrofelix: iirc only the what is in master foo part of the cache is per master17:47
clarkbbut separately there is the listing of all the jobs built locally right?17:47
*** ayoung has joined #openstack-infra17:47
*** Sukhdev has joined #openstack-infra17:47
clarkbso two runs could trample each other on the build side? maybe I am mistaken17:47
electrofelixif you're using updating the files directly on the disk, and point two runs to the same path, yes17:48
clarkbya I think that is what we are trying to prevent17:49
AJaegerrossella_s: the neutron dashboard is now updated every 30 mins on
*** Goneri has joined #openstack-infra17:49
electrofelixclarkb: slightly different issues, but still probably a bad idea to allow two runs on the same machine, targeting the same jenkins master via the API and using the same cache file without locking.17:49
clarkbelectrofelix: the idea behind the lock was to prevent two runs on the same machine17:50
clarkbregardless of master17:50
clarkb(I still think not using a lock and instead juts mv'ing the cache at the end would be fine17:51
electrofelixtwo runs on the same machine is not necessarily invalid, since it is perfectly fine to target different masters simultaneously.17:51
electrofelixensuring a single run regardless probably requires a global lockfile17:52
clarkbelectrofelix: correct, which is why I think the simple just mv the cache idea is good enough17:52
clarkbelectrofelix: basically I agree with you on what you are trying to do, I disagree on the implementation17:52
openstackgerritKiall Mac Innes proposed openstack/requirements: Add openstack/designate-tempest-plugin to projects list
clarkbelectrofelix: If we are going to go through the trouble of using a lock that lock should be global IMO17:52
clarkbbecause the issue for what you are trying to fix can just be handled iwth smarter file writing aiui17:53
electrofelixwhen prevents two runs targetting different masters, that would not clash17:53
clarkband is much simpler than a lockfile17:53
clarkband has the added benefit of not corrupting the cache on unfortunate shutdown17:54
*** tosky_ has quit IRC17:54
openstackgerritKiall Mac Innes proposed openstack/requirements: Add openstack/designate-tempest-plugin to projects list
electrofelixI'm not sure that it covers all use cases, but yes moving would make the write atomic and less prone to issues.17:55
*** ybathia has quit IRC17:55
*** ybathia has joined #openstack-infra17:55
electrofelixI'll think on the different use cases, and add that to the review first. already marked it as a WIP in the mean time.17:55
*** sigmavirus24 is now known as sigmavirus24_awa17:55
clarkbelectrofelix: in the case of two or more runs against the same master using mv and not a lockfile they will fight each other over the jenkins api but that is jenkins' problem and I believe it does the correct thing. Then when each one finishes they will mv their files over and the last one to complete wins17:56
*** SumitNaiksatam has joined #openstack-infra17:56
clarkbwhich is a slightly different behavior than last one to start wins17:56
*** electrofelix has quit IRC17:57
clarkband in really unfortunate cases you will have them overwrite each other in potentially unfortunate ways17:57
openstackgerritmelanie witt proposed openstack-infra/project-config: Add a new devstack plugin based src job for ceph
clarkbbut the issue we have been having is the shtudown case more so than the run multiples case17:57
AJaegerteam, project-config change by hashar has not seen any review. It's titled "Pass ZUUL_CACHE_NO_HARDLINKS to prevent hardlinking". Could one of the zuul experts have a look at it, please?17:57
*** baoli has joined #openstack-infra17:58
clarkbAJaeger: I thought we wanted hardlinks...17:58
clarkbhuh I misread. Zuul currently does not hardlink17:59
clarkbso this is preserving the old zuul behavior once zuul updates17:59
AJaegerclarkb: the other patch hasn't merged either - so we could wait some more as well...18:01
*** vikram has quit IRC18:01
clarkbI wonder why we don't want hardlinks18:01
clarkbseems like that would be a good idea18:01
*** doug-fis_ has joined #openstack-infra18:02
amrithAJaeger, found another one with the same problem as the trove-dashboard. please take a look at
Shrewsmordred: i left a comment for you on 30906018:04
amrithclarkb ^^18:04
rossella_sAJaeger, thanks a lot, you are my hero :)18:04
AJaegerthanks, amrith18:04
AJaegerrossella_s: pabelanger did the detective work for us!18:04
*** doug-fish has quit IRC18:04
rossella_sthanks pabelanger I really appreciate!18:05
openstackgerritmelanie witt proposed openstack-infra/project-config: Add a new devstack plugin based src job for ceph
*** flepied has joined #openstack-infra18:05
pabelangerAJaeger: rossella_s: np!18:06
AJaegerclarkb: thanks, let's wait some more I guess ;)18:06
AJaegerWe have  to make the beaker tests *voting* on infra puppet modules. fungi, clarkb, mordred, yolanda, pleia2: Ok to merge? nibalizer and pabelanger gave their +2 but the +W is missing. And since this changes quite a bit, I wanted to ask for feedback...18:08
openstackgerritMerged openstack/requirements: Updated from generate-constraints
clarkbdid we fix centos7 beaker package installs?18:08
*** mriedem has quit IRC18:09
*** sigmavirus24_awa is now known as sigmavirus2418:09
crinkleclarkb: should have
openstackgerritOpenStack Proposal Bot proposed openstack-dev/hacking: Updated from global requirements
AJaegerclarkb: yes, we did18:10
clarkbAJaeger: seems fine to me then18:10
*** mriedem has joined #openstack-infra18:10
openstackgerritOpenStack Proposal Bot proposed openstack/diskimage-builder: Updated from global requirements
*** inc0 has quit IRC18:11
openstackgerritMerged openstack/requirements: Bump oslo.log and oslo.il8n for april 18th releases
*** yamahata_ has quit IRC18:13
*** EricGonc_ has joined #openstack-infra18:14
*** degorenko is now known as _degorenko|afk18:14
*** mrmartin has joined #openstack-infra18:15
EmilienMfungi: re- tarballs, you mentionned I would have to provide a list of project names, branch names and resolved branch tip git shas for all of them -- can't we use HEAD of each branch?18:15
*** yamamoto has joined #openstack-infra18:16
*** EricGonczer_ has quit IRC18:16
*** baoli has quit IRC18:19
*** claudiub has joined #openstack-infra18:19
*** mrmartin has quit IRC18:19
bknudsonI could use some help setting up other-requirements.txt in keystone --
bknudsonhave other projects with db setup done this already?18:21
*** ayoung has quit IRC18:21
clarkbbknudson: you are missing mysql18:21
*** Ng_ is now known as Ng18:22
AJaegerbknudson: you need the db server as well18:22
bknudsonmaybe is a good example18:22
AJaegerbknudson: lines 27-30 is what you need for sure18:22
*** sdake has quit IRC18:23
*** rfolco has quit IRC18:23
*** EricGonczer_ has joined #openstack-infra18:24
bknudsonthanks. We'll see how it goes.18:24
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: support zuul stop commands
*** rfolco has joined #openstack-infra18:25
bknudson is the keystone other-requirements.txt overriding requirements/other-requirements.txt ?18:25
AJaegerbknudson: not at all - it's overriding project-config/jenkins/data/bindep-fallback.txt18:25
*** yamamoto has quit IRC18:25
*** EricGonc_ has quit IRC18:25
fungiEmilienM: an explicit tarball job invocation looks like: sudo /opt/zuul/tools/ --url= --job puppet-aodh-branch-tarball --project openstack/puppet-aodh --pipeline post --newrev 607957dd427cb7369e5a8af8a3544a5340556f0e --refname refs/heads/stable/liberty --logpath 60/607957dd427cb7369e5a8af8a3544a5340556f0e/post/puppet-aodh-branch-tarball/manual_000/18:26
bknudsonAJaeger: ok, thanks. Maybe I should copy from there.18:26
fungiEmilienM: or we can reenqueue all jobs in post for a given branch tip like: sudo zuul enqueue-ref --trigger gerrit --pipeline post --project o18:26
fungipenstack/puppet-aodh --ref refs/heads/stable/liberty --newrev 607957dd427cb7369e5a8af8a3544a5340556f0e18:26
fungi(minus the stray newline in there)18:26
AJaegerbknudson: copy from there and remove what you don't need ;)18:27
*** arxcruz has quit IRC18:27
bknudsonlooks like that uses mariadb for rpm18:27
AJaegerbknudson: the bindep file is what keystone currently uses18:27
*** megm has joined #openstack-infra18:27
AJaegerso, your other-requirements overrides bindep-fallback18:27
fungiEmilienM: but either way we need the result of git show-ref for each branch tip to compose the parameters for the command18:27
EmilienMfungi: could you just try with puppet-aodh for master, stable/mitaka and stable/liberty - so we first see if the stable builds are ok18:28
fungiEmilienM: sure, i can do that18:28
EmilienMfungi: thanks18:28
EmilienMfungi: I can write a script that prepare all commands automatically so we don't have to waste time on it18:28
*** fhubik has quit IRC18:30
*** Sukhdev has quit IRC18:30
*** inc0 has joined #openstack-infra18:32
*** vhosakot has quit IRC18:33
*** vhosakot has joined #openstack-infra18:33
*** yamahata_ has joined #openstack-infra18:33
*** etoews has left #openstack-infra18:34
*** woodster_ has joined #openstack-infra18:34
*** salv-orlando has quit IRC18:44
*** armax has quit IRC18:44
*** armax has joined #openstack-infra18:45
*** ihrachys has joined #openstack-infra18:45
*** pvaneck has joined #openstack-infra18:48
*** ybathia has quit IRC18:49
*** armax has quit IRC18:50
openstackgerritMerged openstack-infra/devstack-gate: Fail the test if the pre test hook fails
*** mrmartin has joined #openstack-infra18:52
*** Sukhdev has joined #openstack-infra18:55
*** cody-somerville has joined #openstack-infra18:57
*** flepied1 has joined #openstack-infra18:58
*** bpokorny_ has joined #openstack-infra18:59
*** mylu has joined #openstack-infra18:59
fungiEmilienM: btw, i've got the liberty one queued, but it's taking a while to get a worker assigned i think19:00
EmilienMfungi: 193 jobs in post :-)19:00
fungiEmilienM: oh, also i had the syntax subtly wrong for, it's --refname stable/liberty not --refname refs/heads/stable/liberty19:01
*** flepied has quit IRC19:01
*** inc0 has quit IRC19:01
*** lennyb has quit IRC19:01
*** bpokorny_ has quit IRC19:01
*** esker has joined #openstack-infra19:02
*** esker has quit IRC19:02
*** baoli has joined #openstack-infra19:02
EmilienMfungi: tbh, I prefer the cost of doing this manually versus triggering ~400 jobs19:02
*** bpokorny_ has joined #openstack-infra19:02
*** esker has joined #openstack-infra19:02
EmilienMfungi: but first, we need to make sure the stable builds are okay, that's why I asked if we can first check puppet-aodh19:02
*** bpokorny has quit IRC19:03
*** madhu_ak has quit IRC19:04
*** amotoki has quit IRC19:05
*** doug-fis_ has quit IRC19:06
AJaegerfungi, ok to make the beaker jobs voting for infra puppet?
*** edmondsw has joined #openstack-infra19:07
*** esker has quit IRC19:07
*** esikachev has joined #openstack-infra19:08
*** doug-fish has joined #openstack-infra19:08
*** baoli has quit IRC19:08
*** mrmartin has quit IRC19:09
*** salv-orlando has joined #openstack-infra19:10
EmilienMAJaeger, crinkle: I'm not sure it's a good idea - some modules have failures, see
mtreinishfungi, jeblair: if you get a sec I think we should be good to restart the subunit-workers now19:10
jeblairmtreinish: that sounds like a fun lunchtime activity, sure19:10
EmilienMAJaeger, crinkle: that will be fixed by
jeblairmtreinish: we expect them to not be running right now, correct?19:11
mtreinishjeblair: right, unless puppet started them up without us noticing19:11
jeblairmtreinish: nope, i don't see them.  just wanted to make sure that was expected state, and it seems to be.19:11
*** HeOS has quit IRC19:12
EmilienMcrinkle: well, +1 anyway. See comment19:13
jeblairmtreinish: worker A on 01 is running19:13
jeblairit's handling jobs.  so far they are *entirely* 404s19:13
crinkleEmilienM: :)19:13
mtreinishjeblair: \o/19:13
jeblairmtreinish: which seems right since they are almost all translation update jobs19:14
mtreinishheh, yeah there isn't any subunit emitted by those19:14
jeblairoh it got one!19:14
*** ybathia has joined #openstack-infra19:14
*** esker has joined #openstack-infra19:14
jeblair2016-04-21 19:14:25,158 Handling event: {"fields": {"build_name": "gate-tempest-dsvm-neutron-full", "build_status": "SUCCESS", "build_branch": "master", "build_master": "", "build_ref": "refs/zuul/master/Z5045bdd19db04f71ae250ba6021cbba7", "log_url": "", "build_change": "308171", "node_provider": ...19:14
jeblair... "rax-dfw", "build_node": "ubuntu-trusty-rax-dfw-391573", "project": "openstack/requirements", "build_zuul_url": "", "filename": "logs/testrepository.subunit", "build_uuid": "2066ea133b544eb4924a71f941b63c68", "build_queue": "gate", "build_short_uuid": "2066ea1", "build_patchset": "1", "voting": "1"}, "tags": ["testrepository.subunit"]}19:14
jeblair2016-04-21 19:15:08,683 Sending packet to <gear.Connection 0x244f150 host: port: 4730>: <gear.Packet 0x11addd0 type: WORK_COMPLETE handle: H:>19:15
jeblairand done19:15
jeblairmtreinish: and then continued19:15
jeblairmtreinish: so all looks good here19:15
jeblairmtreinish: i'll restart 02 to help work through the backlog19:16
mtreinishjeblair: yay, yep should be good for now19:16
mtreinishjeblair: ++19:16
jeblairmtreinish: then later we can see whether we need the second host19:16
*** MarkAtwood has quit IRC19:16
mtreinishjeblair: sure, I was also thinking we should be able to start multiple workers on each node now19:16
*** esker has quit IRC19:16
jeblairmtreinish: yeah, i was just thinking the same thing19:16
*** cloudtrainme has quit IRC19:16
mtreinishwe had to stop doing that before because of the memory pressure. But the single threaded approach should avoid that concern19:16
jeblairthey have 8vcpus, so starting all 4 should be fine assuming we're ok on ram19:17
*** esker has joined #openstack-infra19:17
clarkband if they keep up maybe remove the second worker?19:17
*** dkehn_ has quit IRC19:17
*** dkehn has quit IRC19:17
mtreinishclarkb: yeah19:17
jeblairmtreinish, clarkb: the init script is not correct for non-A workers19:17
mtreinishjeblair: yeah, I need to push a system config patch to resync those19:18
*** mhickey has joined #openstack-infra19:18
mtreinishwe removed that chunk when we stopped using 4 workers19:18
mtreinishone sec I'll push the revert19:18
jeblairokay, i shut down B on 01 (which was briefly running with who-knows-what config).  and i have started A on 0219:18
jeblairA on 02 looks happy19:19
*** rcernin has quit IRC19:19
*** dkehn has joined #openstack-infra19:20
*** dkehn_ has joined #openstack-infra19:20
openstackgerritMerged openstack-infra/puppet-reviewday: Add /usr/local/bin to path for cron job
openstackgerritMerged openstack-infra/puppet-gerrit: Set gerrit login cache to 2m
openstackgerritMerged openstack-infra/elastic-recheck: Remove query about yum missing on ubuntu
openstackgerritMatthew Treinish proposed openstack-infra/system-config: Revert "Only start 1 subunit worker per node"
mtreinishjeblair, clarkb: ^^^ the revert19:20
mtreinishjeblair: the other thing I need to fix is to make all the db operations performed by the worker happen in a single transaction (so we can rollback on failures or crashes)19:23
mtreinishthat should be a pretty quick subunit2sql patch19:23
*** ybathia has quit IRC19:24
*** mrmartin has joined #openstack-infra19:26
openstackgerritmelanie witt proposed openstack-infra/project-config: Add a new devstack plugin based src job for ceph
*** mylu has quit IRC19:30
openstackgerritMerged openstack-infra/tripleo-ci: Make the delorean-ci repository available when build overcloud images
AJaegerinfra-root, I'll approve the voting of the beaker jobs for infra-puppet tomorrow if there's no -1 on
*** madhu_ak has joined #openstack-infra19:30
crinklethanks AJaeger for raising attention on it19:31
*** mylu has joined #openstack-infra19:31
fungicrinkle: AJaeger: seems fine to me. if we have any modules still failing those jobs which are really hard to just fix, we can roll it back to non-voting for them on a case by case basis19:34
fungiand we have a lot of +2's already. adding mine to the mix now19:35
fungialso approving19:35
*** Yarboa_Zzzzz has joined #openstack-infra19:36
*** Yarboa_Zzzzz is now known as Yarboa19:36
fungicrinkle: thanks for driving that effort home. i'm excited to see what it catches that our apply tests miss19:39
*** Guest3894 is now known as jgriffith19:40
AJaegerthanks, fungi!19:40
AJaegerand thanks, crinkle !19:40
*** maishsk has joined #openstack-infra19:42
openstackgerritMerged openstack-infra/project-config: Make Infra's puppet beaker jobs voting
EmilienMcrinkle: does infra have plans already for a common gem, like we have in puppet-openstack? same question for integration testing jobs19:42
*** baoli has joined #openstack-infra19:43
*** mylu has quit IRC19:44
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: TEST: Run df to see status of undercloud storage
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Revert "Turn off rax-ord"
clarkbI was just going to ask if you were ready for that19:45
*** ihrachys has quit IRC19:45
fungibombs away19:46
*** ihrachys has joined #openstack-infra19:46
crinkleEmilienM: we've brought it up, no one's done the work yet though19:46
clarkbfungi: any idea if our traceroutes were providing useful info?19:46
*** dkehn_ has quit IRC19:46
*** dkehn has quit IRC19:46
fungiclarkb: doubtful. i think our traceroutes probably just showed that packets were headed over the circuits between ord and dfw19:47
*** mylu has joined #openstack-infra19:47
fungiclarkb: but it was instrumental in getting the ticket past level 1 support to level 2 and then to the noc after some further explaining19:47
clarkbnibalizer: oh wow one of those shards finally finished, maybe this is just really inconsistently slow? /me holds out hope19:48
*** dkehn has joined #openstack-infra19:48
clarkbfungi: shibboleet19:48
*** dkehn_ has joined #openstack-infra19:49
*** mylu has quit IRC19:49
fungiafter years of talking my way through uunet/mci/worldcom, sprintnet, at&t, et cetera... rackspace's noc requires much less shibboleet19:49
*** doug-fis_ has joined #openstack-infra19:49
*** doug-fis_ has quit IRC19:50
*** doug-fis_ has joined #openstack-infra19:50
*** ihrachys has quit IRC19:50
*** ihrachys_ has joined #openstack-infra19:50
mtreinishsigh, now it looks like has diasapeared19:50
AJaegerfungi: +2A - do you want to enqueue and promote it so that it merges soonish?19:51
*** mylu has joined #openstack-infra19:51
fungiAJaeger: will do. looking at the backlog in check, it's not terrible, but this will help shorten it sooner19:51
mtreinishI'm guessing something screwed up in the puppet when building the js and it removed everything from the dir19:52
mtreinishis there anything in the logs?19:52
fungienqueue should be plenty sufficient. promoting only makes sense if there's a bunch of other project-config changes queued u pin the gate pipeline though19:52
AJaegerfungi, ah. Learned something new ;) In that case enqueue is indeed enough...19:53
fungiand it's in the gate now19:53
fungiyeah, promoting it would do absolutely nothing, it's already at the front of its particular queue19:53
clarkbon the ipv6 front I am told they are working on it, but haven't heard more details19:53
*** Goneri has quit IRC19:53
fungiclarkb: rax?19:53
*** doug-fish has quit IRC19:53
fungias in getting teh data into their configdrive?19:54
openstackgerritMerged openstack-dev/hacking: Updated from global requirements
*** doug-fis_ has quit IRC19:54
*** IlyaG has joined #openstack-infra19:54
*** ayoung has joined #openstack-infra19:55
*** mylu has quit IRC19:56
* AJaeger waves good bye19:56
*** tqtran is now known as tqtran-afk19:56
funginight AJaeger!19:56
*** mylu has joined #openstack-infra19:57
openstackgerritMerged openstack-infra/project-config: Revert "Turn off rax-ord"
*** MarkAtwood has joined #openstack-infra20:00
*** gyee has joined #openstack-infra20:00
openstackgerritMonty Taylor proposed openstack-infra/shade: Fail if FIP doens't have the requested port_id
mordredclarkb, Shrews: k. I think I covered both of your concerns20:02
*** matrohon has joined #openstack-infra20:02
mtreinishhmm, well it's back now20:03
*** mylu has quit IRC20:03
*** Nikolay_St has joined #openstack-infra20:04
*** maishsk has quit IRC20:06
fungii'm going to pop out for a short walk while that finds its way onto nodepool.o.o20:06
*** armax has joined #openstack-infra20:07
*** mylu has joined #openstack-infra20:07
*** bpokorny_ has quit IRC20:08
fungialso, looking at the test nodes graph, i have a feeling osic and bluebox have been doing very little for the past ~10 hours20:08
fungiother than complaining about fip quota20:08
fungihence the broad yellow "building" band there20:08
*** bpokorny has joined #openstack-infra20:09
*** sigmavirus24 is now known as sigmavirus24_awa20:09
*** Sukhdev has quit IRC20:09
*** baoli_ has joined #openstack-infra20:09
*** fawadkhaliq has quit IRC20:11
*** abregman has quit IRC20:12
*** baoli___ has joined #openstack-infra20:12
*** baoli has quit IRC20:12
*** apoorvad has quit IRC20:12
*** maishsk has joined #openstack-infra20:12
*** rcernin has joined #openstack-infra20:12
*** fawadkhaliq has joined #openstack-infra20:13
*** sigmavirus24_awa is now known as sigmavirus2420:13
*** esikachev has quit IRC20:13
*** baoli_ has quit IRC20:14
*** megm has quit IRC20:14
*** nwkarste_ has joined #openstack-infra20:14
*** mrmartin has quit IRC20:15
*** nwkarste_ has quit IRC20:15
*** nwkarste_ has joined #openstack-infra20:16
*** kgiusti has left #openstack-infra20:17
*** megm has joined #openstack-infra20:17
*** inc0 has joined #openstack-infra20:18
*** nwkarsten has quit IRC20:18
*** esker has quit IRC20:18
*** fawadkhaliq has quit IRC20:19
*** fawadkhaliq has joined #openstack-infra20:20
*** baoli___ has quit IRC20:20
*** inc0 has quit IRC20:21
*** baoli has joined #openstack-infra20:21
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/shade: Add support for Designate zones
*** maishsk has quit IRC20:23
*** mrmartin has joined #openstack-infra20:23
*** mrmartin has quit IRC20:23
openstackgerritElizabeth K. Joseph proposed openstack-infra/publications: Update slide deck for 2016 Newton Summit
*** e0ne has quit IRC20:24
pleia2pabelanger, docaedo ^^ screenshots I wanted to update have been updated20:24
*** fawadkhaliq has quit IRC20:24
pabelangerpleia2: ack20:24
*** e0ne has joined #openstack-infra20:24
*** fawadkhaliq has joined #openstack-infra20:25
pleia2pabelanger, docaedo: I didn't rework slide 3 because I have too much on my plate today, might do tomorrow or something, but didn't want to hold up any other proposed changes from either of you20:25
pleia2in another change20:26
pabelangerpleia2: wfm20:26
docaedopleia2: thanks for that!20:26
pabelangerI'll get my changes up for tomorrow20:26
*** maishsk has joined #openstack-infra20:27
*** jamielennox|away is now known as jamielennox20:29
*** mhickey has quit IRC20:30
*** armax has quit IRC20:30
*** doug-fish has joined #openstack-infra20:32
*** apoorvad has joined #openstack-infra20:32
*** thorst has quit IRC20:34
*** maishsk_ has joined #openstack-infra20:34
*** avarner_ has quit IRC20:35
*** maishsk has quit IRC20:35
*** maishsk_ is now known as maishsk20:35
*** ybathia has joined #openstack-infra20:36
*** rkukura has quit IRC20:36
*** thorst has joined #openstack-infra20:37
*** avarner has joined #openstack-infra20:37
*** piet has joined #openstack-infra20:38
openstackgerritIhar Hrachyshka proposed openstack-infra/project-config: grafana: added periodic oslo master jobs to unit test 'gate' dashboard
*** BigWillie has quit IRC20:38
*** trown is now known as trown|outtypewww20:39
*** claudiub has quit IRC20:40
Shrewsyolanda: mordred: since magnum support was added, shade jobs seem to all be failing in the same peculiar way. coincidence?20:40
*** thorst has quit IRC20:41
*** avarner has quit IRC20:42
mordredShrews: how about we revert that?20:43
openstackgerritMonty Taylor proposed openstack-infra/project-config: Revert "Enable magnum plugin on shade and use our atomic generated images"
Shrewsmordred: seems logical, though i have no proof of it other than the christmas tree of jenkins reports20:44
mordredinfra-root: ^^ insta-approve real quick?20:44
mordredShrews: yah. I think it's probably something that will be easy enough to diagnose ... but I need to finish fixing nodepool FIPs right now20:44
*** avarner has joined #openstack-infra20:44
mordredwe can debug magnum next week or the week after20:44
mordredclarkb: thanks20:45
* clarkb wants workin nodepool ;)20:45
*** _ari_ has quit IRC20:45
mordredtoday really seems to be magnum day for mordred20:45
mordredbetween this patch and the mailing list thread I'm somehow on20:46
* mordred is confused20:46
*** inc0 has joined #openstack-infra20:46
clarkbmordred: I want an api that makes rainbows. curl -X GET and bam rainbow outside20:47
clarkber s/&/?/20:47
mordredclarkb: NO. BAD ENGINEER NO COOKIE20:47
*** ihrachys_ has quit IRC20:48
mordredpleia2: wow. that slide had marconi on it!20:48
pleia2mordred: hehe, yeah20:48
pleia2mordred: and I was so thrilled to delete the list of clients20:48
pleia2long live OSC20:48
*** ihrachys has joined #openstack-infra20:48
*** krotscheck has quit IRC20:48
*** stewie925 has joined #openstack-infra20:48
*** krotscheck has joined #openstack-infra20:48
pleia2mordred: thinking of reworking that slide to not be a list anymore, and something blah blah big tent, but my brain today /o\20:49
mordredpleia2: oh, I hear that20:49
*** avarner has quit IRC20:50
mordredpleia2: I think you should actually just make the slide say "blah blah big tent" :)20:50
clarkb use that picture20:50
pleia2clarkb: hahaha <320:50
*** matrohon has quit IRC20:50
*** avarner has joined #openstack-infra20:50
openstackgerritThanh Ha proposed openstack-infra/jenkins-job-builder: Fix CFP configuration for >=2.10.0
mordredclarkb: ++20:52
*** dkehn__ has joined #openstack-infra20:52
zxiirozaro: mind reviewing my patch ^20:53
*** matrohon has joined #openstack-infra20:53
mtreinishclarkb: I normally use this:
*** fawadk has joined #openstack-infra20:54
*** fawadkhaliq has quit IRC20:54
*** thorst has joined #openstack-infra20:54
*** thorst has quit IRC20:54
*** Yarboa has quit IRC20:54
*** aeng has joined #openstack-infra20:55
*** thorst has joined #openstack-infra20:55
*** avarner has quit IRC20:55
*** dkehn has quit IRC20:55
*** flwang has quit IRC20:56
*** ams_ has joined #openstack-infra20:56
*** maishsk has quit IRC20:56
nibalizerclarkb: cool20:56
*** thorst has quit IRC20:56
*** thorst has joined #openstack-infra20:57
nibalizerclarkb: looks like only one shard initializng now20:57
clarkbnibalizer: ya I think what it is doing is when one host fails to restore it it moves on ot another host and that is happening slowly. Then eventually it gets to the one with the lock maybe and it works20:57
clarkbthis doesn't fix the deadlock but it does work with it?20:57
rcarrillocruzi'll retry my patchsets tomorrow when magnum revert settles if that's the issue20:59
rcarrillocruzi call it a day20:59
* rcarrillocruz waves20:59
pabelangerclarkb: looks like ceph jewel will have xenial packages:
*** fawadk has quit IRC20:59
*** maishsk has joined #openstack-infra21:00
pvanecknibalizer: hey, can you take a look at ?21:01
clarkbpabelanger: if nova/ceph are ok with moving to jewel then that seems like a reasonable migration path (a little nanoying I set up the hammer repo for 2 weeks but whatever learned how to do it)21:01
odyssey4mehmm, it would appear that zuul queue lengths and results are climbing, and only climbing21:02
clarkbodyssey4me: last I checked they were falling pretty rapidly21:02
clarkbya gearman job graph seems to ocnfirm21:03
pabelangerclarkb: will ask around next week21:03
odyssey4methey're growing slowly right now - and I have several successful post jobs completeed, but still displaying21:03
odyssey4mealso, +w was given to a patch several minutes ago and it's not in the gate queue yet21:03
clarkbodyssey4me: post is the lowest priority, you'll see that check is quickly falling and as we get our heads above water on the backlog post will pick up21:03
clarkbodyssey4me: that is because zuul has ~2k events and results to process21:04
odyssey4meyeah, it figures - lemme be a little more patient21:04
odyssey4meah, there it shows21:04
clarkbbasically things are looking much better now than it has been for the last few days21:04
* mordred hands odyssey4me a beer21:04
*** jheroux has quit IRC21:04
openstackgerritMerged openstack-infra/publications: Update slide deck for 2016 Newton Summit
pabelangerOh, nice. rax-ord is back21:04
mordredclarkb: we have so much clouds back!21:04
odyssey4menote that a lot of IRC messages have been dropping lately21:04
*** sdague has quit IRC21:04
odyssey4mebasically review submissions or merge events just haven't shown all the time - only around 10% of the time21:05
* odyssey4me thanks mordred :)21:05
clarkbodyssey4me: freenode has been under attack aiui21:05
odyssey4meoh nice, that's fun21:05
clarkbwhich is causing netsplits and servers to go offline which impacts the bots21:05
odyssey4meapparently some people have too much time on their hands :p21:05
pabelangerYa, post pipeline is getting crushed right now21:06
*** tristanC_ is now known as tristanC21:07
mordredclarkb: can I restart nodepool again?21:07
pabelangerlast 1 hour / refresh every 5s21:07
*** e0ne has quit IRC21:07
*** fawadkhaliq has joined #openstack-infra21:07
clarkbmordred: if you do all of the image uploads that are happening will be killed, that is the tradeoff21:08
*** fawadkhaliq has quit IRC21:08
mordredclarkb: nevermind then. I will wait21:08
*** edmondsw has quit IRC21:08
odyssey4mewowzer, yeah I saw the list of my stuff disappear21:08
clarkbmordred: it could be a while, uploads are slow21:08
*** e0ne has joined #openstack-infra21:08
*** fawadkhaliq has joined #openstack-infra21:08
*** fawadkhaliq has quit IRC21:08
odyssey4meI was trying to do some stuff in DFW earlier and the links were damned slow21:08
odyssey4methey picked up after 30 mins or so21:08
*** flepied1 is now known as flepied21:09
clarkbmordred: might be better to try and get nodepool reliable now then let tomorrow's uploads happen21:10
openstackgerritYe Na Rhee proposed openstack-infra/grafyaml: Add datasource option for graph and singlestat panel
mordredclarkb: yah. I wanna put the fail-early-with-no-port-id change in place21:10
openstackgerritMatthew Thode proposed openstack/diskimage-builder: Add Gentoo to the dhcp-all-interfaces element
*** rfolco has quit IRC21:11
*** matt-borland has quit IRC21:12
openstackgerritMerged openstack-infra/project-config: Revert "Enable magnum plugin on shade and use our atomic generated images"
*** Sukhdev has joined #openstack-infra21:15
*** rhallisey has quit IRC21:16
*** maishsk has quit IRC21:17
*** flwang has joined #openstack-infra21:19
ianwpabelanger: so in terms of removing devstack-centos7, people have worked around the lack of loclae tools and selinux issues?  because fixes for those change are still outstanding21:19
openstackgerritMerged openstack-infra/puppet-openstack_health: Add packages needed to build lxml
*** lennyb has joined #openstack-infra21:22
krotschecknode on xenial makes me happy21:23
*** sridhar_ram1 is now known as sridhar_ram21:23
*** gordc has quit IRC21:25
*** vhosakot has quit IRC21:26
*** lakshmiS has quit IRC21:26
*** vhosakot has joined #openstack-infra21:26
openstackgerritMerged openstack-infra/system-config: Add database backups for
*** nikhil_k has joined #openstack-infra21:27
*** jtomasek has quit IRC21:27
*** HeOS has joined #openstack-infra21:28
openstackgerritFei Long Wang proposed openstack-infra/project-config: Support multi backends for Zaqar tempest job
openstackgerritMatthew Thode proposed openstack/diskimage-builder: Add Gentoo to the dhcp-all-interfaces element
pabelangerianw: and we have21:32
pabelangerianw: so, once we land those changes, we can remove all the work arounds21:32
*** nwkarsten has joined #openstack-infra21:32
pvaneckthanks nibalizer ^^21:33
*** Na3iL has quit IRC21:33
*** matrohon has quit IRC21:33
nibalizerpvaneck: for sure21:33
pabelangermordred: did you restart nodepool?21:35
mordredpabelanger: yes I did21:36
pabelangermordred: ack21:36
*** nwkarste_ has quit IRC21:36
*** nwkarsten has quit IRC21:36
pabelangerYa, post is down to 25 now.  nodepool crushing it21:38
*** piet has quit IRC21:38
*** Na3iL has joined #openstack-infra21:39
*** rcernin has quit IRC21:39
pabelanger looks over all happy now. We are using 1/2 our capacity right now21:40
pabelangerwith pipelines dropping21:40
prometheanfirebnemec: brainfart :D21:40
openstackgerritMatthew Thode proposed openstack/diskimage-builder: Add Gentoo to the dhcp-all-interfaces element
*** nikhil has quit IRC21:40
*** nikhil_k is now known as nikhil21:40
bnemecprometheanfire: We've all been there. :-)21:41
*** IlyaG has quit IRC21:43
EmilienMfungi: ok we're almost there:
EmilienMfungi: we have puppet-aodh-stable-liberty.tar.gz instead of puppet-aodh-7.0.0.tar.gz21:45
*** |-paul-| has joined #openstack-infra21:45
EmilienMI guess I did something wrong in the script. /me looking21:45
*** aimeeU has quit IRC21:46
openstackgerritMerged openstack/diskimage-builder: Updated from global requirements
EmilienMfungi: well, in fact that's cool, other projects also have this model21:48
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add remaining pipelines to zuul-status dashboard
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add remaining pipelines to zuul-status dashboard
*** fawadkhaliq has joined #openstack-infra21:52
flwangclarkb: hi could you pls help review this ? need some suggestion to fix the jenkins failure21:55
*** gordc has joined #openstack-infra21:55
fungiEmilienM: i triggered the branch tarball script (the one that runs each time you merge a change to the branch tip). if you want tarballs built from tags instead, those are release pipeline jobs that need retriggering21:56
*** mriedem has quit IRC21:56
EmilienMfungi: I don't think we have the jobs in place yet21:57
*** mylu has quit IRC21:57
EmilienMfungi: afik, I only added the post job for branch-tarball21:57
*** flepied has quit IRC21:58
fungiEmilienM: ahh, yep21:58
fungiyou are correct21:58
EmilienMfungi: step by step21:58
fungiso anyway, that seems to work. i can trigger the ones for stable/mitaka and master21:59
EmilienMfungi: so, what do you suggest if we want tarballs for all projects. Should I write a script that find the latest SHA1?21:59
fungifor branch tarballs, latest sha on each branch yes21:59
fungifor tag tarballs, sha of the tags and the tag names are what we need22:00
EmilienMfungi: we'll see tags later. Let's do branch first22:00
fungifor the tag sha, it's teh result of `git show-ref 7.0.0` or whatever22:00
EmilienMoh well, I can also provide the tag.22:00
fungifor branches, you can `git show-ref stable/liberty`22:00
fungithe tags will only do us any good once you have release tarball jobs though22:00
*** mylu has joined #openstack-infra22:01
fungii've triggered a work request for stable/mitaka next22:02
fungiclarkb: oh, too bad, zuul caught up on its waiting queue just after rax-ord went back into service, so we didn't really get a chance to peg it at all22:03
*** Sukhdev has quit IRC22:03
clarkbfungi: just slightly22:03
pabelangerclarkb: ianw: if we want to land 308105 tonight, I can kick off centos-7 dibs and monitro22:03
clarkbflwang: you have updated your job template but not instantiated the new jobs with that template so the test fails22:03
clarkbflwang: you need totell jjb to make jobs using that template with mongodb and redis22:03
* clarkb mumbles something about redis being a terrible message queue backend22:03
*** HeOS_ has joined #openstack-infra22:04
*** Sukhdev has joined #openstack-infra22:04
flwangclarkb: that's an interesting point, why do you think redis is a terrible backend for mq?22:04
*** netsin has joined #openstack-infra22:04
clarkbflwang: because I tried to use it for logstash's queiing and I broke it in 5 minutes22:05
clarkbflwang: it doesn't scale up well22:05
flwangclarkb: oh, really? did you use redis cluster?22:05
*** armax has joined #openstack-infra22:06
clarkbno just a single instance22:06
clarkbI was using their prescribed system architecture22:06
fungithis was a couple years ago, right?22:06
clarkbfungi: ya22:06
fungior more recent?22:06
clarkbbut redis hasn't really gotten better22:07
clarkbit uses all your memory, forks, OOMs then dies22:07
fungimaybe they've fixed redis to be "web scale" now22:07
mtreinishhmm, puppet hasn't run on health.o.o in a while:
flwangclarkb: it uses all your memory  --- since it's a memory DB :)22:07
clarkbflwang: ya but then it forks22:07
clarkbflwang: the issue is with the forking22:07
mtreinishI'm still waiting on the next build which I hope will be able to work (and add the rss support)22:08
clarkbbecause that doesn't work when you have no memory22:08
flwangclarkb: ok, not sure if the issue is still there, but for zaqar, you have mongodb as well :)22:08
fungiEmilienM: now you have a stable-mitaka tarball too, and you already had a master tarball from a change that merged to that repo earlier today22:08
pabelangermtreinish: last errors I see on health.o.o:
*** Sukhdev has quit IRC22:09
*** burgerk has quit IRC22:10
EmilienMfungi: awesome22:10
EmilienMfungi: I'm doing the script now22:10
pabelangermtreinish: puppet just finished running, so you should see an update22:10
mtreinishpabelanger: ok, cool. Yeah that last error ran before (well right around) the patch to puppet-subunit2sql merged22:11
*** flepied has joined #openstack-infra22:12
*** cody-somerville has quit IRC22:12
fungiEmilienM: do you need master too, or are they moving fast enough not to bother?22:12
pabelangermtreinish: facts should be posting now22:12
EmilienMfungi: are you scripting it?22:12
EmilienMfungi: if yes, I would ask master22:13
EmilienMif not, we can live without22:13
fungiEmilienM: you said you were writing the script, i just noticed you didn't have any master branches listed in that paste22:13
pabelangermtreinish: and done22:14
EmilienMfungi: sorry misunderstanding. I'm writting the script22:14
clarkbpabelanger: did dib release with the change we need?22:14
EmilienMfungi: for master, should also provide the latest commit?22:14
fungiEmilienM: yes, if you want those22:14
*** tqtran-afk is now known as tqtran22:14
*** doug-fish has quit IRC22:14
pabelangerclarkb: Ya, looks like ianw landed it. Not sure about release status22:14
fungimaster is treated just like any other branch22:14
clarkbpabelanger: we might be able to just merge both changes and get it through22:14
*** Sukhdev has joined #openstack-infra22:15
clarkbpabelanger: it hasn't so nevermind I will just approve my change22:15
clarkber the first one22:15
pabelangerclarkb: okay22:15
clarkbpabelanger: actualyl there may be a small issue with it gonnatest first22:15
fungiclarkb: bad news, we have some new hits on ord now that we've cranked it back up. hit two jobs about an hour ago22:15
pabelangerclarkb: ack22:16
*** armax has quit IRC22:16
pabelangerianw: is the plan to release diskimage-builder today?22:16
flwangclarkb: "flwang: you need totell jjb to make jobs using that template with mongodb and redis"  would you mind pointing out a link where i should fix?22:16
mtreinishpabelanger: cool, yep I see it now22:16
clarkbpabelanger: I am worried about -x "" but quick local tests says it works22:16
*** fawadkhaliq has quit IRC22:16
ianwpabelanger: yep i can, especially if a few of us are around to monitor.  be nice to get prometheanfire's 299694 change in22:16
clarkbpabelanger: so I think it is alright to merge22:17
*** asettle has joined #openstack-infra22:17
mtreinishpabelanger: hmm, although I'm not seeing the new feature (that the xml libs were need for though)22:18
fungiclarkb: ouch, these jobs were also getting failures connecting locally to our mirror server in ord22:19
clarkbmordred: did you restart nodepool with the port thing?22:19
flwangclarkb: lovely, i owe you a beer22:19
clarkbfungi: :(22:19
pabelangerclarkb: WFM22:19
fungiclarkb: so at least in these two cases, it was networking broken in general for the workers, not just reaching to dfw22:19
pabelangerianw: great22:19
*** baoli has quit IRC22:20
clarkbfungi: and it should be a problem wiht the private IP routes because mirrors are all public IP22:20
*** inc0 has quit IRC22:21
fungiunfortunately it's hard to tell what the exact network issues were because apt-get silently fell back to trying ipv6 and then complained about network unreachable once that didn't work22:21
fungialso these weren't jobs with our net-info macro, so no traceroutes22:21
krotscheckjeblair: Is that vos-release still going?22:22
fungiEmilienM: almost. the first two character of the logpath are supposed to be the first two characters of the git sha22:23
fungiEmilienM: so it's right for puppet-aodh liberty there (which was the example you were working from presumably)22:23
EmilienMfungi: we are saving 1408 CI jobs :)22:23
openstackgerritMerged openstack-infra/project-config: Use setfiles to restore unbound selinux policy
fungiEmilienM: indeed. quite the savings!22:24
clarkbpabelanger: ^ there it is22:24
openstackgerritFei Long Wang proposed openstack-infra/project-config: Support multi backends for Zaqar tempest job
pabelangerclarkb: okay, let me start centos-7 build22:24
clarkbpabelanger: you may need to wait for a puppet run? I can't rmember how those elements get updated22:25
*** fawadkhaliq has joined #openstack-infra22:25
EmilienMfungi: the 2 caracteres of the logpath are for post queue, isn't?22:25
EmilienMfungi: I checked22:25
pabelangerclarkb: Yup, going to confirm on host first22:25
*** netsin has quit IRC22:25
EmilienM is for post logs, iiuc22:26
fungiEmilienM: you have 60/607957dd427cb7369e5a8af8a3544a5340556f0e for the first line, so that's right. the second line you have 60/ec4473ff0bed4a0e2bda381f9af079239b94ad0c when it should be ec/ec4473ff0bed4a0e2bda381f9af079239b94ad0c22:26
pabelangerclarkb: do you use image-build or 2 step dib-image-build / image-upload22:27
*** cody-somerville has joined #openstack-infra22:27
fungiEmilienM: we use an extra directory level sharded on the first two characters of the sha to avoid having too many subdirectories in a single directory22:27
EmilienMfungi: how can I kow the caracters?22:27
clarkbpabelanger: I use two step22:27
EmilienMfungi: let me fix that22:27
pabelangerclarkb: ya, do the same.22:27
*** fawadkhaliq has quit IRC22:28
*** bpokorny_ has joined #openstack-infra22:28
pabelangerclarkb: okay, patch landed. Starting builds22:28
jeblairReleased volume mirror.npm successfully22:29
jeblairreal    4847m44.626s22:29
jeblairkrotscheck: ^22:29
mordredclarkb: yes22:29
clarkbpabelanger: I would upload to osic first because it is fastest and can confirm if this fixes dns22:29
jeblairkrotscheck: shall i kick off another?22:29
fungithat's a lot of minutes22:29
jeblairyeah, that was the bulk of the npm mirror22:29
*** Adri2000 has quit IRC22:29
mordredclarkb: we're still leaking, but not as fast as before22:29
*** fawadkhaliq has joined #openstack-infra22:30
mordredclarkb: I am not running with the extra shade debug patch, becuase I need to rewrite it now22:30
clarkbmordred: is it still worse than before we switched to shade?22:30
pabelangerclarkb: ack22:30
clarkbpabelanger: cloud sthat use qcow2 win the upload game22:30
*** vhosakot has quit IRC22:30
jeblairalso just win22:30
pabelangerclarkb: okay, didn't know that22:30
EmilienMI'm going to save this script22:30
krotscheckjeblair: yeah, go ahead. you can also kill the mirror update script (but don't release the lock).22:31
mordredclarkb: yes, I think so. but the rate is slow enough that the reaper should take care of it and allow further debugging22:31
clarkbmordred: gotcha22:31
jeblairkrotscheck: is there a reason to kill it?22:31
krotscheckjeblair: not really. It'll keep chugging.22:31
krotscheckjeblair: Once lands we can kill it.22:31
jeblairkrotscheck: that sounds like a plan then22:31
*** bpokorny has quit IRC22:31
jeblairkeep the delta small till then22:31
krotscheckSee y'all tomorrow.22:32
*** krotscheck is now known as krotscheck_dcm22:32
*** netsin has joined #openstack-infra22:32
*** ihrachys has quit IRC22:32
EmilienMfungi: when you tell me it's ok, I leave to go running22:33
*** stewie925 has quit IRC22:33
fungiEmilienM: lgtm, it's running (all except puppet-aodh which i left out since those were already run), so you can run too22:34
*** ihrachys has joined #openstack-infra22:34
fungiEmilienM: i don't get feedback on whether these complete successfully, so you'll want to double-check that all the expected tarballs show up later22:35
EmilienMfungi: thanks a ton for your help22:35
*** sigmavirus24 is now known as sigmavirus24_awa22:35
fungimy pleasure22:35
EmilienMI can go running now :)22:35
fungihave fun!22:36
EmilienMthanks :)22:36
mtreinishclarkb, fungi, pabelanger: if you get a sec can you check the version of openstack-health installed inside the venv on health.o.o? I still don't think it's updating22:39
mtreinish should be an rss feed if it's got the latest commit (I dunno where the empty json is coming from)22:40
*** fawadkhaliq has quit IRC22:40
fungimtreinish: i can't remember, is it installed system-wide or in a virtualenv?22:40
mtreinishfungi: it's in an venv22:40
fungi'splains why i didn't see it22:41
*** sdague has joined #openstack-infra22:41
*** gordc has quit IRC22:41
mtreinishit's been on the list to just system install it22:41
pabelangermtreinish: what likely happen, is the install didn't retrigger after the missing depends were added22:42
mtreinishpabelanger: ah, ok. So if we waited for another commit to merge that would take care of it22:42
fungimtreinish: openstack-health (0.0.1.dev576, /opt/openstack-health)22:42
pabelangermtreinish: Right22:43
mtreinishfungi: heh, I guess I need to count commits :)22:43
*** |-paul-| has quit IRC22:43
fungimtreinish: openstack-health==0.0.1.dev576  # git sha 8093b5b22:44
fungiforgot i wanted pbr freeze not pip list22:44
nibalizerthis, uh, doesn't really look right to me
clarkbnibalizer: ya their docs are a mess right now, the last attempt to fix them ran into other issues iirc22:45
fungimtreinish: also that sha doesn't match what's checked out from git22:45
nibalizerit used to be so gorgeous22:45
mtreinishfungi: ok, well I just approved a couple of patches on my review list so hopefully after they land and puppet reruns everything will jump forward22:45
fungimtreinish: seems to be four gerrit changes and a few merge commits behind master22:45
mtreinishfungi: yeah it probably stopped updating after the rss patch landed22:46
fungimtreinish: yeah, looks like puppet has been checking out the latest ref in /opt/openstack-health but not successfully installing/upgrading22:46
mtreinishand the other commits came in before the xml libs were added22:46
*** mylu has quit IRC22:46
fungimtreinish: probably because of missing the lxml build deps, yes22:46
*** ihrachys has quit IRC22:47
fungii see those dep packages are installed now so ought to work22:47
mtreinishwell hopefully22:47
fungiunless it's gotten wedged somehow from earlier failed install attempts, but if so i can blow away the venv and recreate22:48
* mtreinish wants to blame fungi for suggesting rss in the first place :)22:48
fungihey, i _also_ said "atom"22:48
jeblairi say 'finger'22:48
fungiit's a world-class protocol22:49
*** ihrachys has joined #openstack-infra22:49
mtreinishfungi: heh, I just need to change: to call atom_str() instead to use atom22:50
fungithat reminded me to check that finger still works. it does!22:50
*** eil397 has joined #openstack-infra22:52
fungiwow, that takes me back22:52
openstackgerritMerged openstack-infra/project-config: Add a new devstack plugin based src job for ceph
mtreinishfungi: hmm, have you seen this before:
*** ihrachys has quit IRC22:57
*** mylu has joined #openstack-infra22:57
fungimtreinish: not in this situation, no22:58
clarkbfungi: I installed finger just for that :)22:59
pabelangerha ha, same22:59
*** xyang1 has quit IRC22:59
mtreinishas did I :)22:59
fungiclarkb: what crazy unix are you on that doesn't have finger by default?22:59
fungijust kidding, a lot of distros have dropped it from their default package set these days22:59
*** Sukhdev has quit IRC22:59
mtreinishfungi: arch doesn't even keep it in the regular repos I had to build it from the AUR23:00
fungiclarkb: did you see that job error mtreinish got? looks like it thought sudo wasn't runnable by jenkins but i don't see it being revoked either23:01
*** mylu has quit IRC23:01
clarkbmtreinish: at that point you might as well just write your own finger client in python23:01
clarkbfungi: I hand't looked, but I can now23:02
fungifinger is basically a tcp raw socket after all, with like a couple of commands it can accept23:02
mtreinishclarkb: heh, nah it just means I've got a -march native -O2 built version of finger on my desktop23:02
mtreinishclarkb: although I'm sure someone has already written a finger client in python23:03
openstackgerritMerged openstack/diskimage-builder: Add Gentoo to the dhcp-all-interfaces element
clarkbmtreinish: I am mostly a fan of not aur'ing23:03
clarkbso much badness23:03
*** Sukhdev has joined #openstack-infra23:04
clarkbfungi: ya it looks like it wants to ask jenkins for a password23:05
mtreinishclarkb: sure, I can understand that. I don't mind it as much. I also maintain a couple packages there:
fungii love that finger script. it's two dozen lines of stdlib pure python, if you ignore the comments23:05
mtreinishfungi: yeah it's pretty awesome23:06
openstackgerritMerged openstack-infra/jenkins-job-builder: Preserve data type when constructing new container
*** HeOS has quit IRC23:06
fungia good job of demonstrating how minimal the protocol is23:06
*** rbrndt has quit IRC23:07
fungithe second sentence of rfc 742 starts out "this is a simple protocol"23:08
openstackgerritMerged openstack-infra/jenkins-job-builder: Use provided mapping object type instead OrderedDict
fungialso telling is that it's a 7-page rfc which is mostly examples23:09
pabelangerfungi: clarkb: that looks like an OLD version of install-distro-packages.sh23:09
pabelangerwe do sudo apt-get -qq update now23:09
fungithe protocol description itself barely extends onto page 323:09
fungipabelanger: so very old ubuntu-trusty image in rax-iad?23:10
pabelangerfungi: maybe23:10
pabelangerfungi: I can check23:10
clarkbwouldn't surprise me, mordreds hacking has essentially put a standstill to image updates and rax uploads are the least reliable23:10
pabelangerclarkb: centos-7 finished. starting upload23:10
mtreinishfungi: the other thing I just noticed about that finger script is it was posted 15 yrs ago23:12
fungino wonder it only uses stdlib23:12
clarkbdid it run on python 1.x?23:12
pabelangerfungi: I see 1 image from 4 days ago23:13
pabelangerother is only 22 hours old23:13
clarkbpabelanger: fungi oh wait is that part of the job or the image?23:13
openstackgerritMerged openstack-infra/jenkins-job-builder: Allow id's for different templates to follow same naming
fungiclarkb: it could have been cutting edge and only supported python 2.1 15 years ago23:14
mtreinishclarkb: it must have, python 2.0 wasn't released when it was posted23:14
pabelangerclarkb: the script is in the image23:14
pabelangerso, possible it booted the old image?23:14
fungiclarkb: slave script install-distro-packages.sh23:14
pabelangernot sure how23:14
mtreinisherr, or I don't think it was (just noticed all the dates on the python archive are the same across multiple releases)23:14
fungipabelanger: how old would that have to have been?23:14
openstackgerritMerged openstack-infra/jenkins-job-builder: Wrap stdout object only once with wrap_stream
pabelangerfungi: apparently 26 days go we merged our silence distro package23:15
pabelangerso something is going on here23:15
*** sputnik13 has quit IRC23:16
mtreinishclarkb: err it might have, python 2.0 was released about 6 months before the post23:16
fungisunday was 15-year anniversary of the python 2.1 release23:17
*** Adri2000 has joined #openstack-infra23:18
jeblair1.5.2 forever23:18
fungipabelanger: i've confirmed the gate-{name}-npm-run-{command} job-template is definitely using our install-distro-packages macro and no revoke-sudo until later in the job23:18
openstackgerritFei Long Wang proposed openstack-infra/project-config: Support multi backends for Zaqar tempest job
*** bpokorny_ has quit IRC23:18
pabelangerclarkb: upload finished to osic23:19
pabelangergoing to check the node before continuing23:19
*** cody-somerville has quit IRC23:19
fungiclarkb: i think this may be a fresh example of that bug i said last week we could close out23:20
mtreinishclarkb: I built python 1.6 and the finger script does not work. It doesn't understand +=23:20
clarkbfungi: fun23:20
*** markvoelker has quit IRC23:20
fungiwhere it looks like nova boots an instance from an image which was supposed to have been deleted ages ago23:20
clarkbmtreinish: wow that is dedication23:20
fungiinstead of the image we request23:20
fungilike indices get cross-wired in some nova host's local image cache23:21
pabelangerfungi: Ya, not sure23:21
fungiwhich would imply this isn't snapshot-related, because we never did snapshot-based for ubuntu-trusty23:21
mtreinishfwiw, the script doesn't do everything:
*** dizquierdo has joined #openstack-infra23:23
fungimtreinish: anyway, recheck if you haven't already. this is a very strange behavior we see very infrequently and only ever in rackspace, which i'm almost certain is a problem in their systems23:23
mtreinishfungi: yep, already pushed the recheck :)23:24
openstackgerritIan Wienand proposed openstack/diskimage-builder: dhcp-all-interfaces depends on dhcp
openstackgerritIan Wienand proposed openstack/diskimage-builder: dhcp-all-interfaces depends on dib-init-system
pabelangerclarkb: wow, is slow to login23:25
clarkbpabelanger: is it? I don't really use the gui except for making networks23:25
*** dimtruck is now known as zz_dimtruck23:26
pabelangerclarkb: finally, loaded.23:26
pabelangerya, I should use openstack client more23:26
pabelangerneed to brush up on shade syntax23:26
fungishouldn't need shade syntax to use osc cli23:27
openstackgerritIan Wienand proposed openstack/diskimage-builder: Use generic "dhcp-client" name
*** EricGonczer_ has quit IRC23:27
pabelangerfungi: sorry, export syntax for os-client-config23:28
fungiand the grammer is nearly identical to the individual python clients23:28
*** lakshmiS has joined #openstack-infra23:29
*** salv-orl_ has joined #openstack-infra23:29
pabelangerclarkb: -rw-r--r--. root root    system_u:object_r:named_conf_t:s0 forwarding.conf23:30
pabelangerlooks good23:30
pabelangerunbound is running too23:31
clarkbyay, pabelanger run dig git.openstack.org23:31
clarkband check the /etc/resolv.conf but that looks promising23:31
zxiirowaynr: Saw you reviewing some patches. You mind reviewing this one too? I want it merged before we cut a release
*** bpokorny has joined #openstack-infra23:31
pabelangerclarkb: if you want to poke around23:32
*** salv-orlando has quit IRC23:32
clarkbpabelanger: ya that lgtm23:33
*** vhosakot has joined #openstack-infra23:33
*** cody-somerville has joined #openstack-infra23:34
*** mriedem has joined #openstack-infra23:36
openstackgerritMerged openstack-infra/irc-meetings: Move OSC team meeting even weeks to 1300 UTC Thursday
pabelangerclarkb: starting uploads to other clouds23:39
*** asettle has quit IRC23:39
*** zz_dimtruck is now known as dimtruck23:40
*** asettle has joined #openstack-infra23:40
*** Swami has quit IRC23:42
*** Sukhdev has quit IRC23:42
flwangclarkb: i have fixed the previous issue with your help, but the new error is also out of my knowledge scope
*** jrmyjohn has quit IRC23:45
*** mylu has joined #openstack-infra23:46
*** dizquierdo has quit IRC23:46
pabelangerif only regex worked for image-upload --provider23:47
*** e0ne has quit IRC23:47
mtreinishfungi, pabelanger: it doesn't look like puppet picked up the latest patches: is the first run after additional changes landed23:48
*** Goneri has joined #openstack-infra23:49
pabelangermtreinish: fungi: new errors... 1 src23:50
pabelangermtreinish: fungi:
mtreinishpabelanger: hmm, still missing a dep?23:51
*** mylu has quit IRC23:52
pabelangermtreinish: looks like you are missing: zlib1g-dev23:53
mordredclarkb: 2 leaked fips on osic since 1.5 hours ago23:53
*** Guest82423 has quit IRC23:54
openstackgerritJoshua Harlow proposed openstack/requirements: Bump uc for oslo weekly release for apr 18th
*** eil397 has left #openstack-infra23:55
mtreinishpabelanger: sigh, ok I'll push another patch to add that package23:57
*** ajmiller_ has quit IRC23:57
flwangclarkb: never mind, fixed :)23:57
openstackgerritFei Long Wang proposed openstack-infra/project-config: Support multi backends for Zaqar tempest job
*** mylu has joined #openstack-infra23:58
*** SumitNaiksatam has quit IRC23:58

Generated by 2.14.0 by Marius Gedminas - find it at!