Thursday, 2016-04-07

*** zz_dimtruck is now known as dimtruck00:00
*** xarses_ has quit IRC00:02
pleia2Kiall: btw, not sure if someone actively did anything, but I had looked at the designate-tempest-plugin merge about an hour before you asked for wznoinsk and it was still in the post queue, it's up on git.o.o now00:02
*** thorst has quit IRC00:02
*** Goneri has quit IRC00:02
*** akshai has joined #openstack-infra00:03
clarkbpleia2: ya ansible stopped running whihc kept puppet form updating the things00:03
clarkbpabelanger sorted that out while I nodepoold00:03
pleia2clarkb: ah, gotcha00:03
pleia2thanks pabelanger \o/00:03
openstackgerritMerged openstack-infra/project-config: Add non-voting pylint job to os-brick experimental
pabelangerYup, things should be back to normal for puppet now00:05
greghaynesclarkb: and uploads are going fine?00:05
pabelangerclarkb: do we have any docs to enqueue a periodic job in zuul?00:06
*** arxcruz has quit IRC00:08
clarkbgreghaynes: mostly, occasionally have to restart ovh uploads00:09
clarkbpabelanger: uh zuul enqueue?00:09
clarkbpabelanger: you can also likely just trigger it in jenkins directly00:10
*** akshai_ has joined #openstack-infra00:10
clarkbgreghaynes: I am saving rax for last since those are fun00:10
pabelangerclarkb: tried to enqueue but didn't take. Wonder if I need to use a --ref00:11
clarkbperiodic isn't ref'd though00:11
clarkbdid you set the trigger to timer?00:11
pabelangerclarkb: seems the job requires zuul params, so web UI won't work00:11
clarkbinstead of gerrit?00:11
pabelangerYa, I used gerrit00:11
clarkbI think it may need to be timer00:11
pabelangerthere we go00:11
*** mriedem has left #openstack-infra00:12
*** akshai has quit IRC00:13
*** dimtruck is now known as zz_dimtruck00:13
openstackgerritJoshua Hesketh proposed openstack-infra/system-config: Add election alias to
*** tqtran has quit IRC00:15
pabelangerianw: okay, not sure I did it right. But will wait and see00:15
*** sflanigan has quit IRC00:17
*** sputnik13 has quit IRC00:18
ianwjhesketh / pabelanger : so everything seems to have built correctly, so presumably the newly uploaded devstack-centos7 images will unblock puppet CI00:18
*** sputnik13 has joined #openstack-infra00:19
clarkbianw: note they probably won't be put into use until after the release at 1400UTC today (UTC)00:19
*** mriedem has joined #openstack-infra00:19
pabelangerLooks like kolla won't be tagging mitaka until the 15th now... so devstack-centos7 might live a little longer00:20
clarkbpabelanger: meh00:20
jheskethianw: cool, thanks :-)00:20
pabelangerclarkb: indeed00:20
clarkbat this point we have all the disk we need :)00:20
pabelangerclarkb: going to get an experiment job and test their gates still00:21
*** amrith is now known as _amrith_00:21
pabelangerrally should be off bare-precise by tomorrow00:21
*** sflanigan has joined #openstack-infra00:21
clarkboh good, I am slightly worriedabout bare-precise because it is a snapshot image so I can't build it with nodepoold offline :/00:22
clarkbI can proably edit the db directly to add the existing ones in00:23
*** sputnik13 has quit IRC00:23
clarkbor we just wait an hour ofr one to build00:23
clarkbthe release doesn't need it so not too worried00:23
*** thorst has joined #openstack-infra00:25
pabelangerthink we should be safe to let nodepoold build it00:25
pabelangerjust means system-config / project-config are blocked00:25
*** ddieterly[away] has quit IRC00:26
*** mtanino__ has joined #openstack-infra00:27
*** Qiming has quit IRC00:28
*** mtanino has quit IRC00:28
openstackgerritPaul Belanger proposed openstack-infra/project-config: Experimental gate-kolla-dsvm-build-centos-7-binary
pabelangerYa, so just kolla left (and us for devstack-centos7)00:32
*** asettle has quit IRC00:32
pabelanger^ will give us an experimental job to test drive in kolla00:32
*** mtanino__ has quit IRC00:33
openstackgerritIan Wienand proposed openstack-infra/nodepool: Add some tracking for image sizes
openstackgerritPaul Belanger proposed openstack-infra/project-config: Migrate rally to centos-7
ianwdistro: oraclelinuxnode: 'devstack-centos7'00:36
pabelangerYa see that00:36
pabelangerseems odd00:36
ianwnot sure i want to even know what's going on00:36
*** erlon has quit IRC00:37
*** akshai_ has quit IRC00:37
*** achanda has quit IRC00:40
*** ddieterly has joined #openstack-infra00:40
*** _amrith_ is now known as amrith00:40
*** tiswanso has quit IRC00:40
openstackgerritPaul Belanger proposed openstack-infra/project-config: Experimental gate-kolla-dsvm-build-centos-7-binary
jheskethSo has anybody discussed mirroring npm in infra before?00:41
*** sc68cal has quit IRC00:42
openstackgerritPaul Belanger proposed openstack-infra/project-config: Set min-ready for devstack-centos7 to 0
pabelangerjhesketh: I think krotscheck is working on it00:43
clarkbjhesketh: I think krotscheck was working on it and there is an afs volume and everything00:43
clarkbunfortunately I don't think it will fix all of our issues if mirroring external link things works the same way as with pypi00:43
clarkbwould be good to figur ethat out00:43
jheskethI'll see if I can find the changes00:44
jheskethyeah but it's something we should be doing for the same reasons as pypi00:45
clarkbexcept that I htink it won't fix any of the recent fails we have had00:45
pabelangerI am surprised we have xenial-backports enabled:
*** thorst has quit IRC00:45
pabelangerdo we do that for trusty too?00:45
clarkbthe first was bitbucket rejecting downloads on external links, the other was json encoded in a way that npm couldn't understand00:45
clarkbhaving a mirror won't help either case aiui00:45
*** thorst has joined #openstack-infra00:46
clarkbpabelanger: I think there was a thing that spamaps fixed with that?00:46
*** keedya has quit IRC00:46
jheskethclarkb: oh yeah, agreed.. just thinking about it in general00:46
*** Jeffrey4l has joined #openstack-infra00:48
pabelangerclarkb: Ya, looks like we mirror it with reprepro too00:48
pabelangerso, some projects must us it00:48
*** annegentle has joined #openstack-infra00:49
krotscheckjhesketh: Yes. I have an approach, it requires some modification of an existing registry mirror.00:50
jheskethkrotscheck: cool, do you have a review topic or a list of required work?00:50
krotscheckjhesketh: It's pretty far down my todo list what with the summit coming up. I have nothing written down.00:52
*** r1chardj0n3s has joined #openstack-infra00:52
jheskethokay, no worries00:52
krotscheckjhesketh: the project is here
Shrewsjhesketh: it wasn't the merge speed so much as there were things that were missed. i +A'd that one00:53
jheskethkrotscheck: oh how annoyign to have to work around that limit00:53
krotscheckr1chardj0n3s, jhesketh: The project right now is a bunch of noop hooks. I managed to get so far as to have the tarballs be rewritten into the appropriate folder structure, but the mirror doesn't permit doing the same with the json files.00:54
Shrewsjhesketh: i'm going to be writing up a reviewer guide so mordred and myself aren't the only ones holding the context you were missing  :)00:54
jheskethShrews: noted. It was certainly my mistake, sorry.00:54
clarkball image builds complete with no issues00:54
clarkband I have started the rax uploads00:54
Shrewsjhesketh: no worries00:54
*** zz_dimtruck is now known as dimtruck00:54
*** thorst has quit IRC00:54
clarkbonce !rax is done I am just going to queue up all the rax uploads00:54
clarkbthen stop computering for a bit00:54
*** annegentle has quit IRC00:54
krotscheckr1chardj0n3s, jhesketh: So the next step is to go to registry-static and the yahoo team and check with them how to modify the existing script to permit that.00:54
r1chardj0n3skrotscheck: is there some background on that porject and why existing npm mirroring solutions aren't being used?00:55
pabelangerclarkb: are you doing the cut over tonight, or going to wait until tomorrow morning?00:55
krotscheckr1chardj0n3s: Existing solutions like what, mirroring the CouchDB?00:55
clarkbpabelanger: I think tomorrow morning after release team releases00:56
clarkbthat way I don't have to work until 10pm tonight00:56
krotscheckr1chardj0n3s: NPM explicitly disallows that one now. Their fullfat db has been deprecated and is no longer guaranteed to exist in the future.00:56
pabelangerclarkb: sounds good. I'll sign off now then00:57
pabelangerlater all00:57
*** zhurong has joined #openstack-infra00:58
krotscheckI just got "E: Unable to locate package apt-transport-https" on a rax-dfw node, any ideas?00:59
r1chardj0n3skrotscheck: I'm not sure what you're referring to with the couchdb stuff; I've only just started looking into the mirroring and started at
*** harlowja has quit IRC01:00
krotscheckr1chardj0n3s NPM on-site is their $$$ solution01:01
r1chardj0n3sah, right01:02
krotscheckr1chardj0n3s So we're doing this instead:
clarkbkrotscheck: where?01:02
clarkbkrotscheck: always always provide job log links :)01:02
*** bpokorny has quit IRC01:02
r1chardj0n3skrotscheck: ok, cool.01:02
*** pvaneck has quit IRC01:03
*** harlowja has joined #openstack-infra01:03
r1chardj0n3skrotscheck: I think I'll step back and stop interfering. If I can help though, please ask.01:03
krotscheckr1chardj0n3s: Would _love_ the help. But I have dinner with the fam now, so can't do much right now to loop you in :)01:04
r1chardj0n3skrotscheck: okie; also, I've thrown jhesketh under the "help if possible from the infra side" bus :-)01:04
clarkbkrotscheck: best guess based on that is you are lacking an apt-get update before apt-get install and it doens't know where the package is as a result01:04
openstackgerritFei Long Wang proposed openstack-infra/project-config: Add Tempest gate job for Zaqar
*** jcooley_ has joined #openstack-infra01:07
*** r1chardj0n3s has left #openstack-infra01:08
*** tqtran has joined #openstack-infra01:11
*** tqtran has quit IRC01:16
*** tiswanso has joined #openstack-infra01:18
*** achanda has joined #openstack-infra01:18
*** sridhar_ram1 has quit IRC01:18
*** asettle has joined #openstack-infra01:22
*** dprince has joined #openstack-infra01:24
*** keedya has joined #openstack-infra01:24
openstackgerritFei Long Wang proposed openstack-infra/project-config: Add Tempest gate job for Zaqar
*** Qiming has joined #openstack-infra01:26
*** shashank_hegde has joined #openstack-infra01:26
*** asettle has quit IRC01:28
openstackgerritMatt Riedemann proposed openstack/requirements: Require fixtures 2.0.0
*** sdake has quit IRC01:28
*** aysyd has quit IRC01:30
*** sdake has joined #openstack-infra01:32
*** ajmiller has joined #openstack-infra01:38
*** ajmiller has quit IRC01:39
*** julim has joined #openstack-infra01:40
*** baoli has joined #openstack-infra01:44
*** baoli has quit IRC01:47
*** shashank_hegde has quit IRC01:47
*** ddieterly is now known as ddieterly[away]01:47
*** baoli has joined #openstack-infra01:48
*** thorst has joined #openstack-infra01:51
*** nwkarsten has joined #openstack-infra01:52
*** cody-somerville has joined #openstack-infra01:53
*** Thelo has joined #openstack-infra01:55
TheloHi all, I got this error message in zuul "This change was unable to be automatically merged with the current state of the repository. Please rebase your change and upload a new patchset" (, however I dont understand why the merge would fail since the patchset has no conflict (, Does anyone know why? and what I can do to fix that?01:55
*** esikachev has joined #openstack-infra01:55
fungiThelo: it's saying it conflicts with the state of the target branch, not with other proposed changes01:57
*** nwkarsten has quit IRC01:57
fungialso the merge resolver in the jgit library gerrit uses is not as smart as the variety of conflict resolution algorithms cgit employs, so something that rebases cleanly locally may still be something gerrit is unable to figure out how to resolve itself01:58
*** thorst has quit IRC01:59
*** amrith is now known as _amrith_01:59
fungiand now i'm caught up with scrollback... missed a very exciting day. sorry all! :/01:59
fungiclarkb: thanks for building us a new nodepool02:00
*** esikachev has quit IRC02:00
clarkbfungi: np02:00
clarkbimage uploads are progressing well should be ready to switch servers tomorrow02:00
fungithere's one other server that can _probably_ be deleted to free up quota now: the old running precise. but since we just replaced it yesterday i'm hesitant to say for certain we won't need to switch back02:01
fungiwhen that goes we'll get back another 15gb memory quota02:01
clarkbafstest was sufficient for my needs02:01
fungii saw in skimming02:01
fungii didn't mean to leave us that tight on quota02:02
Thelofungi: So if I understand it has nothing to do with this patchset but with the other party of the merge. If so, how can I rebase locally? Do you know where zuul maintain it?02:02
*** apoorvad has quit IRC02:02
fungialso the old groups.o.o may also be able to be deleted if mrmartin and yolanda have finished with that02:02
*** kengo_sakai has joined #openstack-infra02:02
*** salv-orlando has joined #openstack-infra02:03
fungiThelo: download the change, rebase it on the current branch tip, then push it back up for review02:03
fungiif i understand what you're asking02:03
*** rajinir has quit IRC02:05
*** shashank_hegde has joined #openstack-infra02:05
fungiThelo: oh, _your_ zuul is reporting that it can't merge a change?02:06
fungii thought you were saying our zuul was leaving a comment on a change of yours that it couldn't merge02:06
*** baoli has quit IRC02:06
Thelofungi: Yes02:06
fungiThelo: you may need to look at the log for your merger worker. one cause could be a corrupt openstack/cinder repo on disk02:09
fungiThelo: basically the merger worker reported to zuul that it was unable to create a merge commit in response to a merge work request02:09
Thelofungi: ok, thanks I ll check that02:13
*** mriedem has quit IRC02:16
*** piet has joined #openstack-infra02:17
*** dprince has quit IRC02:19
*** sdake_ has joined #openstack-infra02:20
*** ddieterly[away] is now known as ddieterly02:21
*** sdake has quit IRC02:22
*** salv-orlando has quit IRC02:23
*** shashank_hegde has quit IRC02:24
*** ddieterly is now known as ddieterly[away]02:26
*** salv-orlando has joined #openstack-infra02:28
*** asettle has joined #openstack-infra02:29
*** salv-orlando has quit IRC02:33
*** edmondsw has quit IRC02:36
*** piet has quit IRC02:37
*** yamamoto_ has joined #openstack-infra02:37
*** gildub has quit IRC02:42
*** shashank_hegde has joined #openstack-infra02:45
*** jamesmcarthur has joined #openstack-infra02:47
clarkbthis ovh bhs one upload is going forever02:48
clarkbits funny that the cross atlantic uploads are quicker02:48
*** jamesmcarthur has quit IRC02:50
*** jamesmcarthur has joined #openstack-infra02:51
*** piet has joined #openstack-infra02:55
krotscheckclarkb: So why did it work in other nodes and not this one?02:55
clarkbkrotscheck: index was up to date at the time02:55
clarkbou should always apt-get update propr to doing an install if you want the install to succeed02:56
*** thorst has joined #openstack-infra02:56
*** eliqiao has quit IRC03:01
*** openstackstatus has quit IRC03:01
*** zhurong_ has joined #openstack-infra03:02
*** zhurong has quit IRC03:02
*** xarses has joined #openstack-infra03:02
*** eliqiao has joined #openstack-infra03:02
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Reordered builders in npm builds
krotscheckclarkb: You were right, order of operations error. ^^03:03
*** thorst has quit IRC03:04
*** tphummel has quit IRC03:04
*** jamesmcarthur has quit IRC03:05
fungioops, approved when i just meant to +203:05
krotscheckfungi: Wow, fast response :)03:05
fungistill needs another reviewer, but i agree03:05
fungithat should solve it03:05
krotscheckfungi Most of the horizon builds should now be running though, they're using a different job.03:06
*** sdake_ has quit IRC03:06
krotscheckAt the moment, I believe only the ironic webclient and eslint-config-openstack are still impacted03:06
fungifinally caught up on irc and e-mail for the day, so about to drop off again03:06
fungiclarkb: sorry again for disappearing right before you discovered dib bug of doom :(03:07
*** sdake has joined #openstack-infra03:07
clarkbjust 7 more image uploads to go!03:07
fungii too did not leave myself ssh'd into nodepool.o.o, so couldn't have helped on that recovery option03:08
clarkbthey are all queued I will probably drop off myself and get back to it tomorrow03:08
krotscheckFastest code review ever. I should submit more patches around 8PM :)03:08
fungii'll be around and solid at a computer again at or before 1800 utc, so can help with the nodepool switchover if it starts sometime after that03:09
*** achanda has quit IRC03:09
fungidriving home a good chunk of tomorrow so will not likely be online much prior to that though03:10
funginight all!03:10
clarkbI have an erand I have to run prior to 1030 local time tomorrow so nodepool things may start later than initially anticipated03:10
fungiworks fine for me in that case03:12
fungihappy to pitch in when the time comes03:13
*** sdake has quit IRC03:17
*** sdake has joined #openstack-infra03:18
*** asselin has quit IRC03:19
*** yamamoto_ has quit IRC03:20
*** asselin has joined #openstack-infra03:20
*** prometheanfire has quit IRC03:22
*** prometheanfire has joined #openstack-infra03:23
openstackgerritMerged openstack-infra/project-config: Reordered builders in npm builds
*** jmccrory has quit IRC03:26
*** ddieterly[away] has quit IRC03:26
*** jmccrory has joined #openstack-infra03:26
*** kengo_sa_ has joined #openstack-infra03:28
*** kengo_sa_ has quit IRC03:28
*** sdake has quit IRC03:28
*** kengo_sakai_mba has joined #openstack-infra03:28
*** piet has quit IRC03:29
*** kengo_sakai_mba has quit IRC03:30
*** kengo_sakai_mba has joined #openstack-infra03:30
*** gildub has joined #openstack-infra03:32
*** tiswanso has quit IRC03:32
*** dimtruck is now known as zz_dimtruck03:32
*** kengo_sakai_mba has quit IRC03:34
*** piet has joined #openstack-infra03:34
*** ramishra has joined #openstack-infra03:37
*** fitoduarte has joined #openstack-infra03:41
*** FallenPegasus has joined #openstack-infra03:42
*** shashank_hegde has quit IRC03:42
*** mylu has quit IRC03:45
*** keedya has quit IRC03:47
*** salv-orlando has joined #openstack-infra03:50
*** Daviey has quit IRC03:50
*** Daviey has joined #openstack-infra03:50
*** flepied1 has quit IRC03:50
*** flepied has joined #openstack-infra03:51
*** salv-orlando has quit IRC03:52
*** grassass has quit IRC03:52
*** salv-orlando has joined #openstack-infra03:53
*** piet has quit IRC03:56
*** julim has quit IRC03:57
*** salv-orlando has quit IRC03:58
*** jamespage has quit IRC04:00
*** gnuoy has quit IRC04:00
*** Nikolay_St has quit IRC04:00
*** gnuoy has joined #openstack-infra04:01
*** jamespage has joined #openstack-infra04:01
*** FallenPegasus has quit IRC04:01
*** thorst has joined #openstack-infra04:01
*** mylu has joined #openstack-infra04:02
*** fedexo has joined #openstack-infra04:02
*** yaume has joined #openstack-infra04:03
*** thorst has quit IRC04:09
*** yamamoto_ has joined #openstack-infra04:10
*** Sukhdev has joined #openstack-infra04:11
*** shu-mutou has joined #openstack-infra04:11
*** links has joined #openstack-infra04:12
*** jcooley_ has quit IRC04:15
*** asettle has quit IRC04:17
*** asettle has joined #openstack-infra04:18
*** yfried has joined #openstack-infra04:20
*** amotoki has joined #openstack-infra04:20
*** shashank_hegde has joined #openstack-infra04:22
*** achanda has joined #openstack-infra04:22
*** maishsk has quit IRC04:26
*** persia has quit IRC04:31
*** persia has joined #openstack-infra04:33
shu-mutouHi everyone, Error occurred at python jobs (docs, pep8 and py27) in Magnum-UI. According to logs, pbr seems to fail to parse upper-constraints.txt.
shu-mutoueveryone: Does following patch cause the error?
*** achanda has quit IRC04:37
*** yaume has quit IRC04:40
*** eliqiao_ has joined #openstack-infra04:43
*** eliqiao has quit IRC04:44
*** mylu has quit IRC04:45
openstackgerritIan Wienand proposed openstack/diskimage-builder: Fix disk usage report
*** xyang1 has quit IRC04:48
*** salv-orlando has joined #openstack-infra04:57
*** armax has quit IRC04:59
*** abregman has joined #openstack-infra05:00
*** yfried has quit IRC05:01
*** salv-orlando has quit IRC05:01
*** kengo_sakai_mba has joined #openstack-infra05:02
*** asettle has quit IRC05:02
*** oanson has joined #openstack-infra05:03
*** asettle has joined #openstack-infra05:03
*** rossella_s has quit IRC05:03
*** achanda has joined #openstack-infra05:03
*** rossella_s has joined #openstack-infra05:03
*** kushal has quit IRC05:04
*** thorst has joined #openstack-infra05:06
*** salv-orlando has joined #openstack-infra05:11
*** jaosorior has joined #openstack-infra05:13
openstackgerritxiangxinyong proposed openstack/requirements: Add python-smaugclient into requirements
*** thorst has quit IRC05:13
*** ramishra has quit IRC05:16
*** harlowja_at_home has joined #openstack-infra05:17
*** kushal has joined #openstack-infra05:17
*** nwkarsten has joined #openstack-infra05:19
openstackgerritMasayuki Igawa proposed openstack-infra/subunit2sql: Fix lock range, again
*** nwkarsten has quit IRC05:24
sc`pabelanger: worked around the pip uninstall in my bootstrap script. my first successful integration build post devstack-centos7 migration just completed. i intend to spend a few cycles looking at chef's infra bits and seeing how they can be improved05:24
*** shu-mutou is now known as shu-mutou-AFK05:26
openstackgerritIan Wienand proposed openstack/diskimage-builder: Fix disk usage report
openstackgerritIan Wienand proposed openstack/diskimage-builder: Skip gentoo test
openstackgerritIan Wienand proposed openstack/diskimage-builder: Add qcow2 generation for better test coverage
*** flepied has quit IRC05:33
*** achanda has quit IRC05:33
openstackgerritJoshua Harlow proposed openstack/requirements: Bump voluptuous to 0.8.9
*** Daisy has joined #openstack-infra05:39
*** fedexo has quit IRC05:39
*** Sukhdev has quit IRC05:40
*** fedexo has joined #openstack-infra05:42
*** rcernin has joined #openstack-infra05:45
openstackgerritDoug Hellmann proposed openstack-infra/project-config: clean up sahara-extra release jobs
openstackgerritIan Wienand proposed openstack/diskimage-builder: Fix disk usage report
*** ybathia has joined #openstack-infra05:50
*** yfried has joined #openstack-infra05:53
*** jcooley_ has joined #openstack-infra05:54
AJaegershu-mutou-AFK: error is "git history requires a target version of pbr.version.SemanticVersion(1.0.1), but target version is pbr.version.SemanticVersion(0.0.1)"05:55
openstackgerritMerged openstack-infra/irc-meetings: Update the Glare team meeting chair
*** achanda has joined #openstack-infra05:59
*** aeng has quit IRC05:59
*** cody-somerville has quit IRC06:02
AJaegerwow, new zuul queue length record AFAIK: "Queue lengths: 80242 events, 51 results. "06:05
openstackgerritIan Wienand proposed openstack/diskimage-builder: Fix disk usage report
*** Nikolay_St has joined #openstack-infra06:07
*** Hal has joined #openstack-infra06:07
*** Hal is now known as Guest7161106:08
*** mikelk has joined #openstack-infra06:10
*** thorst has joined #openstack-infra06:11
*** harlowja_at_home has quit IRC06:12
*** kushal has quit IRC06:12
*** cody-somerville has joined #openstack-infra06:17
AJaegerinfra-root, I cannot login . Do we have a problem with openstackid again?06:18
*** thorst has quit IRC06:18
*** kengo_sakai_mba has quit IRC06:19
AJaegerinfra-root: Didn't fungi move openstackid to a new server? Is there perhaps an old IP address or something like that in use?06:19
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml
*** asettle has quit IRC06:20
*** achanda has quit IRC06:22
*** scheuran has joined #openstack-infra06:22
yolandaAJaeger, will take a quick look. I need to leave for doctors the whole morning, but I'm still here06:23
yolandaAJaeger, what's the problem you are seeing?06:24
*** xiangxinyong has joined #openstack-infra06:25
*** shashank_hegde has quit IRC06:25
*** pcaruana has joined #openstack-infra06:26
yolandai just logged successfully06:26
*** kushal has joined #openstack-infra06:26
AJaegeryolanda: no login possible at all, clicking on login button does not do anything06:26
*** akaszuba has joined #openstack-infra06:26
yolandai am logged at translate now06:26
AJaeger"This site can’t be reached06:26 took too long to respond.06:26
AJaegerOk, then we have a connectivity problem ;(06:26
yolandacan you ping to the server? which ip is giving you?06:27
* AJaeger will try later then.06:27
openstackgerritDoug Hellmann proposed openstack-infra/project-config: clean up sahara-extra release jobs
*** fedexo has quit IRC06:27
AJaegerIP is
AJaegerping doesn't work for me...06:27
yolandawhen i ping i have
yolandalet me double check, it may be related to server mov06:28
* AJaeger checks other system06:28
AJaegeryes, 23... is correct.06:28
*** eliqiao_ is now known as eliqiao06:28
* AJaeger has an old IP address in /etc/hosts ;(06:29
AJaegeryolanda: user error on my side -thanks06:29
* AJaeger removed the IP address now06:29
yolandacool, if all problems were like those :)06:29
AJaegerthanks for the help, the IP hint was essential06:30
AJaegereverything fine now, I'm in - yolanda all the best for your day!06:30
*** flepied has joined #openstack-infra06:31
*** Thelo has quit IRC06:31
*** asettle has joined #openstack-infra06:33
*** mrmartin has joined #openstack-infra06:34
yolandathanks AJaeger, i'll be back on early afternoon06:34
* AJaeger will try not to break anything ;)06:35
*** asettle has quit IRC06:37
*** ramishra has joined #openstack-infra06:39
*** ybathia has quit IRC06:41
openstackgerritxiangxinyong proposed openstack/requirements: Add python-smaugclient into requirements
*** tqtran has joined #openstack-infra06:43
*** tesseract has joined #openstack-infra06:45
*** tesseract is now known as Guest1287606:45
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints
*** vgridnev has joined #openstack-infra06:46
*** tqtran has quit IRC06:47
*** camunoz has quit IRC06:48
*** yaume has joined #openstack-infra06:51
*** salv-orlando has quit IRC06:52
*** jcooley_ has quit IRC06:56
*** woodster_ has quit IRC06:57
*** ramineni_ has joined #openstack-infra06:57
ramineni_AJaeger: hi06:59
openstackgerritMerged openstack-infra/project-config: Normalize projects.yaml
ramineni_i have added a job to publish the docs to docs.openstack , I can see the docs also , but it is not added here , under openstack-services07:01
AJaegerramineni_: That page needs to be edited manually.07:01
AJaegerIt lives in openstack-manuals/www/developer/openstack-projects.html. Please send a change request...07:02
ramineni_AJaeger: oohk , thanks a lot for clarifying :)07:02
*** maishsk has joined #openstack-infra07:04
*** placko_ has joined #openstack-infra07:05
*** Nikolay_St has quit IRC07:07
*** mrmartin has quit IRC07:08
openstackgerritIan Wienand proposed openstack/diskimage-builder: [WIP] Fix disk usage report
*** nmagnezi has joined #openstack-infra07:11
*** ramishra has quit IRC07:11
*** ihrachys has joined #openstack-infra07:11
*** jlanoux has joined #openstack-infra07:13
*** yaume has quit IRC07:15
*** tosky has joined #openstack-infra07:16
*** thorst has joined #openstack-infra07:16
*** arxcruz has joined #openstack-infra07:17
openstackgerritThierry Carrez proposed openstack-infra/project-config: Give project teams access to approve stable/mitaka changes
*** ramineni_ has quit IRC07:19
*** kengo_sakai_mba has joined #openstack-infra07:19
openstackgerritvikram.choudhary proposed openstack-infra/project-config: Adding neutron-dynamic-routing as neutron's stadium project.
*** vikram has joined #openstack-infra07:20
vikramAJaeger: Can you please bless
*** rlandy has quit IRC07:23
*** thorst has quit IRC07:24
*** vincentll has joined #openstack-infra07:24
*** Daisy has quit IRC07:24
*** Daisy has joined #openstack-infra07:25
AJaegervikram: I'll review it in the next days for sure. With release day today I'm only pushing critical stuff through.07:25
vikramAJaeger: Thanks.. np07:25
vikramAJaeger: BTW, thanks for your pointer on the repo split.. It helped07:25
*** ramishra has joined #openstack-infra07:27
*** kengo_sakai_mba has quit IRC07:28
*** matrohon has joined #openstack-infra07:29
*** Daisy has quit IRC07:29
openstackgerritIan Wienand proposed openstack/diskimage-builder: Fix disk usage report
jaosoriorHey guys, I just saw a commit to barbican with the title: "Merge tag '2.0.0'" ... it says it shouldn't contain any changes, however there is a change that removes the default branch from the .gitreview file. Is this alright?
AJaegerjaosorior: ask on #openstack-release - and this change is perfectly fine07:31
*** dizquierdo has joined #openstack-infra07:31
*** daemontool has joined #openstack-infra07:32
AJaegerjaosorior: gerrit just doesn't get it ;)07:32
*** yfried_ has joined #openstack-infra07:32
*** yfried has quit IRC07:32
jaosoriorAJaeger: alright, thanks! though I guess the commit message should be changed in the bot, since it explicitly tells people to ask people in #openstack-infra07:33
*** ramishra has quit IRC07:36
*** dtantsur|afk is now known as dtantsur07:36
*** mrmartin has joined #openstack-infra07:36
*** ramishra has joined #openstack-infra07:39
*** flepied1 has joined #openstack-infra07:41
*** asettle has joined #openstack-infra07:42
AJaegerdhellmann, ttx ^07:43
*** flepied has quit IRC07:43
*** cody-somerville has quit IRC07:45
*** jamielennox is now known as jamielennox|away07:45
ttxjaosorior: yes, that's a commit that Gerrit has trouble representing07:45
*** Daisy has joined #openstack-infra07:45
ttxIt's about merging tags from release branches back into master07:45
*** asettle has quit IRC07:45
ttxIt should be approved. I think that's documented somewhere.07:46
* ttx looks07:46
*** shu-mutou-AFK is now known as shu-mutou07:46
shu-mutouAJaeger: Thank you for your help!!!07:47
*** jaosorior has quit IRC07:47
*** shardy has joined #openstack-infra07:47
ttxhmm, can't find doc07:47
*** jaosorior has joined #openstack-infra07:47
*** yaume has joined #openstack-infra07:47
jaosoriorttx: I see07:48
*** zeih_ has joined #openstack-infra07:51
*** tosky has quit IRC07:51
*** matrohon_ has joined #openstack-infra07:51
*** tosky has joined #openstack-infra07:51
*** matrohon_ has quit IRC07:51
ttxjaosorior: found ref, let me link07:54
*** salv-orlando has joined #openstack-infra07:54
*** ramishra has quit IRC07:55
jaosoriorI see07:55
jaosoriorI'll approve it now then07:55
jaosoriorthanks for the info ttx07:55
robcresswellGlad I got caught the tail end of that, I was also a little confused.07:57
*** yfried_ has quit IRC07:58
*** HeOS has joined #openstack-infra08:00
*** zeih_ has quit IRC08:01
*** vgridnev has quit IRC08:02
*** yolanda has quit IRC08:03
*** sflanigan has quit IRC08:04
*** jordanP has joined #openstack-infra08:06
*** Nikolay_St has joined #openstack-infra08:08
*** zeih_ has joined #openstack-infra08:09
*** salv-orlando has quit IRC08:10
*** zeih_ has quit IRC08:14
*** ramineni_ has joined #openstack-infra08:15
*** ramishra has joined #openstack-infra08:15
AJaegerinfra-root, the topic can be cleared now I guess - could anybody issue a #status ok, please?08:15
*** Jeffrey4l has quit IRC08:16
*** ramineni_ has quit IRC08:16
*** ramineni_ has joined #openstack-infra08:17
jheskethAJaeger: what was the resolution?08:18
AJaegerjhesketh: update npm locally, let me grab you the change...08:20
* AJaeger can't copy and paste today ;(08:21
AJaeger is the one I want08:21
AJaegerThat's the real change, the other is a fix08:21
*** thorst has joined #openstack-infra08:21
jheskethAJaeger: thanks08:21
jheskethAJaeger: I assume those jobs are passing?08:22
*** Jeffrey4l has joined #openstack-infra08:22
AJaegerjhesketh: reading backscoll and checking api-site and searchlight-ui: Yes08:23
jheskethcool, thanks08:23
jhesketh(sorry thought you may have known off the top of your head, otherwise I Could have checked)08:23
jhesketh#status ok jobs depending on npm are now working again08:24
jheskethhmm, where's the status bot08:24
*** salv-orlando has joined #openstack-infra08:26
*** ramishra has quit IRC08:26
* jhesketh prods the bot08:28
*** openstackstatus has joined #openstack-infra08:29
*** ChanServ sets mode: +v openstackstatus08:29
*** thorst has quit IRC08:30
*** jistr has joined #openstack-infra08:30
jhesketh#status ok jobs depending on npm are now working again08:30
openstackstatusjhesketh: sending ok08:30
AJaegerthanks, jhesketh !08:32
*** jcoufal has joined #openstack-infra08:32
-openstackstatus- NOTICE: jobs depending on npm are now working again08:33
jheskethAJaeger: hmm, but I think restarting the bot was a mistake... it's lost its memory of the channel topics08:33
*** esikachev has joined #openstack-infra08:33
jheskethI don't think I had a choice though as it wasn't logging anything for the last 6 or so hours and appeared to be hung08:33
jhesketh(I should have straced and poked a little deeper, but didn't think restarting would be harmful)08:34
AJaegerOOps;( No idea on the bot08:34
jheskethI'll send a notice after this for channel ops to reset their own topics08:34
*** ramineni_ has quit IRC08:34
AJaegercan you reset ours as well, please?08:35
toskyand the -meeting- ones08:36
openstackstatusjhesketh: finished sending ok08:36
zigoAJaeger: Hi there! Can I ask you for an advice?08:36
zigoAJaeger: We have the issue that "gbp buildpackage" fails because the $WORKSPACE git repository is detached from HEAD, so git-buildpackage doesn't know in which branch it's running.08:37
openstackgerritDavanum Srinivas (dims) proposed openstack/requirements: Updated from generate-constraints
zigoAJaeger: Can we checkout a branch or something? How can we fix?08:37
*** dtantsur is now known as dtantsur|brb08:38
tobiashjhesketh: I just observed a strange behaviour in our zuul08:39
AJaegerzigo, there are environment variables you can use like ZUUL_REFNAME08:39
zigoAJaeger: I should use it to do what? How? I'm not sure I get it here ...08:40
AJaegerzigo: check your job in jenkins and see the environment variable settings for it.08:40
zigoAJaeger: I do know the branch name, it's referenced in debian/gbp.conf, that's not what I need.08:40
zigoAJaeger: I need the git repo to not be detached from HEAD.08:40
zigoAJaeger: Can I do a "git checkout debian/mitaka"? Will that work?08:41
*** daemontool has quit IRC08:41
tobiashI have pipelines check, test, etc. where test is manually triggered by comment. My project only defines builds for the test pipeline, but every patch enters the check pipeline triggering the zuul-merger and no builds and no voring08:41
AJaegerzigo, you need the git repo with the change you're testing applied to it.08:41
AJaegerAnd that's the detached state you get...08:41
zigoAJaeger: But I need the git repo to be on a branch name, otherwise git-buildpackage gets very much confused...08:42
AJaegerzigo: Now I'm confused - and don't have time to dig into this with you.08:42
zigoAJaeger: The change isn't commited to git, is it?08:42
zigonv mind08:43
*** yolanda has joined #openstack-infra08:43
*** ramishra has joined #openstack-infra08:45
*** yolanda has quit IRC08:49
*** ramineni_ has joined #openstack-infra08:50
*** cdent has joined #openstack-infra08:50
*** yolanda has joined #openstack-infra08:50
*** e0ne has joined #openstack-infra08:50
*** shu-mutou is now known as shu-mutou-AFK08:51
*** salv-orl_ has joined #openstack-infra08:52
*** salv-orlando has quit IRC08:55
*** jaosorior has quit IRC08:59
*** jaosorior has joined #openstack-infra09:00
zigoAJaeger: FYI, looks like I fixed the issues, though now I need to log off and re-login to get sbuild working... :/09:02
openstackgerritKiall Mac Innes proposed openstack-infra/project-config: Add new jobs to openstack/designate-tempest-plugin
*** rossella_s has quit IRC09:03
*** rossella_s has joined #openstack-infra09:03
dimsAnyone here know the pypi mirror setup? need to tickle it to pickup latest reno release09:04
*** nijaba has quit IRC09:05
*** sambetts|afk is now known as sambetts09:05
*** nijaba has joined #openstack-infra09:07
*** ifarkas has joined #openstack-infra09:07
*** derekh has joined #openstack-infra09:09
*** zhurong_ has quit IRC09:09
openstackgerritThomas Goirand proposed openstack-infra/project-config: Execute sbuild after relogin
*** ramineni_ has quit IRC09:10
zigoAJaeger: <--- This should fix the last issue I found.09:11
zigo(no worries if you're busy right now...)09:12
openstackgerritAndreas Jaeger proposed openstack/requirements: Require reno 1.6.2 or newer
*** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs | bugs | source | channel logs"09:12
AJaegerzigo: please do not add me to reviews, I'll review via gerrit/gertty and not via emails09:14
*** vgridnev has joined #openstack-infra09:17
openstackgerritKiall Mac Innes proposed openstack-infra/project-config: Add new jobs to openstack/designate-tempest-plugin
*** hichihara has quit IRC09:23
*** kzaitsev_mb has joined #openstack-infra09:23
*** thorst has joined #openstack-infra09:27
*** ramishra has quit IRC09:28
*** ramineni_ has joined #openstack-infra09:29
*** ramishra has joined #openstack-infra09:30
openstackgerritDong Ma proposed openstack-infra/subunit2sql: Add more db api tests
vkramskikhhey folks, does anybody know if it's decided to stay with node v4 on jenkins workers or node will be reverted back to 0.10 after fix of ?09:31
*** links has quit IRC09:31
*** alex_xu has quit IRC09:32
AJaegervkramskikh: once we go to xenial, we will have newer version for sure. Best discuss with krotscheck later09:34
*** thorst has quit IRC09:34
*** alex_xu has joined #openstack-infra09:36
*** nadya has joined #openstack-infra09:36
*** yamamoto_ has quit IRC09:40
*** Daisy has quit IRC09:40
vkramskikhand when is switch to xenial planned?09:42
AJaegervkramskikh: in a couple of months09:42
AJaegerit'S not released yet but we've started testing already09:42
*** links has joined #openstack-infra09:45
eliqiaohi there, I have trouble while passing Jenkins gate. the thing is: I can pass Jenkins check, but faied to pas Jenkins gate with same job. Is there any sepcify word like 'reverify'/'recheck' I can use to kick infra to only run Jenkins gate?09:48
openstackgerritColleen Murphy proposed openstack-infra/puppet-pip: Add new pip provider to use pypi mirrors
eliqiaoI tried 'reverify', infra will trigger Jenkins check too09:48
AJaegereliqiao: no, it always goes back to check queue09:49
AJaegereliqiao: so, there's some nondeterminism in the job/change/repo - let's investigate first and then decide what to do09:50
eliqiaoAJaeger: okay, but it's really time costing.09:50
AJaegereliqiao: if you enqueue again the gate and it fails again, it's much more time costing09:51
AJaegerIT could fail in the gate due to changes that merged before or newer packages - so we first run in check to ensure that everything is fine.09:51
openstackgerritMarton Kiss proposed openstack-infra/groups: Fix strict warning on user group page
eliqiaoAJaeger: I get it now. Make sense.09:52
*** kzaitsev_mb has quit IRC09:52
*** gildub has quit IRC09:52
eliqiaoAJaeger: can I ask a question. I see some patches (only has doc changes) only run specify jobs but other run all jobs, do you konw how to configure it? for example
AJaegereliqiao: all done in zuul/layout.yaml - but that is complicated and has some pitfalls.09:54
AJaegereliqiao: lunch time now, will be back later...09:54
eliqiaoAJaeger: thanks, have good lunch.09:54
*** dtantsur|brb is now known as dtantsur09:55
*** achanda has joined #openstack-infra09:56
*** k4n0 has joined #openstack-infra09:57
*** kzaitsev_mb has joined #openstack-infra09:59
fricklerthere's still lots of channels with broken status lines09:59
*** k4n0 has quit IRC10:00
*** obondarev has quit IRC10:00
fricklersomehow statusbot did not reset them properly?10:01
*** ramineni_1 has joined #openstack-infra10:01
fricklerah, just found it in the backlog, sorry for disturbing ;)10:02
*** achanda has quit IRC10:02
*** ramineni_ has quit IRC10:04
*** ociuhandu has quit IRC10:05
*** obondarev has joined #openstack-infra10:06
*** obondarev has quit IRC10:12
*** kzaitsev_mb has quit IRC10:12
*** Qiming has quit IRC10:13
mrmartinwhat is wrong here: ? bindep error as: "ERROR: These requested packages were not installed"10:13
*** Kiall has quit IRC10:14
*** Kiall has joined #openstack-infra10:14
*** kzaitsev_mb has joined #openstack-infra10:15
*** cdent has quit IRC10:16
vkramskikhcould anybody please help with review ? there is a stable/mitaka branch created for fuel-ui repo, but fuel-ui-core group can't merge into that branch10:17
*** esikachev has quit IRC10:17
*** mrmartin has quit IRC10:18
*** vtapia_ has quit IRC10:19
openstackgerritVitaly Gridnev proposed openstack-infra/project-config: [sahara] setup additional jobs for sahara-dashboard
*** sdague has joined #openstack-infra10:20
*** vtapia_ has joined #openstack-infra10:21
dimsfungi : good morning, please join us when you get in :) #openstack-release10:22
*** daemontool has joined #openstack-infra10:22
*** yaume_ has joined #openstack-infra10:24
*** redixin has joined #openstack-infra10:24
*** yaume has quit IRC10:27
*** kmartin has quit IRC10:28
redixinHi all. *.css files are text/css right? So http servers should set Content-Type header to "text/css" right?10:29
redixinOr I missed something? Like should serve .css files as "text/x-asm" for some reason?10:31
*** obondarev has joined #openstack-infra10:31
*** thorst has joined #openstack-infra10:32
*** _degorenko|afk is now known as degorenko10:32
*** thorst has quit IRC10:39
*** links has quit IRC10:39
openstackgerritVitaly Gridnev proposed openstack-infra/project-config: [sahara] setup additional jobs for sahara-dashboard
*** yamamoto has joined #openstack-infra10:42
openstackgerritMerged openstack/requirements: Bump gnocchiclient version to 2.2.0
*** nwkarsten has joined #openstack-infra10:43
*** abregman has quit IRC10:44
*** abregman has joined #openstack-infra10:44
*** xwizard has joined #openstack-infra10:44
*** Guest12876 is now known as tesseract10:47
*** tesseract is now known as Guest7403610:48
*** BobBall-1WOL is now known as BobBall10:49
*** andymaier has joined #openstack-infra10:53
*** alex_xu has quit IRC10:55
*** links has joined #openstack-infra10:56
*** kengo_sakai_mba has joined #openstack-infra10:56
*** alex_xu has joined #openstack-infra10:57
*** flongo has joined #openstack-infra10:58
*** jlanoux has quit IRC10:58
*** achanda has joined #openstack-infra10:58
*** nadya has quit IRC10:59
*** maishsk has quit IRC11:01
*** jlanoux has joined #openstack-infra11:03
*** pblaho has joined #openstack-infra11:03
*** achanda has quit IRC11:03
*** maishsk has joined #openstack-infra11:03
*** yamamoto has quit IRC11:04
*** jlanoux_ has joined #openstack-infra11:05
*** yamamoto has joined #openstack-infra11:06
*** jlanoux has quit IRC11:07
*** thorst has joined #openstack-infra11:11
*** thorst_ has joined #openstack-infra11:11
*** cdent has joined #openstack-infra11:14
*** thorst has quit IRC11:15
*** kengo_sakai_mba has quit IRC11:15
*** gildub has joined #openstack-infra11:17
*** tpsilva has joined #openstack-infra11:22
*** Qiming has joined #openstack-infra11:28
*** ramineni_1 has left #openstack-infra11:29
openstackgerritMerged openstack-infra/tripleo-ci: Add execute permissions to
*** fhubik has joined #openstack-infra11:33
*** ldnunes has joined #openstack-infra11:35
*** vgridnev has quit IRC11:39
*** alex_xu has quit IRC11:41
*** nadya has joined #openstack-infra11:43
*** alex_xu has joined #openstack-infra11:43
openstackgerritKiall Mac Innes proposed openstack-infra/project-config: Add designate-tempest-plugin to Designate DSVM jobs
*** _amrith_ is now known as amrith11:44
Kiallany project-config cores around? Hoping to land ^ quickly, as I'm gone on holidays soon and want to get everything ready + passing for core's to +A :)11:44
AJaegerKiall: I think it's not sufficient, see my comment for a quick review11:46
KiallAJaeger: our DSVM plugin is already enabled + used etc - Is that what you mean?11:47
Kiall etc manages to suck it in11:47
AJaegerI miss a line like:11:47
Kiall(It's old and crufty and in need of some TLC to cleanup)11:47
AJaegerexport DEVSTACK_LOCAL_CONFIG="enable_plugin zaqar git://"11:48
AJaegerin designate.yaml11:48
AJaegerright now you enable the designate one - not sure whether that works with your new plugin as well.11:48
AJaegerexport DEVSTACK_LOCAL_CONFIG="enable_plugin designate git://"11:48
AJaegerThat'S what you have - better ask sdague or somebody else about this.11:49
KiallOh, right - it's not needed for this.11:49
* AJaeger doesn't feel confident to +2 that quickly11:49
Kiall handles adding the tempest plugin11:49
Kiallenable_plugin is for devstack plugins, rather than tempest ones :)11:49
sdagueKiall: seems fine11:50
Kiallthanks sdague :) Once we move our existing tests over, I plan to spend some time cleaning the organic mess that is the designate DSVM jobs up ;)11:51
*** mrmartin has joined #openstack-infra11:51
KiallAll the plugin supports things have been gaining means less hacks needed to wedge designate in ;)11:52
*** maishsk has quit IRC11:53
*** moravec has quit IRC11:53
*** keedya has joined #openstack-infra11:55
*** cdent has quit IRC11:55
*** gordc has joined #openstack-infra11:55
*** weshay has joined #openstack-infra11:56
*** maishsk has joined #openstack-infra11:57
*** cdent has joined #openstack-infra11:59
*** andymaier has quit IRC11:59
*** BobBall_ has joined #openstack-infra12:01
*** BobBall_ is now known as Guest3075812:01
openstackgerritMerged openstack-infra/project-config: Add designate-tempest-plugin to Designate DSVM jobs
*** dingyichen has quit IRC12:03
jheskethFYI I've reset all of the channels that statusbot manages by pulling out the status's from the logs12:03
jheskethAJaeger: ^12:03
*** vgridnev has joined #openstack-infra12:03
*** aysyd has joined #openstack-infra12:04
*** vincentll has quit IRC12:04
*** trown|outtypewww is now known as trown12:04
*** BobBall has quit IRC12:04
*** vincentll has joined #openstack-infra12:05
*** wfoster is now known as wfoster_afk12:05
*** Guest30758 has quit IRC12:05
*** BobBall has joined #openstack-infra12:06
*** dingyichen has joined #openstack-infra12:06
*** BobBall is now known as Guest3417712:06
*** jhesketh has quit IRC12:07
*** achanda has joined #openstack-infra12:07
*** jhesketh has joined #openstack-infra12:07
openstackgerritNate Johnston proposed openstack-infra/irc-meetings: Add Nate Johnston as neutron-qos co-chair
krotscheckvkramskikh: I have no plans to move back at this point, unless there's a significant reason to do so. Node 4 is LTS, under support, and will remain so for at least another year, which node 012 is not.12:11
*** kengo_sakai_mba has joined #openstack-infra12:12
*** baoli has joined #openstack-infra12:13
*** moravec has joined #openstack-infra12:13
*** baoli_ has joined #openstack-infra12:13
*** achanda has quit IRC12:15
*** moravec has quit IRC12:16
*** kengo_sakai_mba has quit IRC12:17
*** baoli has quit IRC12:18
*** dprince has joined #openstack-infra12:19
krotscheckjtomasek: Yeah, let's chat here. Probably more relevant.12:20
jtomasekkrotscheck: ok12:20
krotscheckFirst time running npm install it got Killed. Odd that12:21
krotscheckAnd again.12:21
jtomasekkrotscheck: you mean npm install errors?12:21
krotscheckHrmYeah, need more memory on the instance, sec.12:22
jtomasekkrotscheck: ah12:22
*** edmondsw has joined #openstack-infra12:23
*** doug-fish has joined #openstack-infra12:26
*** akshai has joined #openstack-infra12:26
*** salv-orl_ has quit IRC12:28
*** |-paul-| has joined #openstack-infra12:31
krotscheckjtomasek: Def have a repro. Found a couple of related github issues, may need to ditch shrinkwrap altogether.12:31
*** placko_ has quit IRC12:32
krotscheckjtomasek: As an unrelated comment, patternfly looks pretty slick12:33
*** cbader has quit IRC12:33
*** pradk has joined #openstack-infra12:34
*** dizquierdo has quit IRC12:34
*** kgiusti has joined #openstack-infra12:34
jtomasekkrotscheck: nice, thank you! yeah, it is built on top of bootstrap12:34
jtomasekkrotscheck: so it seems the problem is again related to node + npm versions?12:35
AJaegerjhesketh: Wow, thanks a lot!12:36
*** nwkarsten has quit IRC12:36
*** nwkarsten has joined #openstack-infra12:36
jheskethno worries... if it happens again I'll write a tool for it12:36
*** lucasagomes has quit IRC12:37
mescanef_hello. please review project config change:
krotscheckjtomasek: Ok, so it appears that the issue is in patternfly - their npm-shrinkwrap file contains all of the dev-dependencies, which shrinkwrap run in the parent module will pick up and yell about.12:38
mescanef_AJaeger: :+)12:38
*** lucasagomes has joined #openstack-infra12:38
*** kzaitsev_mb has quit IRC12:38
jheskethtobiash: sorry I was preoccupied when you pinged earlier. Do you have any more detail or a copy of the configs?12:38
krotscheckjtomasek I'm guessing that patternfly generates that shrinkwrap file using --dev?12:39
jtomasekkrotscheck: no idea. I can try to ask at #patternfly12:39
*** salv-orlando has joined #openstack-infra12:40
*** nwkarsten has quit IRC12:40
krotscheckjtomasek: Yeah, lemme check something real quick. We may get away with using the _actual_ npm lts version.12:40
tobiashjhesketh: just bisected to the root cause:
jtomasekkrotscheck: nice12:41
tobiashjhesketh: but I don't have a clue yet, how this breaks12:41
*** erlon has joined #openstack-infra12:42
*** cbader has joined #openstack-infra12:42
*** doug-fish has quit IRC12:42
jheskethtobiash: do you have a paste of your layout?12:42
pabelangersc`: thanks for the update12:42
*** doug-fish has joined #openstack-infra12:42
*** ramishra has quit IRC12:44
*** aysyd has quit IRC12:44
openstackgerritNisha Agarwal proposed openstack/diskimage-builder: Install proliantutils in IPA's virtualenv
*** rfolco has joined #openstack-infra12:45
*** ramishra has joined #openstack-infra12:45
krotscheckjtomasek: NO dice, asking over in #patternfly and submitting an issue against the project.12:45
tobiashwhen pushing a change to testing/foo into review, the check pipeline is entered (without doing real builds, but with invoking zuul-merger and posting the start message to gerrit)12:45
*** lucasagomes has quit IRC12:45
pabelangeralso, morning12:45
tobiashjhesketh: before that change, all is ok12:46
tobiashbut I didn't understand this patch yet12:46
jheskethtobiash: hmm .. what happens if you have a real job rather than 'noop' (which is a builtin exception)12:46
*** lucasagomes has joined #openstack-infra12:46
tobiashjhesketh: the noop was just for test case simplification12:47
*** doug-fish has quit IRC12:47
*** sigmavirus24_awa is now known as sigmavirus2412:47
*** achanda has joined #openstack-infra12:47
tobiashwith a real build it posts the start message, invokes zuul-cloner and does nothing after that12:47
tobiashalso voting isn't done in this case12:48
*** piet has joined #openstack-infra12:50
*** aysyd has joined #openstack-infra12:50
jtomasekkrotscheck: ok, thanks a lot12:50
*** julim has joined #openstack-infra12:51
*** annegentle has joined #openstack-infra12:51
jheskethtobiash: and it works before 207094?12:51
*** rlandy has joined #openstack-infra12:52
*** burgerk has joined #openstack-infra12:52
krotscheckjtomasek I think I have a fix though12:53
*** mrmartin has quit IRC12:53
*** kengo_sakai has quit IRC12:54
tobiashjhesketh: yes, did a git bisect12:54
*** doug-fish has joined #openstack-infra12:54
jtomasekkrotscheck: that would be great, because updating patternfly to a version which would eventually include the fix is a bit problematic12:55
*** kzaitsev_mb has joined #openstack-infra12:55
tobiashjhesketh: this patch definitely does something related to this (stuff with project names), but I was unable to find the issue with this yet12:56
*** doug-fis_ has joined #openstack-infra12:56
*** kengo_sakai_mba has joined #openstack-infra12:56
jheskethtobiash: you aren't trying to enqueue something with a depends-on?12:56
*** akshai has quit IRC12:56
jtomasekkrotscheck: I tried to install latest version as part of the debugging and now it has a dependency on globally installed bower, so that would probably be another problem at the gate12:57
*** akshai has joined #openstack-infra12:57
krotscheckjtomasek: Urm, what? Globally installed bower?12:57
tobiashjhesketh: happens to any patch regardless of the depends-on in the commit message12:57
krotscheckjtomasek: That seems like a thing you can easily patch for them, no?12:57
jheskethtobiash: any notable differences between the logs before/after the commit?12:57
*** achanda has quit IRC12:58
jtomasekkrotscheck: this is the issue treeview is a patternfly dependency12:59
*** doug-fish has quit IRC12:59
jtomasekkrotscheck: it should not be a problem to fix I think12:59
*** keedya has quit IRC12:59
*** mrmartin has joined #openstack-infra13:00
*** keedya has joined #openstack-infra13:00
*** kengo_sakai_mba has quit IRC13:00
*** rodrigods has quit IRC13:01
*** rodrigods has joined #openstack-infra13:01
*** |-paul-| has quit IRC13:02
*** rossella_s has quit IRC13:03
*** nwkarsten has joined #openstack-infra13:03
*** rossella_s has joined #openstack-infra13:03
*** nwkarste_ has joined #openstack-infra13:04
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Call npm prune before npm shrinkwrap
krotscheckjtomasek: ^^13:05
*** Guest34177 is now known as BobBall13:05
tobiashjhesketh: found a difference in scheduler.addChange13:07
tobiashduring debug13:07
*** nwkarsten has quit IRC13:07
jtomasekkrotscheck: perfect! thanks again!13:07
*** achanda has joined #openstack-infra13:08
*** dingyichen has quit IRC13:08
tobiashhope that makes the runtime difference clear13:10
krotscheckjtomasek: No prob, that's why I'm here :)13:10
*** xarses has quit IRC13:10
*** eric_lopez has quit IRC13:10
*** Goneri has joined #openstack-infra13:10
*** flongo has quit IRC13:10
*** amitgandhinz has joined #openstack-infra13:11
*** eric_lopez has joined #openstack-infra13:11
*** flongo has joined #openstack-infra13:11
*** achanda has quit IRC13:13
tobiashjhesketh: seems like getChangeQueue causes the issue13:13
tobiashit no longer returns an empty change queue for the project not part of the pipeline13:13
*** akshai has quit IRC13:14
*** xarses has joined #openstack-infra13:15
tobiashthis causes the change to enter any change queue and processing until zuul notices that there are no builds to run13:15
*** xarses has quit IRC13:15
*** xarses has joined #openstack-infra13:15
openstackgerritMerged openstack-infra/project-config: Ironic grenade jobs do not work until after Mitaka
*** esikachev has joined #openstack-infra13:16
tobiashjhesketh: I suspect line 1804 causes the issue13:17
*** esker has quit IRC13:17
openstackgerritMerged openstack-infra/project-config: Add translation-jobs to horizon-cisco-ui
*** daemontool has quit IRC13:17
*** mrmartin has quit IRC13:19
*** _ari_ has joined #openstack-infra13:19
tobiashjhesketh: yes, confirmed, if I revert this line then this issue is gone, but I don't know if this would break the foreign project functionality13:19
openstackgerritMerged openstack-infra/project-config: Add requirements job to horizon-cisco-ui
openstackgerritMerged openstack-infra/project-config: Keystone Experimental patch_cover gate job
*** psilvad has joined #openstack-infra13:20
*** tiswanso has joined #openstack-infra13:20
*** moravec has joined #openstack-infra13:20
jheskethtobiash: hmm so I'm still confused what the actual bug is in terms of functionality.. are you testing this against review.o.o or another gerrit?13:21
*** rcarrillocruz has quit IRC13:22
*** oanson has quit IRC13:22
*** doug-fis_ has quit IRC13:23
*** akshai has joined #openstack-infra13:23
tobiashjhesketh: my own gerrit13:23
jheskethtobiash: what version of gerrit?13:23
*** rcarrillocruz has joined #openstack-infra13:25
jheskethtobiash: and the layout example you gave.. are you trying to enqueue into the check or test pipeline?13:25
jheskethor is it different13:25
*** ociuhandu has joined #openstack-infra13:25
tobiashjhesketh: I just upload the patch and expect that nothing is enqueued directly as foo is only in test pipeline13:25
jheskethtobiash: as in that's expected right?13:26
tobiashbut when I upload a patch it partly enters the check pipeline which is not configured for the project13:26
*** xyang1 has joined #openstack-infra13:26
*** esker has joined #openstack-infra13:26
*** mriedem has joined #openstack-infra13:26
openstackgerritMerged openstack-infra/project-config: Experimental gate-kolla-dsvm-build-centos-7-binary
tobiashjhesketh: yes, expected is that no pipeline gets entered13:26
jheskethtobiash: oh, I see what you're saying.. is it reporting back to gerrit too13:27
jheskethie it shouldn't be13:27
jheskethrighto, lets see if we can replicate that as a test13:28
*** daemontool has joined #openstack-infra13:28
tobiashthe real behaviour is that it enters the check pipeline, posts the start message on gerrit, calls zuul merger for every pipeline matching the requirements (this is pain for us due to limited bandwith) and after that stops processing13:28
*** matt-borland has joined #openstack-infra13:28
*** psilvad_ has joined #openstack-infra13:29
jheskethtobiash: okay13:30
*** cdent has quit IRC13:31
*** rcarrillocruz has quit IRC13:31
*** rcarrillocruz has joined #openstack-infra13:32
*** jamesmcarthur has joined #openstack-infra13:33
*** esker has quit IRC13:36
*** jsavak has joined #openstack-infra13:36
*** esker has joined #openstack-infra13:36
*** roxanaghe has joined #openstack-infra13:37
*** vincentll has quit IRC13:37
openstackgerritRob Cresswell proposed openstack-infra/project-config: Make the Django 1.9 tests voting on OpenStack Auth
*** annegentle has quit IRC13:38
*** moravec1 has joined #openstack-infra13:39
*** esikachev has quit IRC13:39
*** bdemers has quit IRC13:40
*** yamamoto has quit IRC13:40
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Update func tests for latest devstack flavors
*** moravec has quit IRC13:41
*** roxanaghe has quit IRC13:41
Shrewsjhesketh: can you take a peek at ^^^ ? Shade gate is broken with the latest devstack13:41
*** sc68cal has joined #openstack-infra13:42
jheskethShrews: sure, looking13:42
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/shade: Add support for Designate zones
*** nwkarsten has joined #openstack-infra13:44
rcarrillocruzShrews: second try for designate ^13:44
rcarrillocruzlet me know if we want to gate func tests on this13:44
rcarrillocruzif so, we'll need to fiddle with tests to enable devstack designate plugin13:44
*** kaisers has joined #openstack-infra13:44
*** bdemers has joined #openstack-infra13:45
jheskethShrews: change looks fine, do you have a link handy to the updated devstack for checking ?13:45
*** vincentll has joined #openstack-infra13:45
Shrewsrcarrillocruz: yes. we will not accept new APIs w/o func tests to know they work.13:45
*** nwkarst__ has joined #openstack-infra13:45
Shrewsjhesketh: i do not. i just remember seeing a ML email about it, and verified with my local devstack13:45
*** EricGonczer_ has joined #openstack-infra13:45
rcarrillocruzso new test with devstack, or piggy back in some and just add the plugin13:45
jheskethShrews: no worires, gimme a bit to poke at devstack then13:46
rcarrillocruzi lean towards a new one13:46
rcarrillocruznew test with devstack designate i mean13:46
*** links has quit IRC13:46
Shrewsrcarrillocruz: can you rephrase the question? not sure i understand the two options (i've never used designate)13:47
*** nwkarste_ has quit IRC13:47
*** nwkars___ has joined #openstack-infra13:48
jheskethShrews: lgtm :-)13:48
rcarrillocruzto run designate in devstack, you install a plugin13:48
jhesketh(so long as it passes)13:48
*** bdemers_ has joined #openstack-infra13:48
*** nwkars___ has quit IRC13:48
rcarrillocruzmy question is, should I add the plugin to the existing tests (nova, neutron , whatever)13:48
Shrewsjhesketh: thx13:48
rcarrillocruzor create a new job that will enable that plugin13:48
Shrewsrcarrillocruz: ah, now i see13:48
*** mrmartin has joined #openstack-infra13:48
rcarrillocruzi lean towards a new one to not clutter existing ones13:48
rcarrillocruzalthough that adds more jobs to the gate13:49
rcarrillocruznot sure13:49
*** nwkarste_ has joined #openstack-infra13:49
*** piet has quit IRC13:49
*** bdemers has quit IRC13:49
*** nwkarsten has quit IRC13:49
*** nwkarst__ has quit IRC13:50
Shrewsrcarrillocruz: i just enabled heat across all of the tests. i think we should just do the same for designate13:50
* rcarrillocruz goes check how designate folks set the plugin on their jobs13:50
Shrewsrcarrillocruz: otherwise we'll end up with 1 job per service, which is yuck13:50
rcarrillocruzpros and cons13:51
*** nwkarsten has joined #openstack-infra13:51
*** mrmartin has quit IRC13:52
*** nwkarst__ has joined #openstack-infra13:53
*** tonytan4ever has joined #openstack-infra13:53
*** nwkarste_ has quit IRC13:53
Shrewsrcarrillocruz: also, fyi, shade gate is broken until the change i just submitted merges13:54
*** zeih has joined #openstack-infra13:54
*** ddieterly has joined #openstack-infra13:54
*** EricGonczer_ has quit IRC13:55
*** mrmartin has joined #openstack-infra13:55
*** nwkarsten has quit IRC13:55
*** nwkarst__ has quit IRC13:55
*** piet has joined #openstack-infra13:56
*** ddieterly is now known as ddieterly[away]13:56
*** kzaitsev_mb has quit IRC13:56
rcarrillocruzcool thx13:56
*** nwkarsten has joined #openstack-infra13:56
fungijust a reminder, i'm not really around until probably 1800 utc today13:56
fungibriefly on from the passenger seat of a car13:57
fungibut about to drop back off soon13:57
*** yamamoto has joined #openstack-infra13:57
Shrewsfungi: unacceptable13:58
*** EricGonczer_ has joined #openstack-infra13:58
*** dizquierdo has joined #openstack-infra13:58
*** cdent has joined #openstack-infra13:59
krotscheckAnother patch working through some of the fallout from our npm upgrade, only needs one more core:
mrmartinWhat's wrong with bindep here: groups.o.o project13:59
pabelangermrmartin: seems wrong14:00
mrmartinbut what is wrong?14:00
mrmartinthe sudo?14:01
*** nwkarsten has quit IRC14:01
pabelangerat first glance, I would say sudo is wrong some how.14:01
mrmartinit was working before :)14:01
pabelangerI know we recently upgrade the JJB to use bindep14:01
*** yamahata has joined #openstack-infra14:01
fungimrmartin: i think our attempts to make the package installation quieter are overzealous and making that hard to debug14:02
fungii have a feeling apt-get install is erroring somehow but being silenced14:02
*** esikachev has joined #openstack-infra14:02
pabelangermrmartin: fungi I see the issue, we revoke-sudo before running bindep14:02
pabelangerlet me get a patch14:03
mrmartinwho can help to debug this?14:03
mrmartinthanks pabelanger14:03
fungipabelanger: oh, yep install-distro-packages is supposed to always be before revoke-sudo, if present14:03
*** yamamoto has quit IRC14:03
fungithat explains sudo looking for an askpass client14:03
fungididn't even dawn on me14:03
fungiso apt-get is never being called14:03
fungithat's why it was silent :/14:04
openstackgerritPaul Belanger proposed openstack-infra/project-config: revoke-sudo after bindep for groups jobs
fungigood eye14:04
pabelangermrmartin: fungi: ^ should fix it14:04
openstackgerritSean McGinnis proposed openstack/requirements: Update to os-brick 1.3.0
AJaegerpabelanger: LGTM, yeah...14:05
*** cloudtrainme has joined #openstack-infra14:05
*** wfoster_afk is now known as wfoster14:05
fungihopefully that14:05
fungi'll land quickly14:05
openstackgerritPaul Belanger proposed openstack-infra/project-config: Migrate openstackid jobs to ubuntu-trusty
*** esker has quit IRC14:06
pabelangerAlso fixed for openstackid jobs now ^14:06
pabelangerwhich hasn't migrated to ubuntu-trusty yet14:06
*** sdake has joined #openstack-infra14:07
*** thomasem_ is now known as thomasem14:08
openstackgerritZhipeng Huang proposed openstack-infra/project-config: add new project Coupler to openstack repo
*** knikolla has joined #openstack-infra14:10
*** akaszuba has quit IRC14:10
openstackgerritMerged openstack-infra/project-config: Give project teams access to approve stable/mitaka changes
*** nwkarsten has joined #openstack-infra14:11
*** thiagop has joined #openstack-infra14:11
*** sdake has quit IRC14:12
*** rbrndt_ has joined #openstack-infra14:13
*** mylu has joined #openstack-infra14:17
*** julim has quit IRC14:19
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Projects without jobs still enter pipeline
jheskethtobiash: I think ^ proves your bug.. The next part (and probably tomorrows job) is f14:20
jheskethinding the fix14:20
jheskethtobiash: when I remove the check pipeline from that layout the tests pass14:20
*** thorst_ has quit IRC14:21
*** sdake has joined #openstack-infra14:21
*** jamesmcarthur has quit IRC14:21
*** julim has joined #openstack-infra14:22
*** gildub has quit IRC14:22
*** thorst_ has joined #openstack-infra14:22
*** baoli_ has quit IRC14:26
*** thorst__ has joined #openstack-infra14:26
vponomaryovwhat happened to CI? queues kind of freezed14:27
*** thorst_ has quit IRC14:27
pabelangerzuul recalculating the world14:27
openstackgerritThomas Morin proposed openstack-infra/project-config: Add tempest-dsvm-networking-bgpvpn-bagpipe jobs
vponomaryovpabelanger: is it long operation?14:29
*** yamamoto has joined #openstack-infra14:29
*** nwkarsten has quit IRC14:30
openstackgerritMerged openstack/requirements: Require reno 1.6.2 or newer
*** baoli has joined #openstack-infra14:30
*** zz_dimtruck is now known as dimtruck14:30
*** matrohon has quit IRC14:31
*** matrohon has joined #openstack-infra14:31
pabelangervponomaryov: can take a few minutes to complete.14:31
*** Qiming_ has joined #openstack-infra14:32
vponomaryovpabelanger: ok, ty14:32
*** Qiming has quit IRC14:32
*** psanchez has quit IRC14:33
*** eharney has joined #openstack-infra14:34
*** jamesmcarthur has joined #openstack-infra14:35
*** thorst__ has quit IRC14:37
*** mylu has quit IRC14:37
*** thorst_ has joined #openstack-infra14:41
*** mylu has joined #openstack-infra14:42
*** ddieterly has joined #openstack-infra14:42
*** rkukura has quit IRC14:42
*** psilvad_ has quit IRC14:44
*** moravec1 has quit IRC14:46
*** psilvad_ has joined #openstack-infra14:47
*** psilvad_ has quit IRC14:48
*** nwkarsten has joined #openstack-infra14:48
*** nmagnezi has quit IRC14:49
*** esikachev has quit IRC14:49
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Create a test flavor for the pingtest VM
*** jcooley_ has joined #openstack-infra14:51
*** salv-orl_ has joined #openstack-infra14:52
*** jcooley_ has quit IRC14:52
*** kzaitsev_mb has joined #openstack-infra14:52
*** maishsk has quit IRC14:53
*** salv-orl_ has quit IRC14:53
*** nwkarsten has quit IRC14:53
*** esikachev has joined #openstack-infra14:54
*** salv-orlando has quit IRC14:54
*** thorst__ has joined #openstack-infra14:54
*** psilvad_ has joined #openstack-infra14:55
*** sigmavirus24 is now known as sigmavirus24_awa14:55
*** sigmavirus24_awa is now known as sigmavirus2414:55
*** ramishra has quit IRC14:55
*** tphummel has joined #openstack-infra14:56
*** nwkarsten has joined #openstack-infra14:57
*** ramishra has joined #openstack-infra14:57
*** Qiming_ has quit IRC14:57
*** thorst_ has quit IRC14:58
*** kzaitsev_mb has quit IRC14:58
*** regXboi has joined #openstack-infra14:58
*** Qiming has joined #openstack-infra14:59
*** woodster_ has joined #openstack-infra15:00
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Fix lock range, again
*** thorst__ is now known as thorst________15:00
*** tiswanso has quit IRC15:01
*** EricGonczer_ has quit IRC15:02
*** tiswanso has joined #openstack-infra15:03
*** nwkarsten has quit IRC15:04
*** EricGonczer_ has joined #openstack-infra15:04
*** amitgandhinz has quit IRC15:04
vkramskikhit seems node.js upgrade made gate-fuel-ui-npm-run-lint always fail in a very strange way: it seems it fails on "cat npm-shrinkwrap.json" with error "cat: write error: Resource temporarily unavailable"15:05
vkramskikhdoes anybody have any idea why it happens?15:06
vkramskikhkrotscheck: do you know why we need to create shrinkwrap.json and then cat it?15:06
*** nwkarsten has joined #openstack-infra15:07
*** ajmiller has joined #openstack-infra15:07
clarkbonly the ubuntu-trusty upload worked against rax-ord last night. The remaining four uploads I queued all failed15:07
vkramskikhcan I modify this job to avoid generating shrinkwrap if it already exists? (fuel-ui repo has it)15:07
clarkbbut thats ok, all of the other regions have the images we need and rax-ord can catch upafter we switch servers15:07
*** j_king_ is now known as j_king15:07
*** thorst________ is now known as thorst_15:08
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Temporarily disable mistral on the undercloud
*** fitoduarte has quit IRC15:10
*** Guest74036 has quit IRC15:10
*** nwkarsten has quit IRC15:11
vkramskikhtsufiev: maybe you know about shrinkwrap?15:11
*** moravec has joined #openstack-infra15:12
openstackgerritPaul Belanger proposed openstack-infra/project-config: Use noop-jobs for openstack/fuel-ui
*** rkukura has joined #openstack-infra15:13
tsufievvkramskikh, unfortunately no15:14
*** maishsk has joined #openstack-infra15:14
*** nwkarste_ has joined #openstack-infra15:16
tsufievkrotscheck, I wonder how this patch manages to hit npm-run issues again and again :)15:16
tsufieva jinxed patch15:16
tsufievI think it's 3rd failure in a row15:17
vkramskikhit seems there is failure on downloading phantomjs15:17
tsufievah, that's true15:18
tsufievstill it's a mystery to me15:18
tsufievwhat's so special about it?15:18
clarkbits hosted on bitbucket iirc and bitbucket says go away15:18
clarkbwhich is why we don't allow external link packages for python packages15:19
*** baoli has quit IRC15:19
openstackgerritMerged openstack-infra/tripleo-ci: Use split cirros image for ping test
*** akaszuba has joined #openstack-infra15:20
*** nwkarste_ has quit IRC15:20
tsufievclarkb, perhaps part of our distributed system was banned on bitbucket?15:21
openstackgerritMerged openstack-infra/project-config: Migrate rally from bare-precise
*** amitgandhinz has joined #openstack-infra15:21
*** nwkarsten has joined #openstack-infra15:21
openstackgerritMerged openstack-infra/project-config: Migrate rally to centos-7
openstackgerritMerged openstack-infra/project-config: revoke-sudo after bindep for groups jobs
clarkbtsufiev: I think this is a general issue and not one on our end15:22
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Revert "Switch npm-lint to non-voting for fuel-ui"
clarkbtsufiev: we don't really reuse IPs that aggressively15:22
*** zeih has quit IRC15:22
clarkbnor do we talk to bitbucket15:22
AJaegerigorbelikov: See
rcarrillocruzlong time i don't try to simulate a devstack-gate locally15:23
rcarrillocruzand got a freeze15:23
rcarrillocruzmy memory told me i hit this some point15:23
rcarrillocruzclarkb: :-)15:23
*** zeih has joined #openstack-infra15:23
clarkbrcarrillocruz: apt-get is expecting input, use the script15:23
rcarrillocruzwill give it a try again in a sec15:23
clarkbrcarrillocruz: the fix is to close stdin15:23
AJaegervkramskikh: see also
*** achanda has joined #openstack-infra15:24
openstackgerritAliaksandr Buhayeu proposed openstack-infra/jenkins-job-builder: Clarify exception message for missed trigger properties
*** ddieterly is now known as ddieterly[away]15:25
*** alex_xu has quit IRC15:25
AJaegerpabelanger: is not what we do normally15:26
AJaegerpabelanger: I think my -2 was too eager15:26
*** yamahata has quit IRC15:26
*** kmartin has joined #openstack-infra15:27
AJaegerpabelanger: if this would be in the gate queue, we would have a real problem since now all jobs having noop in gate would be in shared queue. For check it might be harmless - but still, I don't think we need this and we have other examples of the pattern.15:27
*** cloudtrainme has quit IRC15:28
*** zeih has quit IRC15:28
*** Swami has joined #openstack-infra15:28
*** alex_xu has joined #openstack-infra15:29
*** armax has joined #openstack-infra15:30
igorbelikovAJaeger: thanks, and thanks for workflow-1, It’ll be better to wait till fuel-ui passes npm-lint15:31
*** cloudtrainme has joined #openstack-infra15:31
*** dangers_away is now known as dangers15:32
openstackgerritMerged openstack-infra/project-config: Allow fuel-ui-core to merge into stable branches
vgridnevAJaeger, is it true that python34 job cannot be added as non-voting?15:32
AJaegerigorbelikov: please tell me in the review once it's fine15:32
AJaegervgridnev: sure, it can.15:32
AJaegervgridnev: which review?15:33
*** tphummel has quit IRC15:33
vgridnevthis one15:33
openstackgerritMerged openstack-infra/project-config: Watcher multinode - disable plugin in subnode
vikramAJaeger: Have doubt about your comment on
igorbelikovAJaeger: sure15:33
AJaegervgridnev: that one adds the template and *also* a single job15:33
AJaegerthat double, use either option. For a non-voting, use the single job and not the template. For voting the template and not the job15:34
vikramAJaeger: This has two branches, do you really want to import stable/mitaka? I suggest to delete it instead - it will only confuse folks later --- You would like me to delete the stable/mitaka branch?15:34
AJaegerthe template adds the job to both check and gate and we do not have non-voting jobs in gate15:34
pabelangerAJaeger: fair enough, I can abandon it.  I just noticed it in passing, since no other job had noop only gate15:34
*** Nikolay_St has quit IRC15:34
AJaegervikram: yes, my suggestion is to delete that branch. Once a repo is imported, branch deletion is a manual step.15:35
AJaegerpabelanger: there should be some - I remember approving some in the past;)15:35
vikramAJaeger: Ok.. Will do15:35
*** wfoster has quit IRC15:35
AJaegervikram: since you have not done a release for mitaka from that repo, I suggest to remove it.15:35
*** jsavak has quit IRC15:35
*** links has joined #openstack-infra15:36
*** sc68cal has quit IRC15:36
vikramAJaeger: Ok got it15:36
pabelangerAJaeger: okay, I'll abandon based on your comments.15:36
vikramAJaeger: Done!15:36
openstackgerritVitaly Kramskikh proposed openstack-infra/project-config: Do not run npm shrinkwrap if npm-shrinkwrap.json already exists
*** mtanino has joined #openstack-infra15:37
openstackgerritVitaly Gridnev proposed openstack-infra/project-config: [sahara] setup additional jobs for sahara-dashboard
*** achanda has quit IRC15:37
*** ddieterly[away] is now known as ddieterly15:37
fungiclarkb: rcarrillocruz: there are also apt-get command-line options that tell it to run noninteractively15:38
clarkbfungi: iirc they stopped working at some point15:38
*** nelsnelson has joined #openstack-infra15:38
clarkbfungi: we never noticed with jenkins because it doesn't give it a working stdin, but when you start from a bash shell on your local box it does?15:39
*** jcooley_ has joined #openstack-infra15:39
openstackgerritEmilien Macchi proposed openstack-infra/project-config: Implement periodic job to update Puppet OpenStack constraints
*** nelsnels_ has quit IRC15:39
openstackgerritVitaly Kramskikh proposed openstack-infra/project-config: Do not run npm shrinkwrap if npm-shrinkwrap.json already exists
fungiclarkb: more likely someone neglected to add the right options to some new apt-get call they introduced15:40
fungii've never seen those options fail to be respected15:40
*** wfoster has joined #openstack-infra15:40
*** vincentll has quit IRC15:40
EmilienMpabelanger: I commented on - please re-consider your vote15:40
*** jsavak has joined #openstack-infra15:40
*** Qiming has quit IRC15:41
pabelangerEmilienM: no -1 from me, just a note about the history15:42
clarkbfungi: ya its possible we just don't know what the entire list of required flags are or if they changed sometime in the last 4 years or so didn't notice because it works in a jenkins env15:42
EmilienMpabelanger: if you ready my commit message and the code I'm adding, what I'm doing and what you did 6 months ago are 2 very different things15:42
*** mikelk has quit IRC15:43
anteayattx: just following up on the merge tag conversation in backscroll, it looks like searchlight: 302579 zaqar: 302580 and congress: 302558 have yet to merge their patches15:43
*** afranc has joined #openstack-infra15:44
*** ddieterly is now known as ddieterly[away]15:44
pabelangerEmilienM: Right, I think the issue was with the automation of it over the functionality.  Either way, going to hold off on voting until other infra-root review the patch.15:44
fungiclarkb: back when i was remotely triggering devstack jobs by hand, i had to explicitly close stdin for other reasons that caused it to hang, so i don't think it's particularly new15:44
clarkbfungi: ah15:44
EmilienMpabelanger: fair enough, thanks15:45
fungiclarkb: this was several years ago at least that it was necessary15:45
pabelangerEmilienM: FWIW: I'd like to see some automation too just need to make sure everybody is on board with it15:45
EmilienMpabelanger: me too, moreover, other projects already have this same automation.15:46
*** kzaitsev_mb has joined #openstack-infra15:46
*** ddieterly[away] is now known as ddieterly15:46
pabelangerEmilienM: /me nods15:47
openstackgerritZhipeng Huang proposed openstack-infra/project-config: add new project Coupler to openstack repo
*** jaosorior has quit IRC15:47
*** scheuran has quit IRC15:47
*** jaosorior has joined #openstack-infra15:47
*** jsavak has quit IRC15:47
*** jsavak has joined #openstack-infra15:48
*** links has quit IRC15:48
*** ramishra has quit IRC15:48
*** baoli has joined #openstack-infra15:49
*** matrohon has quit IRC15:49
AJaegerpabelanger, fungi: We have several of these proposal jobs like EmilienM's - we should discuss these again...15:49
pabelangerclarkb: fungi: only review left to migrate away from bare-precise (minus our jobs)
AJaegerI meant: 3 open reviews15:50
clarkbpabelanger: looking15:50
pabelangerAJaeger: agreed15:50
EmilienMAJaeger: yes, please15:50
* AJaeger puts it on agenda of next meeting and will mark changes as WIP for now...15:51
EmilienMAJaeger: do we need to wait meeting?15:51
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add experimental bindep JJB for kolla
EmilienMAJaeger: I mean, we can discuss it on Gerrit15:51
clarkbpabelanger: I will actually defer to fungi on that one as he is most familiar with the state of things there15:51
pabelangerclarkb: WFM15:51
pabelangerso only kolla is left for devstack-centos715:52
pabelangertesting some experimental jobs today to confirm it will be a smooth cut over15:53
AJaegerEmilienM: this needs some broader discussion and fungi's signoff IMHO.15:53
*** pcaruana has quit IRC15:53
anteayajhesketh: it happens quite frequently that a root needs to dig in statusbot logs to find channel topics and reset them15:53
*** derekh has quit IRC15:53
pabelangerAJaeger: EmilienM: Could be a fishbowl session at summit too15:53
anteayajhesketh: if you are inclined to write a tool I think it would be welcomed15:53
EmilienMAJaeger: I have no doubt about it15:54
*** maishsk has quit IRC15:54
EmilienMpabelanger: can you also re-visit please? AFIK release is done15:54
AJaegerpabelanger: feel free to add to etherpad ;)15:54
*** mrmartin has quit IRC15:55
EmilienMAJaeger, pabelanger: I don't understand why we would need to wait Summit for that, other projects already have automation that sync the requirements, and that's the purpose of my patch15:55
*** moravec has quit IRC15:55
pabelangerEmilienM: sure, let me confirm that with release team15:55
AJaegerEmilienM: those jobs all run on the proposal slae that has access of passwords.15:57
AJaegerThe jobs we currently do that should have been reviewed for security implications.15:57
openstackgerritZara proposed openstack-infra/python-storyboardclient: WIP Proof of Concept Storyboard Updating from Gerrit
AJaegerEmilienM: this is a broader security discussion on what to run.15:58
*** sridhar_ram has joined #openstack-infra15:58
AJaegerEmilienM: Feel free to discuss with fungi and the rest of the team, if you're unhappy with my -1. For now I want this to be a team discussion and like to have a clear guideline on how I shall review this.15:59
anteayaAJaeger: thank you for encouraging the discussion, can't join in now but thank you15:59
*** apoorvad has joined #openstack-infra16:00
EmilienMAJaeger: sure, no worries about -1. I did not have full background16:00
*** moravec1 has joined #openstack-infra16:00
*** nmagnezi has joined #openstack-infra16:01
AJaegerWIP ;)16:01
*** xarses has quit IRC16:01
*** xarses has joined #openstack-infra16:02
*** mtanino has quit IRC16:02
*** rvasilets has joined #openstack-infra16:02
*** armax has quit IRC16:02
tristanCit does sounds risky to have a jenkins jobs submit change... what if those new changes trigger the same job ? :)16:02
*** armax has joined #openstack-infra16:02
*** mtanino has joined #openstack-infra16:03
*** avarner_ has joined #openstack-infra16:03
AJaegertristanC: jenkins *proposes* changes and thus for sure will trigger new jobs16:03
AJaegerBut that's a proposal, so no commit16:03
*** tphummel has joined #openstack-infra16:04
*** angdraug has joined #openstack-infra16:04
openstackgerritMerged openstack/requirements: Updated from generate-constraints
*** jamesmcarthur has quit IRC16:04
openstackgerritMerged openstack/requirements: Bump python-cinderclient to 1.6.0
*** julim has quit IRC16:05
*** yamamoto has quit IRC16:06
*** annegent_ has joined #openstack-infra16:08
*** akshai has quit IRC16:08
*** yamahata has joined #openstack-infra16:08
fungii'll try to take a look, but still very patchy cell signal here (just came out of a 15+ minute dead zone)16:08
*** e0ne has quit IRC16:08
*** e0ne has joined #openstack-infra16:09
*** nwkarsten has quit IRC16:10
krotschecktsufiev Checking16:12
krotschecktsufiev Oh16:12
krotschecktsufiev Yes, have encountered that. Lemme tell ya16:12
krotschecktsufiev, clarkb: Turns out that the binary for phantomjs is hosted on bintray, and it tends to hit over quota issues. The recommended fix they advocate is "Hey go use this mirror in china", which is... LOLworthy.16:13
*** maishsk has joined #openstack-infra16:13
*** jsavak has quit IRC16:14
krotschecktsufiev, clarkb: My recommended fix is: Stop using phantomjs. it's not a real browser, and we run chrome and firefox in the gate.16:14
*** fhubik has quit IRC16:14
*** dtantsur is now known as dtantsur|afk16:14
*** jsavak has joined #openstack-infra16:14
matt-borlandkrotscheck, we can certainly modify the config to run chrome/ff if needed.  Also, seems npm troubles have lifted this morning, may be ephemeral though.16:15
matt-borlandnah, not lifted :)16:15
* matt-borland strikes that last statement16:16
clarkbkrotscheck: any idea why they don't just host on npm propre?16:16
clarkbkrotscheck: solve the issue and everyone can move on with life?16:16
krotscheckmatt-borland: Yep, def not lifted, but we're no node4 and npm2 now, so that should be shaking out.16:16
matt-borlandkrotscheck, thank you16:16
krotscheckclarkb: I think it's because phantomJS is not actually a node thing.16:17
krotscheckclarkb: It's just used a lot by node things.16:17
*** jamesmcarthur has joined #openstack-infra16:17
*** ramishra has joined #openstack-infra16:17
clarkbbut if npm can install it then its fair game to be hosted by npm16:17
*** mylu has quit IRC16:19
*** jlanoux_ has quit IRC16:19
*** EricGonczer_ has quit IRC16:20
openstackgerritMerged openstack/requirements: Bump neutron-lib to 0.0.3 minimum
*** mylu has joined #openstack-infra16:20
*** nwkarsten has joined #openstack-infra16:20
fungistopping for lunch, then planning to take over driving, but should be back online from a proper connection in another couple hours16:20
clarkbfungi: great we can work through nodepool switchover then16:20
*** vgridnev has quit IRC16:21
*** vgridnev has joined #openstack-infra16:21
*** mrmartin has joined #openstack-infra16:21
clarkbmy rough plan is nova shutdown old host, switch dns, double check firewalls update, start service on new server, work through alien cleanup16:21
clarkband I should make sure our ttl is 5 minutes, doing that now16:22
pabelangerclarkb: so, if you have time, I'd be interested to see which devstack-gate job we need to add for experimental ubuntu-xenial16:22
krotscheckclarkb: Sadly I don't own the phantomjs node module.16:22
pabelangerclarkb: after nodepool migration16:22
avarner_Have there been any changes in gerrit in the last 2 days? I can't access any code reviews now.16:22
*** avarner_ is now known as avarner16:22
clarkbkrotscheck: right, when we had this problem with pypi we went to the packages and asked them to change16:22
clarkbkrotscheck: those that didn't we stopped using16:22
*** ddieterly is now known as ddieterly[away]16:23
krotscheckclarkb: I did. They said no. My recommendation is to stop using phantomjs16:23
clarkbpabelanger: I would add base dsvm tempest job, neutron dsvm tempest job, postgres temest job, and grenade16:23
clarkbpabelanger: and add those jobs to devstack-gate, devstack, tempest, and grenade to start16:23
clarkbonce that is working we can add in multinode16:23
krotscheckclarkb: If I recall correctly the reason was *handwave handwave* it's hard *handwave* different binaries for different operating systems *handwave*16:24
*** trown is now known as trown|lunch16:24
pabelangeravarner: not yet, but changes are coming:
openstackgerritMerged openstack/requirements: Updated magnum client version to 2.0.0
openstackgerritMerged openstack/requirements: Bump zaqarclient and barbicanclient versions
*** EricGonczer_ has joined #openstack-infra16:25
*** jcoufal has quit IRC16:26
openstackgerritMerged openstack/requirements: Bump python-ironic-inspector-client to 1.5.0
openstackgerritMerged openstack/requirements: oslo.config>=3.9.0
clarkbkrotscheck: oh its shipping precompiled binaries ya that can make things a little trickier though not impossible16:26
openstackgerritMerged openstack/requirements: Require at least python-neutronclient 4.1.1
openstackgerritMerged openstack/requirements: Bump openstacksdk version to 0.8.4
tsufievkrotscheck, thank you for the info. Need to talk to r1charj0nes about that16:26
*** shashank_hegde has joined #openstack-infra16:27
clarkbnodepool dns is set to 5 minute ttls16:27
openstackgerritMerged openstack-infra/tripleo-ci: ssh less
krotschecktsufiev: There are build macros that make sure that the browsers are available and that xvfb is started and configured. LMK if you have any questions, you can use the jobs in javascript.yml as a template.16:29
*** jsavak has quit IRC16:29
*** bpokorny has joined #openstack-infra16:29
*** jsavak has joined #openstack-infra16:29
*** sridhar_ram has quit IRC16:32
openstackgerritCostin Galan proposed openstack/requirements: Add Windows as supported operating system
tsufievkrotscheck, sure. phantomjs dependency is r1chardj0nes' child, so - if we're going to remove it - I'd like first discuss it with him16:32
anteayaclarkb: looks like you are in a good place for the switchover when fungi is next online?16:32
clarkbanteaya: yes16:34
*** yaume_ has quit IRC16:34
clarkbI think my brain just exploded, you have to delete a file before renaming it in windows16:34
openstackgerritMerged openstack-infra/tripleo-ci: Fix image caching logic
*** moravec has joined #openstack-infra16:34
clarkboh the dest side16:34
anteayathat ws for the windows thing16:35
anteayaand yay on the you are in a good place thing for nodepool switchover16:36
*** mylu has quit IRC16:36
clarkbI wonder if three is a different way to do atomic rename type deals16:36
anteayaI have to go run errands in 20 minutes, I am out of food as I didn't plan on being home this week16:36
clarkbbecause not being able to do that is sadness16:36
*** david-lyle has quit IRC16:36
*** moravec2 has joined #openstack-infra16:36
anteayasorry I am not understanding16:37
anteayawhy do you have to rename windows files16:37
*** LaT0rtue has joined #openstack-infra16:37
*** Nikolay_St has joined #openstack-infra16:37
clarkbanteaya: the patch above for adding windows support to requirements does this16:37
*** moravec1 has quit IRC16:37
ociuhanduclarkb: it’s not a simple rename there, it’s trying to rename one file with the name of an already existing one16:37
clarkbanteaya: I was just reading it out of curiousity16:38
clarkbociuhandu: yes that is a very common pattern and iirc on linux you get an atomic rename on the same fs16:38
clarkbwhich is super handy16:38
anteayaclarkb: ah, now I understand16:38
anteayasorry thought it had something to do with nodepool16:38
*** moravec has quit IRC16:38
*** yaume has joined #openstack-infra16:39
*** lakshmiS has joined #openstack-infra16:39
zigoclarkb: Hi there!16:40
clarkbzigo: hello16:40
zigoclarkb: Could you review this trivial patch?
zigoclarkb: I believe that's the last one before we can actually build a first package.16:41
ociuhanduclarkb: I know, but on windows python throws an error that the target already exists. Does not look like a windows issue, rather the python implementation16:41
openstackgerritMerged openstack-infra/subunit2sql: Fix lock range, again
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add experimental devstack-jobs for ubuntu-xenial
*** mylu has joined #openstack-infra16:42
*** e0ne has quit IRC16:42
pabelangerclarkb: I believe that guess what you asked for ^16:42
*** shashank_hegde has quit IRC16:43
clarkbzigo: I am not sure if that is sufficient but we can try it (I do not know if jenkins creates a new login shell for each shell builder)16:43
clarkbzigo: is that prefered over newgrp?16:43
*** daemontool has quit IRC16:44
*** ddieterly[away] is now known as ddieterly16:46
*** xyang1 has quit IRC16:47
*** nwkarsten has quit IRC16:48
*** annegent_ has quit IRC16:49
*** amitgandhinz has quit IRC16:49
*** pblaho has quit IRC16:49
*** nwkarsten has joined #openstack-infra16:49
*** yaume has quit IRC16:49
*** annegentle has joined #openstack-infra16:49
*** pblaho has joined #openstack-infra16:51
*** rcernin has quit IRC16:52
*** moravec2 has quit IRC16:53
*** moravec has joined #openstack-infra16:53
*** julim has joined #openstack-infra16:53
*** nwkarsten has quit IRC16:53
*** darvon has quit IRC16:54
*** ZZelle_ has joined #openstack-infra16:55
*** pfallenop has quit IRC16:55
*** pfallenop has joined #openstack-infra16:56
*** darvon has joined #openstack-infra16:57
*** cloudtrainme has quit IRC16:57
*** akaszuba has quit IRC16:57
*** abregman has quit IRC16:57
*** angdraug has quit IRC16:58
*** dprince has quit IRC16:58
*** dprince has joined #openstack-infra16:59
openstackgerritArmando Migliaccio proposed openstack/requirements: Tempest 11.0.0 is released
*** stevebaker has quit IRC17:00
*** oanson has joined #openstack-infra17:01
*** jistr has quit IRC17:01
*** mylu has quit IRC17:02
*** xyang1 has joined #openstack-infra17:02
*** jamesmcarthur_ has joined #openstack-infra17:02
*** jamesmcarthur has quit IRC17:03
*** jamesmcarthur_ is now known as jamesmcarthur17:03
*** rossella_s has quit IRC17:03
*** mylu has joined #openstack-infra17:03
*** esker has joined #openstack-infra17:03
*** rossella_s has joined #openstack-infra17:03
*** nadya has quit IRC17:03
*** vgridnev has quit IRC17:04
*** ramishra has quit IRC17:05
*** yamahata has quit IRC17:06
*** david-lyle has joined #openstack-infra17:06
*** oanson has quit IRC17:06
*** yamamoto has joined #openstack-infra17:07
*** pcaruana has joined #openstack-infra17:08
*** ddieterly is now known as ddieterly[away]17:08
*** BobBall is now known as BobBall_AWOL17:10
*** david-lyle has quit IRC17:11
amrithhi folks, is the gerritbot down?17:11
*** salv-orlando has joined #openstack-infra17:12
*** mat128 has joined #openstack-infra17:13
*** ddieterly[away] is now known as ddieterly17:13
*** annegentle has quit IRC17:14
*** oanson has joined #openstack-infra17:14
*** MarkAtwood has joined #openstack-infra17:14
*** annegentle has joined #openstack-infra17:14
*** annegentle has quit IRC17:15
*** flepied1 has quit IRC17:15
*** annegentle has joined #openstack-infra17:15
*** salv-orlando has quit IRC17:15
*** yamamoto has quit IRC17:16
*** baoli has quit IRC17:16
*** migi has joined #openstack-infra17:16
*** baoli has joined #openstack-infra17:17
*** fitoduarte has joined #openstack-infra17:17
*** david-lyle has joined #openstack-infra17:17
*** jamesmcarthur has quit IRC17:18
*** briancline has quit IRC17:18
*** moravec has quit IRC17:19
*** xarses has quit IRC17:20
*** kushal has quit IRC17:20
*** cdent has quit IRC17:20
pabelangeramrith: openstackgerrit should we working17:20
*** fitoduarte has quit IRC17:20
*** briancline has joined #openstack-infra17:20
sc`i recently submitted a change and openstackgerrit didn't call it out17:20
sc`within the last 5 minutes or so17:20
sc`i saw it pop up in the zuul status page, so it's enqueued17:21
amrithpabelanger, thx17:21
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Dump end of logs to console.log on failure
sc`there it is again17:22
pabelangerglitch in the matrix17:22
sc`i knew i saw my black cat cross my doorway twice this morning17:22
*** cschwede has quit IRC17:23
*** baoli has quit IRC17:23
*** david-lyle has quit IRC17:23
*** amitgandhinz has joined #openstack-infra17:24
SpamapS hm17:24
*** esikachev has quit IRC17:24
SpamapSI can't seem to pip install tempest using constraints17:24
*** trown|lunch is now known as trown17:24
*** moravec has joined #openstack-infra17:25
SpamapSlifeless: ^ does that error make much sense to you?17:25
*** slagle has quit IRC17:26
*** dizquierdo has quit IRC17:26
SpamapSmordred: ^ pbr/pip thing confusing me17:26
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: TEST: Delete the overcloud when finished
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Test overcloud deletion with net-iso in periodic job
openstackgerritMerged openstack-infra/project-config: Switch ironic-inspector jobs to iPXE
*** flongo has quit IRC17:29
*** sambetts is now known as sambetts|afk17:30
SpamapSmtreinish: you around?17:31
SpamapSmtreinish: I'm trying to figure out why tempest is at 11.0.0 in HEAD, but ===10.0.0 in upper-constraints17:32
*** rcernin has joined #openstack-infra17:33
*** tqtran has joined #openstack-infra17:33
*** moravec has quit IRC17:35
*** fitoduarte has joined #openstack-infra17:36
*** ddieterly is now known as ddieterly[away]17:36
greghaynesSpamapS: 11.0 uploaded on 2016-04-0717:36
greghaynesSpamapS: so the update just hasnt been run yet17:36
greghaynes was the most recent17:37
*** pvaneck has joined #openstack-infra17:37
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Add undercloud idempotency test
SpamapSgreghaynes: oh, so you're thinking tempest hasn't actually landed a patch since they tagged at 11.0.0 ?17:38
SpamapSergo, the constraints are still at 1017:38
SpamapSthat's kind of a weird side-effect of release day17:38
greghaynesSpamapS: or that they dont test that they work with constraints in their gate17:38
clarkbconstraints is updated by a daily job or manual updates17:38
AJaegerSpamapS, greghaynes: We update upper-constraints once every day - via a periodic job that gets queued around 6:00 UTC17:39
clarkbthat ^17:39
AJaegerand then it needs to be approved and merged in17:39
AJaegerYeah, as clarkb says17:39
SpamapSclarkb: OH! I thought it was updated every time the gate passed.17:39
AJaegergreghaynes, SpamapS : So, that job was not run. You can update manually if you need it now - propose a change for requirmenets repo...17:39
greghaynesnope, openstack poposal bot17:39
SpamapSGOT IT17:39
AJaegerSpamapS: proposal bot and review that everything still works17:39
clarkbfungi I am about to run that errand now should be back in 45 minutes ot so17:40
*** degorenko is now known as _degorenko|afk17:40
*** BassT has joined #openstack-infra17:40
SpamapSit's odd though, that the constraint not bumping, makes git uninstallable17:40
greghaynestheres already a requirements patch up to do that
greghaynesand in theory that should fail17:41
greghayneswith the error we are seeing17:41
SpamapSwell.. no17:41
SpamapSgreghaynes: no it should pass!17:41
*** ihrachys has quit IRC17:41
SpamapSbecause it has 11.0.0 in upper-constraints17:41
fungiclarkb: cool, i'm around now and catching up so should be ready to collaborate on nodepoolectomy shortly17:41
fungiclarkb: i _think_ alien cleanup will just "happen"17:42
fungiwe haven't namespaced it17:42
*** lezbar has joined #openstack-infra17:42
*** lezbar__ has quit IRC17:42
clarkbfungi great, for prep we do need to reinstall nodepool to the older commit, fix the nodepool.yaml yo use zuul.o.o for geard, and stop my geard and nodepool builder17:42
clarkbfungi images too? uh that may be a ptoblrm of my uploaded images were deleted :/17:43
pabelangerfungi: if your review queue is clear: will help with centos-7 testing for kolla17:43
greghaynesSpamapS: ah, right. We just hit a narrow window where the two didnt match17:43
clarkbfungi can you see if yhe new images are still in the clouds?17:43
SpamapSgreghaynes: yeah I wonder what other projects we might hit such a window on17:43
pabelangerAJaeger: adds experimental ubuntu-xenial jobs for devstack-gate17:43
greghaynesSpamapS: exactly my thought :/17:44
SpamapSso, cherry picking that in, doesn't fix it17:44
SpamapSCould not satisfy constraints for 'tempest': installation from path or url cannot be constrained to a version17:44
greghaynesSpamapS: Yea, so there might be something else going on17:44
AJaegerpabelanger: LGTM17:45
SpamapSgreghaynes: that error in particular smells wrong17:45
SpamapSI wonder if I have to add a #egg thing17:45
fungiclarkb: i don't think it does alien image deletes, just instances17:46
fungiwe can cleanup images though, yes17:46
*** shashank_hegde has joined #openstack-infra17:46
fungiclarkb: checking to be sure, yes17:46
SpamapSnope, #egg=tempest doesn't help17:46
*** jordanP has quit IRC17:46
fungiSpamapS: the idea with constraints was that if having a newer release of something reflected in constraints was urgent, someone would manually propose that update and get it fast-tracked through review17:47
krotscheckjtomasek: That npm prune fix merged, can you recheck tripleo-ui to see if your issue's been resolved?17:47
AJaegerkrotscheck, jtomasek let's wait 60 mins after merge so that all jenkins are updated before you recheck17:48
SpamapSfungi: ACK.17:48
krotscheckAJaeger: Ok, so another 33 minutes17:48
SpamapSgreghaynes: so, at worst, given the way we're doing CD, it means our CI stops testing things until the proposal bot does it.17:48
fungiSpamapS: so... if it's urgent for you, then go for it. there are no taboos with proposing constraints bumps manually17:48
SpamapSbut, I still can't install tempest17:48
SpamapSfungi: there's already one proposed17:48
SpamapSqueued for 43 min ;)17:48
SpamapSor, 50 now17:48
SpamapSthings are .. a bit backed up ;)17:49
greghaynesSpamapS: Well, you mentioned that pulling in that change locally didnt fix it?17:49
SpamapSgreghaynes: right17:49
SpamapSso I suspect something else17:49
AJaegerkrotscheck: 60 mins is my rule of thumb, could be faster - and if it takes longer, we have a problem ;)17:49
AJaegerkrotscheck: is still needed?17:50
*** SumitNaiksatam has joined #openstack-infra17:51
*** lakshmiS has quit IRC17:52
krotscheckAJaeger Looks like it's updated everywhere except jenkins0217:52
*** esker has quit IRC17:52
krotscheckAJaeger I... looking at the console, I can't actually tell why that failed.17:52
mtreinishSpamapS: right, I just pushed the release a couple of hours ago17:53
*** e0ne has joined #openstack-infra17:53
AJaegerwow, 270 changes in check queue and 99 in post - our systems are busy17:53
mtreinishI don't believe any commits have landed since then17:53
AJaegerpabelanger: could you check whether jenkins02 needs a manual JJB run, please? See krotscheck's comment above17:53
mtreinishSpamapS: fwiw:
krotscheckAJaeger: It won't hurt anything. Let me ponder whether it makes logical sense...17:54
*** ifarkas has quit IRC17:55
*** maishsk_ has joined #openstack-infra17:55
*** kzaitsev_mb has quit IRC17:55
*** david-lyle has joined #openstack-infra17:56
pabelangerAJaeger: sure17:56
*** david-lyle has quit IRC17:56
krotscheckAJaeger: So, the purpose of running npm shrinkwrap in the gate is basically just to tell us which versions a package was being built with. The shrinkwrap file, if already included, basically puts the onus of generating that file on the project itself. I see no reason why adding a switch would harm anything.17:56
*** maishsk has quit IRC17:56
*** maishsk_ is now known as maishsk17:56
*** xarses has joined #openstack-infra17:56
krotscheckAJaeger: I did downvote it for other reasons though17:57
fungithe answer to "is jenkins02 broken?" is almost always "yes"17:57
*** david-lyle has joined #openstack-infra17:58
fungiit seems to be the one most frequently hit by jenkins api timeouts, which sometimes causes jjb updates not to complete, which causes them to pile up more often, which means it eventually gets bad enough they exceed the execution timeout from our automation17:58
*** vgridnev has joined #openstack-infra17:58
openstackgerritMerged openstack-infra/project-config: Add experimental devstack-jobs for ubuntu-xenial
fungiafter that, it pretty much always requires some manual intervention17:59
fungiboth jenkins01 and jenkins02 could stand to be rebuilt on newer nova flavors17:59
*** dprince has quit IRC18:00
*** vikram has quit IRC18:00
*** dprince has joined #openstack-infra18:00
*** cloudtrainme has joined #openstack-infra18:01
*** jsavak has quit IRC18:01
*** jamesmcarthur has joined #openstack-infra18:02
*** jsavak has joined #openstack-infra18:02
*** e0ne has quit IRC18:03
*** julim has quit IRC18:04
*** ociuhandu has quit IRC18:04
*** piet has quit IRC18:06
*** jcooley_ has quit IRC18:06
openstackgerritamrith proposed openstack/requirements: Update upper-constraints for python-troveclient v2.2.0
pabelangerAJaeger: re-running JJB on jenkins0218:07
*** moravec has joined #openstack-infra18:08
*** mrmartin has quit IRC18:09
greghaynesSpamapS: yea, so as I read this it has nothing to do with the current constraints not matching18:10
*** shardy has quit IRC18:12
AJaegerthanks, pabelanger18:12
greghaynesSpamapS: ok, so services are not listed in constraints18:12
greghaynesSpamapS: We are installing tempest from a path, that means it cant determine if that patch fulfills the constraint18:12
SpamapSgreghaynes: ok, so we have to use not-constraints for that18:13
*** e0ne has joined #openstack-infra18:13
greghaynesSpamapS: yep18:13
SpamapSgreghaynes: kinda sux18:13
SpamapSbut ok18:13
greghaynesSpamapS: or we have to remove tempest from our constraints when we use constraints18:13
*** dprince has quit IRC18:13
*** rajinir has joined #openstack-infra18:13
*** sridhar_ram has joined #openstack-infra18:14
*** e0ne has quit IRC18:14
*** dprince has joined #openstack-infra18:15
*** annegentle has quit IRC18:16
*** annegentle has joined #openstack-infra18:16
openstackgerritMark Vanderwiel proposed openstack-infra/project-config: Enable lbaasv2 jobs in check and gate
fungiclarkb: a spot check says that the new nodepool's uuid for the ubuntu-trusty image in rax-dfw appears in nova image-list output there, so we should be fine wrt it not "cleaning up" our images from the new server18:18
fungithough the nodepool alien-image-list cli subcommand looks like it may be broken18:19
*** ociuhandu has joined #openstack-infra18:21
pabelangerAJaeger: krotscheck: and done18:22
krotscheckpabelanger: Cool, we'll check.18:22
*** oanson has quit IRC18:24
*** sdake_ has joined #openstack-infra18:26
*** doug-fish has joined #openstack-infra18:26
*** sdake has quit IRC18:26
jtomasekkrotscheck: ack18:27
krotscheckjtomasek: link to job so I can watch it?18:27
krotscheckjtomasek: oh. 250 things in the gate. Weeee18:28
jtomasekkrotscheck:  change id 30113718:28
*** nwkarsten has joined #openstack-infra18:30
sdaguethere was also a fixtures release that ... made a lot of things unhappy18:30
*** esker has joined #openstack-infra18:32
*** salv-orlando has joined #openstack-infra18:32
*** cody-somerville has joined #openstack-infra18:33
*** tonytan4ever has quit IRC18:34
*** tonytan4ever has joined #openstack-infra18:34
*** ddieterly[away] has quit IRC18:36
*** amitgandhinz has quit IRC18:37
*** salv-orlando has quit IRC18:38
*** jsavak has quit IRC18:38
*** nihilifer has quit IRC18:39
*** slagle has joined #openstack-infra18:39
*** jsavak has joined #openstack-infra18:40
*** cdent has joined #openstack-infra18:40
openstackgerritMark Vanderwiel proposed openstack-infra/project-config: Enable lbaasv2 jobs into check queue
*** ddieterly has joined #openstack-infra18:42
*** akshai has joined #openstack-infra18:42
*** sambetts|afk has quit IRC18:42
*** dprince has quit IRC18:42
*** dprince has joined #openstack-infra18:43
clarkbfungi: I am back now, I will work on getting the right version of nodepool installed, fix the config file, and stop the services I had started to get images up18:44
*** angdraug has joined #openstack-infra18:44
*** ddieterly is now known as ddieterly[away]18:44
fungiright version of nodepool? did we have the wrong version on the new server?18:45
fungianyway, not sure if you saw in scrollback, but your new images are safe and sound18:45
clarkbafdd58c10a8b85ea1466d32c3a29c0d7a1b0aea9 is the sha1 I will be installing18:45
clarkbfungi: the issue is the use shade for all interactions doesn't work yet and has not been reverted18:46
clarkbso old nodepool was running a manually installed older version (the sha above I think but I cannot confirm on old server)18:46
clarkbfungi: I did ty18:46
openstackgerritPaul Belanger proposed openstack-infra/devstack-gate: Test commit
*** sambetts has joined #openstack-infra18:46
*** esker has quit IRC18:46
*** ddieterly[away] is now known as ddieterly18:46
fungioh, got it. i though we had officially reverted again18:47
funginot just manually rolled the server back to an older commit18:47
clarkbnodepool==0.2.1.dev71  # git sha afdd58c so thats done18:48
clarkbfungi: thats it for prep I think18:49
clarkbnow its just a matter of shutting down the old server and updating dns18:49
clarkbthen whatever cleanup we need to do18:49
*** annegentle has quit IRC18:49
clarkbI will get on puppet master to do the nova shutdown18:49
*** annegentle has joined #openstack-infra18:50
openstackgerritPaul Belanger proposed openstack-infra/system-config: Add support for ubuntu-xenial wheel mirror
*** SumitNaiksatam has left #openstack-infra18:51
clarkbfungi: are you ready? do you want to get the dns updates?18:52
clarkbI didn't set up reverse records for new server yet eitehr18:52
*** salv-orlando has joined #openstack-infra18:52
*** annegentle has quit IRC18:52
fungiclarkb: sure, i can do dns updates (forward and reverse). i expect we're going to set zuul back a bit since it's got a backlog and we're currently pegging all our quotas18:53
clarkbfungi: do you think we should wait for zuul to catch up?18:53
fungijobs running on machines which get alien-deleted are hopefully going to restart automatically18:53
clarkbnodepool is running on the other server we will just have to do this if anything goes wrong18:53
clarkbfungi: yup they should18:53
openstackgerritMatt Riedemann proposed openstack/requirements: Temporarily cap fixtures<2.0
mriedemsdague: dims: ^18:54
clarkbmostly want ot get this done before it becomes pressing in an emergency, there may not be a good time at all so just doing it is best18:54
fungiyeah, let's just go for it18:54
*** jsavak has quit IRC18:54
fungii think it'll be transparent18:54
dimsmriedem : +2, sdague : can you please +2A?18:55
clarkbfungi: lte me know when you are ready to start the dns updates and I will nova stop the old server18:55
sdaguedims done18:55
sdaguefungi: there is currently no integrated gate18:55
fungisdague: yeah, just a ton of check/post18:55
sdaguefungi: as such, would you mind promoting to gate18:55
sdagueas it won't stomp out anything18:56
fungisdague: once we're done replacing nodepool18:56
dimsfungi : 2 projects affected :)18:56
*** chem has quit IRC18:56
sdaguedims: well, a bunch of neutron projects, so it's more than 218:56
clarkbfungi: we may need to service iptables restart to pick up the changed ip addr on zuul.o.o and jenkins masters18:57
*** jsavak has joined #openstack-infra18:57
zigoclarkb: newgrp? I'm not sure, here, it's sbuild stuff, it's a bit special, I'm not really sure how it works. On my Jenkins server, I really do have to logout and login again (ie: new ssh session). Others in infra told me doing another shell stuff will do another ssh session. Let's try, as you wrote...18:57
fungiclarkb: addresses updated in dns to and 2001:4800:7815:103:be76:4eff:fe06:95e2 just now18:57
zigo(and thanks for your +2)18:57
clarkbfungi: ok stopping old server now18:57
zigofungi: Can you have a look as well?
*** hashar has joined #openstack-infra18:58
fungizigo: i have a bit of a review backlog which i'm already hoping to find time for after we finish this maintenance18:58
fungibut i'll see18:58
clarkbERROR (CommandError): Unable to stop the specified server(s).18:58
openstackgerritRoland Hochmuth proposed openstack-infra/project-config: Add openstack/monasca-transform project
clarkbThere is no such action: os-stop (HTTP 400)18:58
zigofungi: it's a tiny small patch... :)18:58
clarkbsomething something interop18:58
* clarkb looks into how to actually stop this thing18:58
zigofungi: Just dong 2 - shell: instead of one, to get a new login session after sbuild is installed.18:59
fungiclarkb: there's "nova stop"18:59
fungii assume that's the same thing openstack server stop is trying to call though18:59
clarkbfungi: ya nova stop is what I ran19:00
*** sambetts has quit IRC19:00
clarkblooks like we have suspend and pause as options too19:00
*** e0ne has joined #openstack-infra19:00
*** ddieterly is now known as ddieterly[away]19:00
clarkband shelve19:00
clarkbso we have 4 different verbs19:00
clarkbat least one doesn't work19:00
clarkbsdague: ^ is there a recommended one for "turn off this server"19:01
*** salv-orlando has quit IRC19:01
fungiclarkb: in the rackspace webui, the only thing that looks likely to "stop" the server is delete19:01
sdagueclarkb: what are you going to do with the server after you turn it off?19:01
sdaguebecause, delete is the normal workflow19:01
fungisdague: possibly turn it back on if this doesn't go well19:01
clarkbsdague: probably will delete it, but want to have the disk around just in case we need to mount it forsome reason19:01
sdaguenormally I'd then suggest snapshot + delete19:02
*** esikachev has joined #openstack-infra19:02
fungii suppose we could snapshot it19:02
fungiyeah, that19:02
clarkbso as a user thats broken19:02
clarkbwe have 4 verbs that are stop this thing without deleting it19:02
clarkbbut snapshot and delete is what we need to do...19:02
sdaguewell, you have a provider that didn't implement some of them19:03
fungiwell, pause and suspend and shelve will... probably not work in rax19:03
sdaguebecause, openstack was built of extensions19:03
clarkbya I don't know that its necessarily nova's fault just weird in general19:03
clarkbfungi: I will work on snapshotting it19:03
sdagueshelve would probably also be fine19:03
*** sambetts has joined #openstack-infra19:03
sdaguethat kind of does snapshot + stop19:03
clarkblooks like we have multiple snapshot verbs too19:04
sdagueand I agree, the verbs are all over the place19:04
fungiit's possible rax supports shelve but doesn't expose it in their dashboard19:04
clarkb'backup' and 'image-create'19:04
*** shu-mutou-AFK has quit IRC19:04
sdaguebut, that all happened before I got here19:04
* clarkb does image-create19:04
sdaguefungi: that is possible, I know it's accessable to rax people19:04
clarkbshould I try shelving as a user?19:05
clarkbmaybe I will try that first19:05
fungiclarkb: worth a shot19:05
*** esker has joined #openstack-infra19:05
*** esker has quit IRC19:05
fungiclarkb: one difference i've spotted19:05
*** esker has joined #openstack-infra19:06
clarkbERROR (BadRequest): There is no such action: shelve (HTTP 400)19:06
fungiclarkb: the old server's cinder volume used ssd, new one is sata19:06
openstackgerritArmando Migliaccio proposed openstack-infra/project-config: Update Gerrit ACLs for vendor specific networking-* projects
clarkbfungi: oh19:06
clarkbdoes openstack volume list specify that difference? I might have just missed it19:06
fungii know we switched to ssd at one point because of performance concerns, not sure if they are still concerns19:06
clarkbfungi: based on build times yesterday I don't think it is an issue19:06
fungii expect that if you didn't see poor performance during image updates, then sata is good enough, yes19:07
*** nikhil has quit IRC19:07
clarkbif it persists it is lvm'd and we can swap it around and do the volume mv thing19:07
fungiabsolutely, so let's ignore that for now19:07
clarkbfungi: most of our time is spent uploading not building19:07
clarkbok image-creating now19:07
sdague"openstack volume type list" is where you'd find details out about it I think19:07
openstackgerritVitaly Kramskikh proposed openstack-infra/project-config: Do not run npm shrinkwrap if npm-shrinkwrap.json already exists
fungii added reverse dns for the new server now too19:07
clarkbreally we need bigger pipes to rax-ord and ovh-bhs if we want to speedup the entire image lifecycle19:09
*** jcooley_ has joined #openstack-infra19:09
clarkbit is "SAVING"19:10
*** thiagop_ has joined #openstack-infra19:10
*** thiagop_ has quit IRC19:10
*** jcooley_ has quit IRC19:11
*** nikhil has joined #openstack-infra19:11
*** hdd has joined #openstack-infra19:12
clarkbI have no idea how long this will take, in theory not too bad since we used performance flavors will smaller  /19:13
clarkbfungi: image is called nodepool-no-ssh-host-keys19:13
*** david-lyle_ has joined #openstack-infra19:13
fungiit's giving me time to catch up on the "non-atc recognition working group" meeting19:13
*** vgridnev has quit IRC19:14
*** salv-orlando has joined #openstack-infra19:14
clarkbwhat if it somehow snapshots the half terabyte /opt19:15
*** david-lyle has quit IRC19:15
* clarkb hopes it is smarter than that19:15
*** bkeller` has joined #openstack-infra19:16
*** sigmavirus24 is now known as sigmavirus24_awa19:17
fungiit should only snapshot the rootfs afaik19:17
*** Sukhdev has joined #openstack-infra19:18
*** david-lyle_ has quit IRC19:19
clarkbstill saving19:21
* clarkb reads email and tries to be patient19:21
*** ddieterly[away] is now known as ddieterly19:21
*** bpokorny_ has joined #openstack-infra19:23
*** ihrachys has joined #openstack-infra19:24
openstackgerritRyan Hallisey proposed openstack-infra/tripleo-ci: Allow the container job to run again
*** briancline has quit IRC19:24
*** briancline has joined #openstack-infra19:25
*** bpokorny has quit IRC19:26
*** jaosorior has quit IRC19:27
*** lunarlamp has joined #openstack-infra19:28
*** david-lyle has joined #openstack-infra19:29
*** bkeller` has quit IRC19:30
*** david-lyle has quit IRC19:31
*** david-lyle has joined #openstack-infra19:31
*** Jeffrey4l_ has joined #openstack-infra19:32
clarkbstill going maybe I should find lunch19:33
*** julim has joined #openstack-infra19:33
pvaneckfungi: hey, I wanted to revisit this patch you submitted way back for enabling the mysql-backup cron job for refstack if you have a moment:
*** Jeffrey4l has quit IRC19:34
pvaneckSeeing your last comment in that review, I submitted a patch to puppet-refstack , but yolanda makes the point that these cron jobs are typically declared in system-config.19:34
clarkbyes, I think the idea behind putting backup info in system-config is that it is site specific and not application specific19:34
*** arxcruz has quit IRC19:35
fungiclarkb: pvaneck: the reason it ended up being not straightforward for refstack is that it has no dedicated puppet class in openstack_project19:35
clarkbfungi: ah19:35
fungiit's just a node declaration in the global site manifest19:35
*** tonytan_brb has joined #openstack-infra19:35
nibalizerpabelanger: around?19:35
*** mylu has quit IRC19:35
pabelangernibalizer: I am19:35
pabelangernibalizer: we should probably wait until new nodepool.o.o is online19:36
*** tonytan4ever has quit IRC19:36
fungiso to do it like "most" servers we need a shim class added to openstack_project in the system-config repo that instantiates the refstack module and also instantiates the mysql_backup::backup_remote class19:36
pabelangernibalizer: so, likely next monday now19:36
clarkbfungi: or we could put it in the site.pp19:36
clarkbthat starts to get ugly though19:36
nibalizerpabelanger: wfm19:36
fungiclarkb: well, i tried that with 25442619:36
pvaneckfungi: Seems having it in system-config is the way to go. I think your patch set one is the way to do it19:37
nibalizerfungi: yea a shim class is generally the way to do it19:37
nibalizerkeeps site.pp under control19:37
nibalizerbut see a host like nodepool where it clearly didn't work19:37
pvaneckyou were just missing an apostrophe in the Class['refstack]19:37
*** david-lyle has quit IRC19:37
fungipvaneck: hah, so i was19:37
*** cloudtrainme has quit IRC19:37
fungii'll push an update there--good eye19:37
pvaneckjust tested it out locally to see how it would work and caught that19:38
pvaneckwith that closing apostophe it works for me19:38
clarkbok snapshot is ACTIVE19:38
*** mylu has joined #openstack-infra19:38
clarkbfungi: ready for me to delete the old host?19:38
pvaneckpatchset 1 that is19:38
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Add database backups for
fungipvaneck: nibalizer: ^19:39
fungiclarkb: go for it19:39
openstackgerritKhai Do proposed openstack-infra/system-config: Customize Gerrit's download command and schema
*** tonytan_brb has quit IRC19:39
*** baoli has joined #openstack-infra19:39
*** tonytan4ever has joined #openstack-infra19:39
clarkbyour request for deletion has been accepted19:40
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Update func tests for latest devstack flavors
lifelessSpamapS: yes, it does19:40
fungiclarkb: watching for ping responses to cease19:40
lifelessSpamapS: you need to run edit-constraints, or remove it from the constraints file19:40
clarkband its gone19:40
lifelessSpamapS: since you're basically saying19:40
clarkbat leats according to nova19:40
*** bpokorny_ has quit IRC19:40
lifelesstempest == somever + tempest == someotherver19:41
lifelessSpamapS: and thats an impossible situation19:41
clarkbfungi: I get no ping repsonses, do you?19:41
*** bpokorny has joined #openstack-infra19:41
fungiclarkb: oh, hah, i was still doing it by name, which is now resolving to the new server ;)19:41
fungiyes, old addresses no longer respond, so i think we're good to press forward19:41
clarkbfungi: ok I am starting services on the new server now19:41
*** dprince has quit IRC19:42
*** hdd has quit IRC19:42
clarkbnot seeing new image upload jobs in gearman so I think firewalls do need bouncing19:43
fungiclarkb: we will likely end up with a ton of defunct slaves on the jenkins masters after the alien node cleanup completes19:43
pabelangergreghaynes: krotscheck: when you added the wheel_mirror_slave.pp to system-config, was there a specific reason for not setting thin => true for openstack_project::slave?19:43
clarkbfungi: yup, I am going to bounce iptables service on zuul and the jenkinses19:44
pabelangergreghaynes: krotscheck: trying to see if it is safe to remove the dependency on think_slave.pp19:44
fungijust restart the iptables-persistent "service"?19:44
fungii can do some of them19:44
krotscheckpabelanger: That decision was made before I took over the patches. greghaynes ?19:44
clarkbfungi: yup19:44
clarkbI just did jenkins 0719:44
fungii'll start with jenkins and 01 up19:45
fungidid up to 0319:46
*** BassT has quit IRC19:46
clarkbdoing zuul now19:46
fungiassumed you were counting downward19:46
clarkbthere we go19:46
openstackgerritPaul Belanger proposed openstack-infra/system-config: [WIP] Remove dependency on think_slave for wheel mirrors
clarkbstarting nodepoold now19:46
fungiexpecting eta on a bunch of changes to go back to unknown19:47
openstackgerritPaul Belanger proposed openstack-infra/system-config: Add support for ubuntu-xenial wheel mirror
clarkbit is doing its things so as quickly as the aliens can be deleted19:47
clarkbprobably within the netx 5 minutes or so19:47
fungiclarkb: okay for me to fix the cached ssh host key on puppetmaster and clear the ansible inventory cache?19:48
pabelangerclarkb: did we roll back nodepool to the pre-shade commit?19:48
fungipabelanger: yep19:48
clarkbfungi: yes I think it is19:49
clarkbwe are hitting quota issues but that is expected until aliens go away19:49
clarkband the snapshot builds have started19:49
fungiokay, `sudo ssh` works from puppetmaster.o.o now and /var/cache/ansible-inventory/ansible-inventory.cache has been truncated19:50
nibalizercrinkle: re the pip hax19:52
clarkbfungi: does nodepool cleanup aliens periodically?19:52
nibalizera) i think this is the right amount of hax19:52
*** bpokorny has quit IRC19:52
clarkbfungi: ah found it cleanupLeakedInstances19:52
clarkbfungi: which is why git grep -i alien didn't find it19:53
*** matt-borland has quit IRC19:53
nibalizerif you look at the diff here getting what we want out of the json api is actually super easy19:53
*** bpokorny has joined #openstack-infra19:53
nibalizerand dstufft told me that the json api makes him happy19:53
funginot that said, i don't know if we've ever tested what happens when we spontaneously "leak" some 500-600 instances19:53
dstufftyes, json api is better19:53
SpamapSlifeless: so, it looks like on April 1, tempest got added to upper-constraints19:53
dstufftthan the XMLRPC one anyways, we can't cache XMLRPC in Fastly19:53
clarkbwe have building, ready and used instances so that all looks well19:53
SpamapSlifeless: which is what broke us ultimately.19:54
fungiSpamapS: sounds like an april fool's joke someone forgot to revert?19:54
*** alex_xu has quit IRC19:54
clarkband snapshot images are building19:55
clarkbvexxhost images uploading19:55
clarkblgtm so far19:55
SpamapSfungi: yeah I dunno, the commit message looks legitimate.. things depend on it to work. But at the same time, it means we can't install tempest from a git dir with constraints without making a separate constraints file without tempest in it.19:55
*** alex_xu has joined #openstack-infra19:55
*** tonytan_brb has joined #openstack-infra19:55
clarkbtempest itself is installed to a venv without constraints iirc19:56
fungiSpamapS: yep, i looked at and it does seem legit19:56
SpamapSyeah, we were trying to do that _with_ constraints.19:56
mtreinishclarkb: right, we don't constrain tempest when we run it in the gate19:56
clarkbya thats how I would do it elsewhere then19:56
SpamapSI'm a bit curious at why pip can't just use the version from the dir to match the constraints though.19:56
mtreinishwe do honor constraints for running some config utils as part of the devstack run IIRC19:56
clarkbfungi: we should check for fip leakage on osic and bluebox once things settle a bit more19:57
lifelessSpamapS: or editing your constraints file19:58
lifelessSpamapS: see devstack for examples of editing constraints19:58
fungiclarkb: great point19:58
mtreinishbut that's kinda cheating because tox will make a venv unconstrained and then we pip install with constraints in the same venv after19:58
*** tonytan4ever has quit IRC19:59
SpamapSlifeless: so I wonder is it a violation of the model to allow constraining the git dir one is pip installing, or an implementation limitation?20:00
SpamapSI can sdist the git dir, and pip install the resulting tarball just fine using constraints.20:00
pabelangernodepool.o.o doesn't appear to be logging to statsd anymore20:00
*** yamahata has joined #openstack-infra20:00
lifelessSpamapS: as long as the version matches the constraints you can20:00
clarkbpabelanger: oh need to kick iptables there too20:00
lifelessSpamapS: if the constraints don't match the version, you should get an equivalent error20:01
SpamapSlifeless: right, which is fine. I got into a situation where once we updated constraints, it still did not work.20:01
SpamapSCould not satisfy constraints for 'tempest': installation from path or url cannot be constrained to a version20:01
*** esikachev has quit IRC20:01
lifelessSpamapS: pastebin the constraints file you had?20:01
SpamapSlifeless:  <-- that one20:02
*** jtomasek has quit IRC20:03
*** baoli has quit IRC20:03
*** baoli has joined #openstack-infra20:04
*** jtomasek has joined #openstack-infra20:05
fungipabelanger: are you sure? i see building count spiking20:06
greghaynesSpamapS: lifeless: was raising because was None and was set20:06
fungipabelanger: and used count falling20:06
clarkbfungi: I am going to start fip things20:06
fungiclarkb: you doing osic first?20:06
* SpamapS doesn't give a fip20:06
fungii can take bbox if so20:06
greghaynesfor tempest, since existing req was in constraints, and install_req was the path we were intsalling20:06
*** lakshmiS has joined #openstack-infra20:07
clarkbfungi: yup doing osic since they hav elarger quota20:08
fungii've got fip cleanup underway in bbox now20:08
*** cloudtrainme has joined #openstack-infra20:08
lifelessSpamapS: thats not an edited file20:08
clarkbfungi: ty20:08
lifelessSpamapS: I want the actual file off of disk20:08
lifelessSpamapS: the one you say you updated20:08
fungias predicted lots of stuff in check now has an unknown eta. that's good (we didn't insta-fail everything!)20:08
lifelessSpamapS: I want to make sure that the edit was done correctly20:08
openstackgerritPaul Belanger proposed openstack-infra/project-config: Make sure we run bindep for all puppet-jobs
lifelessgreghaynes: yes, thats deliberate20:09
pabelangerfungi: I was looking on grafana.o.o20:09
pabelangerEmilienM: we likely need that ^20:09
fungipabelanger: shouldn't it pull from the same sources?20:09
pabelangerfungi: it should, let me double check20:09
fungigrafana statsd i mean20:09
clarkbpabelanger: fungi I kicked iptables on graphite.o.o20:09
pabelangerfungi: okay, working now20:09
SpamapSlifeless: oh, I did not update one, I mean, I tried this one, and it failed.20:09
jtomasekkrotscheck: now running
clarkbthe ackets were getting nacked20:09
krotscheckjtomasek: Finally20:10
SpamapSlifeless: and 11.0.0 is the same version as the sdist produces20:10
fungiclarkb: oh, i was about to wonder if that's what we were missing20:10
fungigood call20:10
EmilienMpabelanger: excellent!20:10
*** ddieterly is now known as ddieterly[away]20:10
greghaynesSpamapS: lifeless I was reproing using upper constraints from that review ( and installing tempest from git...20:11
SpamapSI can still reproduce that right now20:11
fungipabelanger: EmilienM: out of curiosity, what packages were missing? this is likely to preinstall a bunch more packages which weren't installed for you on devstack-trusty20:11
funginot that it likely matters much20:11
SpamapSI assumed it was just some limitation of installing from git.20:11
greghaynesSpamapS: not from git, from a path on disk20:12
greghaynesSpamapS: although git would do the same I think20:12
SpamapSgit would have a version20:12
pabelangerEmilienM: once that lands, we can start playing with other-requirements.txt, adding an empty file to ensure the puppet modules install everything that is required20:12
EmilienMfungi: it's only in our puppet4 jobs (experimental)20:12
fungiEmilienM: pabelanger: i see, we were preinstalling libfacter on devstack-trusty?20:13
pabelangerYa, need to see where libfacter is coming from20:13
pabelangerfungi: don't know yet20:13
*** jed56 has quit IRC20:13
fungii don't think our bindep fallback list includes it either, but maybe it does. i haven't looked20:13
*** Nikolay_St has quit IRC20:13
*** hdd has joined #openstack-infra20:14
clarkbfungi: osic fips are cleaned up20:15
*** ddieterly[away] is now known as ddieterly20:15
openstackgerritIhar Hrachyshka proposed openstack-infra/project-config: Run a set of lightweight neutron jobs on upper-constraints.txt changes
krotscheckjtomasek: Hrm, that failed.20:15
jtomasekkrotscheck: :/20:15
pabelangerwonder if it is a locale issue20:15
krotscheckBut _why_20:15
*** bpokorny has quit IRC20:16
fungiclarkb: bbox deletes still going, but should be done in moments20:16
fungiclarkb: as predicted, it was 100% of our fip quota leaked20:16
*** annegentle has joined #openstack-infra20:16
pabelangerEmilienM: which review is that failure from?20:17
pabelangerEmilienM: want to recheck and hold the image, so I can see what is going on20:17
krotscheckjtomasek: You don't happen to know if there's something weird with cat? It' somewhat surprised that the shrinkwrap file wasn't entirely output in th ebuild20:18
jtomasekkrotscheck: only error I see is [31m07 04 2016 20:14:24.080:ERROR [phantomjs.launcher]: [39mFontconfig warning: ignoring C.UTF-8: not a valid language tag20:18
EmilienMpabelanger: check experimental20:18
*** ayoung has quit IRC20:18
jtomasekkrotscheck: no idea20:18
fungipabelanger: i'm not seeing any locale-related errors in the job log. are you assuming they're being silenced somehow?20:18
clarkbjtomasek: do you ru nthat on centos?20:18
pabelangerfungi: maybe, going to hold the node and take a peak20:18
clarkbjtomasek: that isn't a valid local on centos, you can use C or something like en_US.utf-820:18
jtomasekclarkb: the job runs on ubuntu-trusty afaik20:19
clarkbshould be valid there iirc, but maybe my memory is wrong20:19
jtomasekclarkb: I am not sure this is the error which causes the failure though20:19
clarkbfungi: I think the only outstanding thing in new nodepool is images. We have image builds for snapshots and the regions I didn't bother with yesterday due to max servers set low. And cleaning out the alien images20:19
*** kgiusti has left #openstack-infra20:20
fungiclarkb: i think precise is the only snapshot image we still care about now?20:20
fungibare-precise i mean20:20
clarkbfungi: and tripleo's20:20
fungibut yeah, that may mean a long wait for any precise-based jobs20:20
*** maishsk has quit IRC20:21
clarkbthe others are devstack-centos7 in rax beacuse it isn't dib there but is going away soon and vexxhost and rax-ord20:21
pabelangerfungi: Ya, this will be a rubygem not distro package20:21
clarkbI am going to start working on figuring out the alien image list20:21
*** mrmartin has joined #openstack-infra20:21
nibalizercrinkle: haha nvm20:21
fungiclarkb: you mean the exception it's raising when you try to run it?20:21
krotscheckclarkb: Is there something about how the logfiles are accessed by jenkins that would throw up if someone did a `cat` on a 294kb text file?20:22
clarkbfungi: haven't even gotten that far :)20:22
clarkbkrotscheck: shouldn't20:22
clarkbkrotscheck: but it might be ebtter to log that file separately20:22
*** eharney has quit IRC20:22
clarkbkrotscheck: if you want its contents20:22
fungiclarkb: yeah, i tested it a little bit ago when you were erranding and i think it's bitrotted20:22
mtreinishnibalizer: did we ever get the cacti thing fixed? I still don't see subunit-worker02.o.o listed there20:23
*** sdake_ is now known as sdake20:23
clarkbfungi: oh fun, I see it now20:23
*** maishsk has joined #openstack-infra20:23
ihrachysAJaeger: hi. could you please check ? I am not really sure in the syntax.20:24
EmilienMpabelanger: I found more package issue
jtomasekkrotscheck: yeah, the cat is not complete and right after it says 'step 'Execute shell' marked build as failure'20:26
*** tonytan_brb is now known as tonytan4ever20:26
*** eharney has joined #openstack-infra20:26
krotscheckjtomasek: I find that suspicious.20:26
krotscheckfuel's got the same problem.20:26
pabelangerEmilienM: do a recheck, we already fixed that20:26
EmilienMpabelanger: ack20:26
pabelangerEmilienM: TL;DR, removing rdo-release will break bindep, so we need to call bindep first20:26
*** hashar has quit IRC20:27
fungikrotscheck: jtomasek: is the example a log you're looking at on logs.o.o or on one of the jenkins masters directly?20:27
EmilienMpabelanger: ok thx20:27
clarkbfungi: that is an artisanal hand made alien image list for osic20:27
clarkbfungi: any reason to not start deleting those that you can think of? new nodepool seems to be functional20:28
fungiclarkb: nope, double-checked and none of those uuids show up in the new nodepool image-list output20:29
nibalizermtreinish: um likely not20:29
*** tongli has joined #openstack-infra20:29
fungiclarkb: should be fine to go for it20:30
*** moravec has quit IRC20:30
nibalizermtreinish: this is the review20:31
clarkbfungi: deletes running for osic, I will work on the others20:31
clarkbfungi: I think the alien list thing is related to shade stuff and it will likely end up being a mile long thread to pull on20:31
clarkbso not worrying about it yet20:31
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Do not cat npm-shrinkwrap.json
krotscheckjtomasek, fungi, clarkb: Maybe this'll work? ^^20:31
*** _ari_ has quit IRC20:32
ddieterlywhat is a username/password that can be used in the gates to ssh into the DevStack localhost?20:32
jtomasekkrotscheck: he:) the contents of shrinkwrap.json is pretty large, so maybe cat fails because of it20:32
krotscheckjtomasek: Well, I tried running cat on my little trusty vm here, and it didn't seem to have any problems.20:33
mtreinishnibalizer: cool, +1 :)20:33
ddieterlybasically, i need to ssh into the same machine (devstack) inside a test20:33
fungiddieterly: i think they're set by devstack. but i'm looking20:33
nibalizerfungi: clarkb i can babysit  if you want to look at it20:34
nibalizerer 28446620:34
ddieterlyfungi thanks20:34
fungiddieterly: oh, ssh into devstack localhost? that i don't think is set at all20:34
fungiddieterly: the hosts which run these don't enable password auth20:35
fungiddieterly: i think most of the cases where we do this, we have the account set its own keys in authorized_keys so it can ssh localhost without a password20:35
clarkbfungi: ddieterly correct there is a macro for it in the puppet beaker rspec tests20:36
*** jcooley_ has joined #openstack-infra20:36
clarkbnibalizer: kind of swamped with nodepool stuff still :/20:36
*** julim has quit IRC20:36
ddieterlyfungi, ok, thanks20:37
clarkbfungi: osic and bluebox are done. Internap is in progress20:40
*** ericksonsantos has joined #openstack-infra20:41
clarkbthats interesting20:41
*** baoli has quit IRC20:41
clarkbon internap glance image-list and nova image-list do not agree20:41
clarkbmgagne: ^ fyi not sure if that is expected behavior. But it broke my glance image-delete loop20:41
mgagneclarkb: checking20:41
ddieterlyfungi clarkb is openssh setup and running in devstack in the gate jobs?20:42
clarkbbasically all of our images show up under nova image-list but only your cloud wide images show up on glance image-list20:42
clarkbddieterly: yes jenkisn communicates to the slaves via ssh20:42
ddieterlyok, so then in the test, i can setup a user with keys and use that user inside devstack?20:43
clarkbddieterly: yes there is already a jjb macro to do this and is used by the puppet beaker rspec jobs20:43
*** matt-borland has joined #openstack-infra20:43
ddieterlyok, no idea what jjb macro is nor puppet beaker spec job20:43
ddieterlyyes, i'm a lamo ;-)20:44
clarkbjenkins job builder, the tool we use to configure the jenkins jobs20:44
krotscheckjtomasek: I'm about to close out for the day.20:45
ddieterlyit looks like jenkins is setup on devstack to be able to ssh in20:45
*** baoli has joined #openstack-infra20:45
lifelessSpamapS: greghaynes: right, what you are describing is the expected behavour20:46
mgagneclarkb: I don't see any different between the 2.20:46
*** baoli_ has joined #openstack-infra20:46
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Fix patch to deploy.env for mirror puppet modules
clarkbmgagne: huh definitely different for out user/tenant. Maybe its a client version thing?20:46
fungiddieterly: allow-local-ssh-root
mgagneclarkb: will check20:46
*** mriedem has quit IRC20:46
clarkbwe have a lot of leaked images in ovh-gra120:47
SpamapSlifeless: kk, thanks for taking a look. :)20:47
clarkbI think because uploads there fail frequently and we don't cleanup after reliably20:47
fungiddieterly: it's used by jobs like this one
*** Sukhdev has quit IRC20:47
*** burgerk has quit IRC20:47
*** rfolco has quit IRC20:47
lifelessSpamapS: you need to use edit-constraints to tell pip you want a non-version-constrained tempest20:47
clarkbmordred: this is a problem with the glance image upload 2 step process fwiw20:47
lifelessSpamapS: again, see devstack for an example20:47
ddieterlyfungi thanks20:47
clarkbmordred: and I cna't think of a good reason it needs ot be that way but shade should handle it20:47
SpamapSlifeless: yeah, that's exactly what I think we'll do.20:48
* zigo just finished uploading all of Mitaka final to Debian Sid (list of services in this release which I support:
*** ddieterly is now known as ddieterly[away]20:49
mgagneclarkb: I sorted the output for better debug20:49
*** baoli has quit IRC20:50
clarkbmordred: 139 leaked iamges in ovh-gra120:50
*** baoli_ has quit IRC20:50
*** baoli has joined #openstack-infra20:51
ianwSpamapS / greghaynes: I'd really like to get the stack from in a point release soon because a) it's totally broken and b) making nodepool logs about 10x bigger than they need to be20:51
fungistill no new fips leaked in bbox, so that was presumably just from the cutover20:52
*** ddieterly[away] has quit IRC20:52
clarkbmordred: we should consider forcing glance v120:53
*** mylu has quit IRC20:54
*** nihilifer has joined #openstack-infra20:54
*** mylu has joined #openstack-infra20:54
clarkbassuming v1 is less silly, I hvaen't checked that20:54
*** david-lyle has joined #openstack-infra20:56
clarkbseeing quite a few launch fails in gra120:57
*** abregman has joined #openstack-infra20:57
clarkbfungi: ^ maybe you can look into that while I clenaup images?20:58
fungiclarkb: yep, digging now. thanks20:58
clarkbI suppose its possible I derped the image deletes in gra1 and it is trying to use an image that dne20:59
clarkbif that happaned just nodepool image-delete the image and force it to queue an upload20:59
fungiLaunchStatusException: Server 8062923f-0708-4d24-89e6-3b6bbfe74484 for node id: 3040 status: ERROR20:59
*** maishsk_ has joined #openstack-infra21:00
clarkbya I tried nova show and got nothing21:00
clarkbmaybe try manually booting an instnace that is otherwise the same as that one was?21:00
fungii'll check the gra1 list of images against nova image list21:00
*** maishsk has quit IRC21:00
*** maishsk_ is now known as maishsk21:00
*** pcaruana has quit IRC21:01
crinklenibalizer: what is haha nvm, the json api?21:01
*** davideagnello has quit IRC21:02
*** davideagnello has joined #openstack-infra21:02
nibalizercrinkle: ya, we don't have a json api on our mirrors21:02
nibalizerim not sure why I thought we would21:02
*** bpokorny has joined #openstack-infra21:02
openstackgerritMerged openstack-infra/project-config: Execute sbuild after relogin
crinklenibalizer: yeah that is what i figured, it didn't seem like the xml api worked21:02
*** rossella_s has quit IRC21:03
crinklenibalizer: that is why the gross regex21:03
fungiclarkb: that failure was for an ubuntu-trusty boot, and the ubuntu-trusty image in nodepool for gra1 is showing active in nova image-list output still21:03
*** weshay has quit IRC21:03
*** lucasagomes has quit IRC21:03
clarkbfungi: ok21:03
crinklenibalizer: is that something that could be enabled or is it just not supported?21:03
clarkbfungi: probably something on their end then.21:03
fungiso doesn't seem to be an image deleted out from under nodepool afaict. still digging for other possibilities21:03
*** rossella_s has joined #openstack-infra21:03
nibalizercrinkle: i have no idea21:03
nibalizeri think our mirrors are a dumb http server so we'd have to compute the json and spray it out into some directories21:04
*** fitoduarte has quit IRC21:04
*** mylu has quit IRC21:04
*** psilvad_ has quit IRC21:04
*** e0ne has quit IRC21:04
funginibalizer: crinkle: what our mirrors provide is what pypi calls it's "simple" api21:04
*** |-paul-| has joined #openstack-infra21:05
*** jcooley_ has quit IRC21:05
*** eharney has quit IRC21:05
fungiwhich basically means file indices under /simple/packagename21:05
*** rajinir has quit IRC21:05
*** trown is now known as trown|outtypewww21:06
crinklefungi: i made to try to make our puppet use the simple api on the mirrors instead of looking at, would love your feedback when you have some time21:06
openstackgerritIan Wienand proposed openstack/diskimage-builder: Fix disk usage report
pabelangerfungi: EmilienM: failed to initialize logging system due to a locale error: Invalid or unsupported charset:ANSI_X3.4-1968 when you run facter for puppet 421:08
pabelangerso, something related to locale21:08
*** hdd has quit IRC21:09
*** jamesmcarthur has quit IRC21:09
*** maishsk has quit IRC21:09
clarkbovh-bhs1 leaked iamges are being deleted now21:09
*** lucasagomes has joined #openstack-infra21:11
*** e0ne has joined #openstack-infra21:11
*** thorst_ has quit IRC21:12
fungilifeless: dstufft: just to confirm, there's no dry run option for pip install where you could do something like `pip install --upgrade --no-deps --dry-run somepackage` to find out what version of a package it _would_ install for that request, right?21:12
dstufftfungi: closest you can get is pip download21:13
*** maishsk has joined #openstack-infra21:13
fungilifeless: dstufft: wondering whether there's some way to exercise the index scan pip install would do without actually downloading/installing anything, to get the "latest" version of a package from a mirror only providing the simple api21:14
persiaA reputed rune is `virtualenv /tmp/venv; /tmp/venv/bin/pip install foo; rm -rf /tmp/venv `21:14
*** aeng has joined #openstack-infra21:14
dstufftfungi: wellllll21:14
*** ldnunes has quit IRC21:14
dstufftpip has no public API, but you could always just use the PackageFinder class if you wanted21:14
dstufftit might break on pip upgrades though, so I wouldn't ship it somewhere that you don't have control over21:15
fungithe "not downloading" part is more from a performance perspective, as the context here is a puppet module that determines whether it needs to upgrade a package or not21:15
dstufftpip list -o ?21:16
fungii guess the alternative would be to call pip install --upgrade every time, and only report "changed" when that exec actually installed something21:16
fungioh, that might do the trick21:16
fungicrinkle: ^ considered as an option?21:16
clarkbrax-iad in progress now too21:16
fungicrinkle: from pip list --help: -o, --outdated              List outdated packages21:16
crinklefungi: that might be better21:17
fungii suppose that also assumes a new enough pip version that pip list is provided21:17
*** tongli has quit IRC21:17
fungidstufft: do you recall when you implemented --outdated, or should i hunt in changelogs?21:17
*** thorst_ has joined #openstack-infra21:18
dstufftI don't recall exactly, I think it was prior to 1.5 tho21:18
*** salv-orl_ has joined #openstack-infra21:18
*** nwkarsten has quit IRC21:18
crinklefungi: we can generally assume we have latest pip on everything, right?21:18
fungii'd call that ancient enough history for our purposes21:18
*** e0ne has quit IRC21:18
fungicrinkle: on everything we build, yes21:19
nibalizeri think using a pip command to find the latest would be generally in line with how every other provider works21:19
*** mylu has joined #openstack-infra21:19
funginibalizer: yeah, that's what this seems to do21:19
fungifungi@dhole:~$ work/openstack/venvs/virtualenv/bin/pip list --outdated21:19
fungisetuptools (2.2) - Latest: 20.6.7 [wheel]21:19
*** Nikolay_St has joined #openstack-infra21:19
crinkleyes that would be way better, i didn't know about pip list (pip search was not doing the right thing)21:19
nibalizerif we say that we don't need this in puppet core and we just need it for us we can use whatever pip version we like21:20
nibalizercrinkle: i do think this hack was neat21:20
*** mylu has quit IRC21:20
fungiyeah, i'm certainly not opposed to the hack21:20
*** thorst__ has joined #openstack-infra21:21
fungi(haven't even finished looking through it, was just questioning assumptions from the commit message first)21:21
dstufftfungi: fwiw21:21
fungidstufft: thanks!21:22
dstufftlooks like it was added in 1.321:22
*** jsavak has quit IRC21:22
fungiwow, so ubuntu trusty debs even new enough to have this21:22
*** thorst_ has quit IRC21:22
*** salv-orlando has quit IRC21:22
dstufftI think uh, anything newer than Precise, Debian oldstable, etc era is good to go21:22
*** jsavak has joined #openstack-infra21:23
lifelessfungi: you can't deal with transitive deps without downloading21:23
fungilifeless: right, we don't expect to anyway21:23
nibalizerlifeless: i don't think that is required for this21:23
lifelessfungi: don't expect to transitive?21:23
*** mylu has joined #openstack-infra21:24
nibalizerya looking at the output of pip list --outdated it looks like exactly what we need21:24
fungilifeless: this is more from the perspective of "my puppet manifest says i should always automatically upgrade to the latest foopackage, so figure out if there's a newer one"21:24
clarkb165 leaked images in rax-dfw21:24
*** markvan has quit IRC21:24
nibalizerand if its in trusty, then maybe it might make it in to core after all21:24
dstufftat some point we need to add some --json or something to our commands21:24
nibalizerat least for puppet 421:24
dstufftI'm not sure that the output of pip list is stable :(21:24
*** mrmartin has quit IRC21:24
clarkbfungi: that makes meworry that we may need to do cleanup in our swift container too21:24
lifelessdstufft: its not a contract today, no.21:24
clarkbanyways dfw deletes in progress now21:25
fungilifeless: in this case puppet users don't, i think, consider ensure=>latest to necessarily mean "and also update all transitive dependencies even if this package hasn't updated"21:25
*** Sukhdev has joined #openstack-infra21:25
dstufftlooks like it's mostly stayed the same though21:25
lifelessfungi: ack, though I thought puppet was all about reproducability :)21:25
*** bpokorny has quit IRC21:25
*** thorst__ has quit IRC21:25
fungilifeless: normally i'd agree, but i don't think the pip provider has ever really had that guarantee since dependencies are open-ended21:26
dstufftoh no it hasn't, but close enough I think you can hack it21:26
*** bpokorny has joined #openstack-infra21:26
dstufftand I guess if all you care about is "should I upgrade", you can just test for the name of the thing being in the output21:26
dstufftrather than needing to parse out the version21:26
*** hdd has joined #openstack-infra21:26
*** markvan has joined #openstack-infra21:27
clarkbfungi: maybe you want to take a peak into the rax swift image containers to make sure we don't have 500 images in there all crufty21:27
pabelangerEmilienM: Will have to follow up with puppetlabs tomorrow and see what is going on. But cfacter is not happy21:27
zigofungi: lifeless: clarkb: The patch I just made to attempt for the build script to re-login after sbuild is setup, just failed. What do you suggest here, moving forward?21:27
EmilienM_ody: ^21:27
clarkbzigo: fix the failure?21:27
pabelangerEmilienM: do we need to revert to devstack-centos7?21:27
EmilienMpabelanger: no, the jobs are experimental21:28
zigoclarkb: The failure is because one is supposed to logout, and I don't know how this can be done with infra.21:28
pabelangerEmilienM: okay, I'll jump into puppetlabs tomorrow and see if we can get support for facter21:28
EmilienMpabelanger: cool21:28
clarkbzigo: I already told you how in my review...21:28
zigoOk, reading...21:28
fungizigo: yeah, clarkb and i both mentioned alternative solutions21:28
fungiin review comments21:29
annegentlefungi: or clarkb: or jeblair: or someone who knows about the Rackspace cloud account, can I chat with one of you privately for a minute?21:29
fungiannegentle: sure, i'm free21:29
annegentlefungi: thanks!21:29
*** mylu has quit IRC21:29
zigoclarkb: newgrp wont help21:29
clarkbzigo: why not?21:31
zigoclarkb: Hum... I can try!21:31
*** denisra_ has joined #openstack-infra21:31
clarkbalso I don't know what fungi's suggestion was but its proably good too21:31
*** ayoung has joined #openstack-infra21:32
zigoclarkb: He suggested using exec.21:32
zigoBoth of which I can try without a patch to infra.21:32
pabelangerclarkb: fungi: zigo: Heres how I handled the issue when I was trying to get packaging:
pabelangerI did it with puppet before jenkins connected21:32
anteayaclarkb fungi nice work on the nodepool cut over21:33
pabelangersee line #2021:33
*** hdd has quit IRC21:33
*** nwkarsten has joined #openstack-infra21:33
clarkbpabelanger: I don't think we should do it that way21:33
clarkbpabelanger: just like with package deps jobs should manage group membership directly21:33
zigopabelanger: I'm a puppet blind ... :/21:33
openstackgerritFei Long Wang proposed openstack-infra/project-config: Add Tempest gate job for Zaqar
*** denisra has quit IRC21:34
pabelangerclarkb: right, just giving another, other option :)21:35
clarkbya but I will -2 that option now :)21:35
clarkbwe shouldn't get this far into making jobs manage their own dependencies to add stuff like that21:35
zigoclarkb: Doing something like "newgrp - sbuild" will not modify the current env, it will just initialize a new interactive shell, no? Wont it just either wait forever, or exit back to the upper shell?21:36
*** tiswanso has quit IRC21:37
*** lucasagomes has quit IRC21:37
*** nwkarsten has quit IRC21:37
*** baoli has quit IRC21:37
openstackgerritColleen Murphy proposed openstack-infra/puppet-pip: Add new pip provider to use pypi mirrors
*** sridhar_ram has quit IRC21:37
zigoclarkb: I really don't see how it may help...21:38
*** baoli has joined #openstack-infra21:38
clarkbzigo: the first paragraph of the man page says exactly what happens21:38
zigoclarkb: I get it, though isn't this for interactive shell only?21:38
*** jcooley_ has joined #openstack-infra21:38
clarkbI don't think so21:38
*** jcooley_ has quit IRC21:39
*** tphummel has quit IRC21:39
zigoAh indeed... (I just tested it in a local script)21:39
zigoThanks, trying this.21:39
_odypabelanger: You noticed facter broken now on your failing expiremental job?21:40
*** cloudtrainme has quit IRC21:41
_odyAh.  Ok.  I see it.  I'll look into what might be happening.21:42
pabelanger_ody: failed to initialize logging system due to a locale error: Invalid or unsupported charset:ANSI_X3.4-196821:42
pabelanger_ody: when you invoke facter21:42
pabelanger_ody: have to run tonight, but will be working on it tomorrow too21:43
clarkbpabelanger: do we need to set the locale to C.utf-8 in our image builds?21:43
clarkbassuming thats xenial?21:43
pabelangerclarkb: this is centos-721:43
pabelangerclarkb: not explicitly, we should likely populate /etc/locale.conf to it21:43
pabelangersorry, have to run21:44
*** thorst_ has joined #openstack-infra21:44
*** lucasagomes has joined #openstack-infra21:45
*** weshay has joined #openstack-infra21:45
*** tphummel has joined #openstack-infra21:47
*** erlon has quit IRC21:47
*** Goneri has quit IRC21:47
*** thorst_ has quit IRC21:48
*** lucasagomes has quit IRC21:50
fungicentos 7 lacks C.UTF-821:50
fungifedora (22?) added it21:50
fungibut pretty much all distros are adding it by patching newer glibc21:51
fungiupstream is still waffling on whether to make it an actual locale iirc21:51
*** mylu has joined #openstack-infra21:52
*** Sukhdev has quit IRC21:52
clarkbopenstack image delete not working against vexxhost :/21:53
clarkboh wait21:54
clarkbis it paging21:54
clarkbya I think that is a major anti feature21:55
*** nwkarsten has joined #openstack-infra21:56
*** lucasagomes has joined #openstack-infra21:56
*** matt-borland has quit IRC21:56
*** regXboi has quit IRC21:58
*** denisra_ has quit IRC21:58
*** denisra_ has joined #openstack-infra21:58
clarkbI think it should do like git and direct to $PAGER if interactive21:59
zigoclarkb: As I expected, "newgrp - sbuild" didn't help!21:59
zigoclarkb: Any other idea?21:59
*** ybathia has joined #openstack-infra22:00
clarkbzigo: start a new login shell22:00
*** xyang1 has quit IRC22:00
clarkbbash --loging or whatever22:00
*** bpokorny_ has joined #openstack-infra22:01
*** bpokorny_ has quit IRC22:01
*** ddieterly has joined #openstack-infra22:01
*** ayoung has quit IRC22:01
*** bpokorny_ has joined #openstack-infra22:02
fungizigo: it doesn't look like you ran those commands under newgrp, you merely ran them after newgrp22:02
fungioh, i see, it's supposed to modify the current environment, not spawn a subshell22:02
*** nmagnezi has quit IRC22:02
fungizigo: nevermind, yeah i don't know that newgrp is going to pick up groups your current login predates22:03
fungiseems like it won't22:03
*** pvaneck has quit IRC22:03
nibalizerah, sad22:04
*** bpokorny has quit IRC22:04
*** jcooley_ has joined #openstack-infra22:05
*** ihrachys has quit IRC22:05
*** maishsk has quit IRC22:05
funginewgrp is working in my limited testing here. playing around with it a little more22:07
clarkbobservation time: we almost never ever leak snapshot builds22:08
fungialso you probably want to reset your group back to the original one after calling newgrp to the new one22:08
clarkbdib iamge uploads seem to leak at the rate of "lots"22:08
*** mylu has quit IRC22:08
clarkball of our cloud regions should have spring cleaned glance image lists22:08
clarkbI have no looked at the rax image swift container because I am scared22:09
greghaynesclarkb: I wonder if we should periodically delete non public images that the builders dont know about22:09
clarkbgreghaynes: I think first step is fix shade's handling of fialure22:10
clarkbthen ya if it continues to be bad we periodically clean them out22:10
fungioh, indeed newgrp does in fact (per my testing) create a new subshell22:10
openstackgerritArmando Migliaccio proposed openstack-infra/project-config: Make gate-neutron-dsvm-api job non-voting
*** briancline has quit IRC22:14
armaxanyone kind enough to have a look at ^22:15
armaxwe’re trying to put some sanity back into the Neutron CI until we get things sorted22:15
*** briancline has joined #openstack-infra22:17
anteayaarmax: +222:18
*** mikal has quit IRC22:18
armaxanteaya: thanks!22:18
*** mikal has joined #openstack-infra22:18
*** ddieterly is now known as ddieterly[away]22:18
anteayaI hope the spliting out of these test suites continues and is effective22:18
*** flepied has joined #openstack-infra22:19
armaxanteaya: we’re getting there22:19
*** camunoz has joined #openstack-infra22:19
anteayayay progress!22:19
anteayaarmax: good work22:19
*** ddieterly[away] is now known as ddieterly22:19
*** knikolla has quit IRC22:19
clarkbanother perspective. It is early in the cycle when little work gets done. YOu basically have 2 weeks to fix it while few notice22:20
clarkbmake it non voting and that 2 weeks means more regression potential22:20
*** jcooley_ has quit IRC22:20
*** Sukhdev_ has joined #openstack-infra22:21
*** flepied has quit IRC22:21
*** doug-fish has quit IRC22:23
*** briancline has quit IRC22:25
*** ayoung has joined #openstack-infra22:26
*** jcooley_ has joined #openstack-infra22:26
*** Sukhdev_ has quit IRC22:26
*** briancline has joined #openstack-infra22:27
*** jcooley_ has quit IRC22:27
*** jcooley_ has joined #openstack-infra22:27
*** yaume has joined #openstack-infra22:27
*** dimtruck is now known as zz_dimtruck22:28
*** rcernin has quit IRC22:29
*** ddieterly is now known as ddieterly[away]22:30
*** ddieterly[away] is now known as ddieterly22:30
*** yaume has quit IRC22:30
*** cloudtrainme has joined #openstack-infra22:31
anteayaclarkb: that is true, I'm willing to cut armax some slack here as he and I did have a very long, earnest and mutually beneficial discussion at the last neutron mid-cycle about
armaxclarkb: believe me, I am fully aware of the circumstances :)22:32
*** cloudtrainme has quit IRC22:32
anteayahe listened when I explained my personal concern about care of the gate and regressions and he took it to heart and spent some time learning more about elastic recheck and has been submitting some fingerprints for the elastic recheck repo22:32
anteayathank you for that armax22:32
*** angdraug has quit IRC22:32
armaxclarkb: but I think we’d want to rework some of the internal plumbings of the job and that requires us to make it non voting22:32
*** cloudtrainme has joined #openstack-infra22:32
armaxclarkb: I am thinking like moving gate hooks etc22:33
anteayaI feel armax is feeling caught between two uncomfortable places, not entirely of his own making, and is looking for a show of support as he tries to make things better22:33
pleia2anteaya: forgot to follow up yesterday, our wikimedia module is very much incomplete and is based on a 4 year old version of Ubuntu (12.04), so I'm not surprised that you're having dependency/missing packages issues22:33
anteayaso I'm inclined to show him some support22:33
anteayapleia2: awesome, thanks for letting me know :)22:33
fungithe incomplete part is most of what's held us back from rebuilding/upgrading it before now22:34
* pleia2 nods22:34
anteayapleia2: I'm keeping notes about package dependenices here:
pleia2anteaya: oh, great22:34
anteayaat this moment, I'm wondering how I can get my hands on a vm with a xenial image to test the puppet module on22:34
fungiin fact, the situation that server wound up in is one of the reasons we've gotten stricter about new services being fully automated before being in production22:35
pleia2anteaya: do you have the ability locally to spin up an image?22:35
anteayaas it is possible all the badness might just disappear, in a puff of smoke like22:35
anteayapleia2: I don't know22:35
*** annegentle has quit IRC22:35
pleia2anteaya: like, in a virtualbox or with kvm?22:35
anteayapleia2: I will try to figure out if I do tomorrow22:35
anteayayes I have kvm22:35
pleia2xenial hasn't been released yet, so it's early days for clouds to support it22:35
pleia2but you can grab a daily iso from the ubuntu dev folks22:35
anteayacool, I will consider that tomorrow's project22:35
anteayathank you22:36
anteayafungi: glad it was the bad example to create a better workflow22:36
pleia2sure, feel free to let me know if you have trouble finding the image22:36
anteayapleia2: thanks I will do22:36
anteayaprobably by the time I dig up my kvm notes again you will be online :)22:37
*** cloudtrainme has quit IRC22:37
*** tpsilva has quit IRC22:37
anteayabasically image magik is used by image scaler which requires ffmpeg22:37
anteayawhich went away in trusty and is back in xenial22:38
pleia2ah, yeah, ffmpeg is complicated22:38
anteayaso I really really would like to use a distro package for it22:38
fungii saw that in scrollback from yesterday and meant to reply22:38
armaxclarkb: is my knowledge still current when I believe that we can’t use depends-on between project-config and other openstack projects like neutron?22:38
anteayaI'm listening, fungi22:39
fungithe situation with ffmpeg packaging in debian (and transitively ubuntu) is (or was at least) complicated indeed22:39
* pleia2 nods22:39
clarkbarmax: depends on what you are wanting to change. You cannot do it with jenkins jobs because jenkins jobs must be vetted before they are updated22:39
clarkbarmax: but in theory other bits could be depends on'd22:39
fungistarted out with mpeg encoding patents possibly being enforceable and debian not wanting to get itself (or its users) sued for shipping the library22:39
anteayagood reason22:40
armaxclarkb: ack22:40
pleia2libav tends to be recommended ahead of ffmpeg even now, but feature parity is hard, and getting packages to switch over is too22:40
fungifraunhofer encoding patent specifically i think?22:40
fungiand yeah, thus was born the libavcodec fork in debian22:40
armaxclarkb: I am thinking in situations were a change in the pre and post hooks may need to go hand in hand with the hooks located in the project repo itself22:40
bkeroDealing with binary distros and linking with new filenames can be a big pain too22:40
armaxclarkb: maybe I am talking nonsense22:41
fungiwhich lagged behind over time and ffmpeg moved on, and also teh licensing/patent situation got clarified22:41
clarkbarmax: I think you just enable the hooks in the jobs then change them however you want in neutron22:41
pleia2fungi: yeah, at least for a while debian compilied without fraunhofer, I remember a bug report in Ubuntu that complained about it22:41
*** tonytan4ever has quit IRC22:41
clarkbarmax: I don't think that needs tight coupling between the two. Once you get the hooks enabled you should never need ot touch them again22:41
fungialso, for much of this time, hosted unofficial packages of ffmpeg and lots of stuff recompiled against ffmpeg22:42
armaxclarkb: I have seen past situations where we did not fully consider the execution of the jobs across all the branches22:42
armaxand that has burned us a bit22:42
* anteaya really enjoys storytime with fungi22:42
armaxnothing catastrophic though22:42
anteayaso sounds like we are all in agreement that a distro package for ffmpeg is preferred22:44
fungianteaya: yeah, so according to the ffmpeg package in xenial should be actual ffmpeg22:44
anteayafungi: wooooo, real ffmpeg!22:45
*** abregman has quit IRC22:45
pleia2yeah, and if we do a migration (rather than upgrade) it sounds like trying to go directly to xenial will make sense22:46
*** lakshmiS has quit IRC22:46
*** briancline has quit IRC22:46
*** gordc has quit IRC22:46
openstackgerritAnita Kuno proposed openstack-infra/puppet-mediawiki: Updates to move off precise
anteayapleia2: might as well22:47
fungii think debian jessie also wound up with actual ffmpeg packages too, not dummy packages depending on the forked libav22:47
fungior that might only be in jessie-backports22:47
anteayagetting the module in shape for trusty might be more work that for xenial22:47
anteayabut anyway, so far I'm hearing support for testing the module on xenial which will be my next step22:48
anteayathank you22:48
*** daemontool has joined #openstack-infra22:48
fungiahh, yeah just in jessie backports (stable-bpo)22:48
fungiso it'll be in stretch, barring any release-critical bugs which demand its removal from testing before then22:49
anteayais stretch the release name after jessie?22:49
anteayaokay great, thanks22:50
fungibut anyway, that's the debian ffmpeg story in a nutshell, minus years of drama on the debian-devel mailing list and elsewhere22:50
pleia2it's a purple octopus22:50
anteayafungi: I was picturing you dramatically waving your hands about22:50
anteayafungi: so thanks for not leaving all of it out22:50
anteayapleia2: I don't know the purple octopus22:50
*** |-paul-| has quit IRC22:50
pleia2toy story 3 character22:51
*** baoli has quit IRC22:52
anteayapleia2: nice :)22:52
anteayamorning jhesketh22:52
*** briancline has joined #openstack-infra22:52
*** mylu has joined #openstack-infra22:52
fungiyes, "stretch" was a stretchy purple octopus in toy story 322:54
fungijust as "jessie" was a cowgirl doll22:54
*** sdake has quit IRC22:54
*** rbrndt_ has quit IRC22:56
*** zz_dimtruck is now known as dimtruck22:56
*** sdague has quit IRC22:57
*** sdake has joined #openstack-infra22:58
clarkbfungi: look at the jobs launched per hour graph on zuul status22:58
clarkbfungi: barely a blip22:58
* clarkb calls this a success and stops thinking about computers for a bit22:59
fungiindeed, success!23:00
*** moravec has joined #openstack-infra23:00
*** dangers is now known as dangers_away23:00
fungiclarkb: we do indeed have many, many, many files in the images container, some dating back as far as august23:02
*** briancline has quit IRC23:02
*** BillCr5 has joined #openstack-infra23:02
fungiclarkb: those are only temporarily created as part of the upload process and should be removed at the end normally, yes?23:03
clarkbfungi: I think we can delete all of those except for those from the last 2 days or so23:03
clarkbfungi: yes23:03
clarkbideally by shde since it creates them23:03
fungimost of these show as 0 bytes23:03
clarkbhuh failures maybe?23:04
fungi198 files, 142.52 GB total23:05
clarkbfungi: I/we should actually make sure that the ssh host keys are working properly now23:05
fungithere's also a images_segments container23:05
fungiit has 2804 files, 2.66 TB total23:05
*** briancline has joined #openstack-infra23:07
fungioh, ugh, the contents of that are trees of folders nested about 5 deep23:07
clarkbdifferent rsa host keys on centos723:07
*** ybathia has quit IRC23:08
clarkbdifferent on ubuntu-trusty too23:08
clarkbso ya lgtm23:08
clarkbfungi: fun23:08
fungihas template-centos-7-1458125147/1458125129.391523/16780599808/1073741824 as a path under it with 16 segment files23:08
clarkbfungi: I think there is md5summing involved somewhere too23:08
clarkbwhich might be part of that pathing?23:08
greghaynesclarkb: hah, thats a lot better ;)23:08
*** Qiming has joined #openstack-infra23:09
*** akshai has quit IRC23:09
fungiclarkb: yeah, i'm not real sure how to go about using shade to clean this up without embarking on a fair amount of development23:10
clarkbfungi: ya... probably can be dealt with by applying hammer to mordred23:10
fungior looking into setting expiration policies?23:11
*** dingyichen has joined #openstack-infra23:11
clarkbfungi: I think shade should just delete the image from swift after it is successfully imported to glance23:12
fungithe main thing i want to know is whether these are actually ephemeral (temporary staging location for glance import task) or reused (nova actually being instructed by glance to pull from that url into its cache when needed)23:12
*** sdake has quit IRC23:12
clarkbsame as shade should delete the image from glance when uploads fail23:12
*** moravec has quit IRC23:12
clarkbfungi: I am almost positive it is temporary staging because the glance import task takes forever23:12
clarkbimplying all the bits are copied23:12
*** jsavak has quit IRC23:13
fungiif that's the case, then it should just be safe to blow all of this away as long as no rax image uploads are in progress23:13
fungithough their dashboard also doesn't present a recursive delete option23:13
clarkbfungi: sudo -H -u nodepool nodepool image-list | grep rax | grep building shows nothing23:13
greghaynesif only the code was upstream then we'd know...23:13
*** mriedem has joined #openstack-infra23:14
fungii think the only "easy" way to clear the entire images_segments container's thousands upon thousands of nested directories and files is to delete the container and recreate it23:15
*** tosky has quit IRC23:15
clarkbfungi: shade will recreate it by itself too I think23:16
clarkbfungi: maybe get mordred to confirm that his knowledge of this system is that it is a copy23:16
clarkbthe nwe delete23:16
fungithough it's also possible that the images container's zero-byte files are really pointers into those images_containers directories, and that swift would be smart enough to remove them by reference?23:16
fungier, into the images_segments directories23:16
clarkboh right because the images are too big to be a single swift object23:16
clarkbso it does the magical large objcet support stuff however that works23:16
openstackgerritMerged openstack-infra/irc-meetings: Add Nate Johnston as neutron-qos co-chair
fungiit might be that if i just delete all the files from the images container, everything referenced in the images_segments container will also be cleaned up23:17
notmynameswift doesn't keep any reference counting, so it can't automatically remove a large object manifest if the segments are missing23:17
funginotmyname: thanks for confirming. that's a "no" then ;)23:17
notmynamefungi: but yeah, that. if you delete a large object, it can delete the referenced segments for you23:17
fungigot it, so other way around won't work (deleting segments won't clear the pointer)23:18
notmynameyou can't remove the segments and have swift remove a manifest. right23:18
fungibut removing the pointer will let it clean up the segments23:18
notmynamelet me find the doc on that23:18
fungiokay, in that case this is easy23:18
clarkbfungi: but ya I would wait to have modred or rax confirm that those are temp files and not backing the nova use of the images directly23:18
clarkbbecause it will be sad times if we kill all our images in rax23:18
fungiand yeah, in this case as long as mordred confirms it's safe to do so, we can just blow away everything in these23:19
openstackgerritMerged openstack-infra/irc-meetings: Update Horizon meeting times
funginotmyname: thanks! that's useful. exactly what i was hoping for23:20
*** flwang has left #openstack-infra23:20
*** aysyd has quit IRC23:21
*** jcooley_ has quit IRC23:21
notmynamefungi: there's 2 kinds of manifests in swift. let me make sure that works on both23:22
*** briancline has quit IRC23:22
*** salv-orl_ has quit IRC23:24
*** briancline has joined #openstack-infra23:25
*** ayoung has quit IRC23:26
*** jamesmcarthur has joined #openstack-infra23:27
*** tqtran has quit IRC23:27
notmynamefungi: hmm...bad news. that only works on static large objects (SLOs). to be fair, that's what glance *should* be using. but the ?multipar-manifest=delete doesn't work to recursively delete dynamic large objects (DLOs)23:28
*** markvoelker has quit IRC23:29
funginotmyname: is there a particular format to the segment trees which can be used to identify slo vs dlo?23:29
notmynamefungi: based on the way the segments are stored? no. but if you do a GET with ?multipart-manifest=get, you'll see the manifest file itself23:30
funginotmyname: oh, excellent. thanks again23:31
*** sridhar_ram has joined #openstack-infra23:31
notmynamefungi: and a DLO has an empty body with the x-object-manifest header. an SLO has a json body and the x-static-large-object: true header23:31
fungiawesome. i'll query this one here in a bit23:32
*** ddieterly is now known as ddieterly[away]23:33
*** jamesmcarthur has quit IRC23:33
*** ddieterly[away] has quit IRC23:33
fungioh, in fact, the header details on one in the rax dashboard also show that23:33
*** briancline has quit IRC23:34
fungiunfortunately: X-Object-Manifest images_segments/template-ce...23:34
*** jamielennox|away is now known as jamielennox23:36
*** ddieterly has joined #openstack-infra23:36
*** nwkarsten has quit IRC23:37
fungias an experiment, i've tried deleting one dlo manifest through the rackspace dashboard, and am watching to see if the segments tree for it eventually vanishes23:38
fungiit was from march, so shouldn't impact anything current anyway23:38
*** mylu has quit IRC23:38
*** briancline has joined #openstack-infra23:38
fungiso far it's still there23:39
*** bpokorny has joined #openstack-infra23:40
*** bpokorny_ has quit IRC23:40
*** ZZelle_ has quit IRC23:41
*** Jeffrey4l_ has quit IRC23:42
anteayatonyb: does this have any hope of merging before the end of next week?
anteayafungi: sorry was on the phone to my parents, they fly home tomorrow23:44
anteayasounds like they are in good shape for the flight23:44
*** mylu has joined #openstack-infra23:45
tonybanteaya: 95% by the end of *next* week.23:45
*** xarses has quit IRC23:45
fungistill there. i don't think rackspace's dashboard integration for swift cleans up dlo segments either23:46
*** bpokorny has quit IRC23:46
tonybanteaya: we need or land *or* for me to change irc-meetings to pull from git rather than pypi23:46
* tonyb will publish that chnage today and ask ttx to +2 it pending resolution on 30254323:47
*** bpokorny has joined #openstack-infra23:47
anteayatonyb: thank you23:48
anteayatonyb: let me know if I can do anything to support23:48
anteayatonyb: go you!23:48
*** akshai has joined #openstack-infra23:48
*** ddieterly is now known as ddieterly[away]23:48
*** gildub has joined #openstack-infra23:48
mordredfungi: reading23:49
anteayadhellmann: hedgehog cafe!
mordredfungi, clarkb: I actually have no clue if deleting the swift objects has any effect on the image23:50
mordredit's unclear and one thing I do not like about task-import23:50
mordredhowever, I _think_ that someone told me at one point that glance does, in fact, copy the image from the user's swift into its own storage area23:50
clarkbmordred: ok, did you see the other issue with 2 step glance image uploadss? we should delete the imgae if upload fails23:50
clarkbbecause glance does not23:50
mordredclarkb: with 2-step PUTs?23:51
fungido we know whether rackspace is using the default import task (such that we could infer this from public source code)?23:51
fungior who we can ask?23:51
clarkbmordred: using glance you do an aimge create then an image upload. When the image upload fails we need to delete the image23:51
mordredfungi: rosmaita is the person to ask23:51
clarkbmordred: we had somewhere around 600 leaked image23:51
*** moravec has joined #openstack-infra23:51
mordredclarkb: neat!23:51
clarkbmordred: the vast majority (over 95% were uploads not snapshots)23:52
clarkband many were in a SAVING state which I think means that the upload for them failed and we never cleaned them up23:52
mordrednotmyname: fwiw, these are not Large Objects created by glance -- we upload them23:53
mordrednotmyname: so we can upload them differently if there is a better way to upload them23:53
fungimordred: oh, shade actually chooses dlo explicitly for this?23:53
mordredshade explcitly uploads using SwiftService23:53
fungisounds like using slo instead of dlo might make it easier to clean up after the fact if something goes bad23:54
fungisince you don't have to perform your own mapping from the manifest header back to the segment files23:54
*** akshai_ has joined #openstack-infra23:54
mordredswiftclient.service.SwiftUploadObject(source=filename, object_name=name)23:56
*** Sukhdev_ has joined #openstack-infra23:57
*** akshai has quit IRC23:58
notmynamemordred: ah ok. so SLOs are almost always better to use. you should be doing that. they're explicit, faster, and had the nice recursive delete feature23:58
*** markvoelker has joined #openstack-infra23:59
notmynamemordred: and a cool new thing is that you can actually reference *ranges* of other objects in a SLO now23:59
openstackgerritColleen Murphy proposed openstack-infra/puppet-pip: Add new pip provider to use pypi mirrors

Generated by 2.14.0 by Marius Gedminas - find it at!