Thursday, 2016-09-01

*** baoli has quit IRC00:00
openstackgerritMerged openstack-infra/shade: Revert per-resource dogpile.cache work
*** signed8bit_Zzz is now known as signed8bit00:08
ianwI've declared bankruptcy on scrollback today, so not sure I'm much help :)00:08
clarkbianw: uh tl;dr is funny networking in rackspace to the git mirror. Lots of connections being killed under haproxy both on the front and backends. So I filed a ticket after much debugging00:09
clarkbianw: networking-vpp clogged up the tubes again and has been asked to wait until dhellmann is done with release things before continuing00:09
ianwclarkb: ahh, so that was the issue.  jhesketh and I were looking yesterday afternoon, that weird situation where nothing seems to be wrong but something is :)00:10
clarkbianw: osic-cloud8 has a weird floating ip setup for our mirror so to start we are just going to use the mirror in cloud1 (by creating dns records for cloud8 that point at cloud1)00:10
clarkbianw: ya its a fun one whatever it is00:10
clarkbnodepool is now running the shade fixes for dual stack networking00:10
jheskethclarkb: good to know, thanks for digging :-)00:11
clarkband we have enabled infracloud and internap mtl0100:11
clarkbI think at least some of these things got status logged00:11
clarkbI am going to restart the nodepool builder now so that it can learn how to talk to osic-cloud8 and upload images there00:12
ianwclarkb: cool, thanks for the update :)00:14
*** katyafervent_awa has quit IRC00:17
*** rakhmerov has quit IRC00:18
clarkbcloud8 is still set to max servers 0 so should just upload images then idle00:18
*** Hal1 has quit IRC00:19
*** Sukhdev has joined #openstack-infra00:20
*** baoli has joined #openstack-infra00:21
*** signed8bit is now known as signed8bit_Zzz00:23
*** rakhmerov has joined #openstack-infra00:23
*** katyafervent_awa has joined #openstack-infra00:25
*** watanabe_isao has joined #openstack-infra00:25
*** sputnik13 has quit IRC00:27
ianwclarkb: btw you were right about growroot on f24 -> : return status of sfdisk changed00:27 bug 1371761 in util-linux "sfdisk return code breaks growpart" [Unspecified,Post] - Assigned to kzak00:27
bkeroianw: i filed a bug/ml post on util-linux like a year ago for that00:28
*** sflanigan has joined #openstack-infra00:28
bkeroThat didn't get fixed in the release? *sigh*00:28
*** Julien-zte has joined #openstack-infra00:29
ianwbkero: not sure if it's the same issue?  this one went in
ianwi'm really not sure how we're the only people who noticed00:30
*** baoli has quit IRC00:30
bkeroThere's probably a kludge in libguestfs that works around this00:31
*** jamesdenton has joined #openstack-infra00:31
clarkbianw: it seems like we have a fairly consistent set of failure modes now. init + cloud-init/glean fail to network, growroot fails to growroot, random new firewall software ignores the rules I tell it, new package manager is cranky00:31
clarkbrinse and repeat :)00:31
bkeroLooks to be an adjacent issue. Mine was that the ioctl for sfdisk was failing because loopback device00:31
openstackgerritMerged openstack-infra/system-config: Log gear at debug level on nodepoold
ianwclarkb: yeah.  on my todo list is to check that growroot worked in our setup scripts.  the problem was that RAX worked, as there was just enough space.  and when i was running experimental with just one node, i didn't get out to other providers00:32
ianwbut once things started getting heterogeneous ...00:33
*** signed8bit_Zzz is now known as signed8bit00:33
ianwbut, pabelanger we should still see why out-of-disk lead ansible to hang around for it's full 3 hour timeout00:34
*** Hal1 has joined #openstack-infra00:34
clarkbianw: if I am going to guess its doing a write that blocks despite not enough disk and just never returns?00:35
*** kzaitsev_mb has joined #openstack-infra00:35
clarkbhrm though that should ENOSPC00:35
*** baoli has joined #openstack-infra00:35
*** thorst has joined #openstack-infra00:35
*** zhurong has joined #openstack-infra00:36
*** pvaneck has quit IRC00:36
*** tphummel has joined #openstack-infra00:37
ianwclarkb: it was something slightly weirder in that console.html showed everything finishing up, but there was an ansible copy process that got stuck.  i'll have to dig out the logs, i should make a note for posterity00:37
*** kzaitsev_mb has quit IRC00:40
ianwahh, looking at the logs, it's zuul_runner that's behaving odd in this case.  that makes more sense, being our custom bit00:42
*** thcipriani is now known as thcipriani|afk00:43
*** signed8bit has quit IRC00:44
pabelangerianw: ya, I haven't had a chance to dig into fedora24 yet00:44
pabelangerclarkb: cool, osic-cloud8 images already uploaded. We can work on launching servers tomorrow00:46
ianwpabelanger: so i remember later -> .  maybe something in the bg causing ssh exit to hang?00:46
pabelangerianw: is it possible git clone is failing?00:47
pabelangerianw: because we had the same issue today in tripleo-test-cloud-rh100:47
pabelangerianw: not failing, hanging00:47
ianwpabelanger: possibly ... the workspace-setup-new is 0 bytes.  i'm guessing that means whatever output never got flushed to it, rather than it never ran00:48
ianwso what's acutally going on ... shrug?00:48
*** mtanino has quit IRC00:49
*** ociuhandu has joined #openstack-infra00:49
pabelangerianw: also, I don't think it is an SSH issue, because ansible async will poll the server every 10 seconds, and usually ansible will bark is SSH connection fails00:49
pabelangerianw: ya, this looks like what I seen in tripleo-test-cloud-rh1 today, if you get into that node, I suspect you'll see hung git clone process00:50
pabelangerfrom devstack-gate00:50
*** gyee has quit IRC00:51
*** vinaypotluri has quit IRC00:52
*** Goneri has joined #openstack-infra00:54
*** Sukhdev has quit IRC00:55
*** sarob has quit IRC00:55
ianwmtreinish: if around, have some questions on
*** spzala has quit IRC01:02
ianwparticularly what devstack.subunit it's trying to pick up01:02
*** kzaitsev_mb has joined #openstack-infra01:06
*** ociuhandu has quit IRC01:08
*** priteau has joined #openstack-infra01:11
*** Sukhdev has joined #openstack-infra01:12
*** Sukhdev has quit IRC01:13
*** priteau has quit IRC01:16
openstackgerritPaul Belanger proposed openstack-infra/project-config: Revert "Revert "Disable rax-iad due to launch failure rate""
pabelangerianw: ^if you don't mind a +A, I forgot to disable that earlier today. Were still having issues in rax-iad01:17
pabelangerI was able to reproduce the issue manually, so we can keep the region offline until we know the fix01:18
*** rossella_s has quit IRC01:18
*** chlong has joined #openstack-infra01:18
*** shashank_hegde has quit IRC01:18
ianwpabelanger: LGTM01:18
*** rossella_s has joined #openstack-infra01:19
*** tqtran has quit IRC01:19
*** aeng has quit IRC01:20
pabelangerianw: thanks01:20
ianwwho is responsible for stackviz?01:20
pabelangerianw: I think timothyb8901:26
*** yanyanhu has joined #openstack-infra01:27
ianwpabelanger / timothyb89: cool ... well i'm not sure what it thinks it's doing during grenade runs, but i'm pretty sure it's not doing it01:27
*** baoli_ has joined #openstack-infra01:28
*** baoli has quit IRC01:28
*** baoli has joined #openstack-infra01:30
*** esp has quit IRC01:32
*** baoli_ has quit IRC01:33
*** salv-orlando has joined #openstack-infra01:34
*** Benj_ has joined #openstack-infra01:34
openstackgerritMerged openstack-infra/project-config: Revert "Revert "Disable rax-iad due to launch failure rate""
*** changzhi has joined #openstack-infra01:37
*** spzala has joined #openstack-infra01:38
*** Benj_ has quit IRC01:41
*** hockeynut has quit IRC01:43
*** spzala has quit IRC01:43
*** salv-orlando has quit IRC01:44
*** spzala has joined #openstack-infra01:45
*** sarob has joined #openstack-infra01:49
*** esp has joined #openstack-infra01:58
*** woodster_ has quit IRC01:59
*** changzhi has quit IRC02:00
*** sarob has quit IRC02:00
*** itisha has quit IRC02:00
*** ijw has quit IRC02:01
*** zshuo has joined #openstack-infra02:02
*** Apoorva has quit IRC02:03
*** changzhi has joined #openstack-infra02:04
*** spzala has quit IRC02:05
*** apetrich has quit IRC02:05
*** esp has quit IRC02:05
*** sdake has quit IRC02:06
*** apetrich has joined #openstack-infra02:07
*** esberglu has joined #openstack-infra02:09
*** gildub has quit IRC02:10
*** cody-somerville has joined #openstack-infra02:13
*** yamamoto_ has joined #openstack-infra02:14
*** hichihara has joined #openstack-infra02:14
timothyb89ianw: what's the issue?02:17
ianwtimothyb89: have a look at 2016-08-31 07:32:32.699 in
ianwtimothyb: i'm working on a refactor of bits of this anyway, it's incredibly hard to understand IMO02:18
*** tqtran has joined #openstack-infra02:18
*** thorst has quit IRC02:18
*** thorst has joined #openstack-infra02:19
timothyb89ianw: ah, hmm. I thought I had the paths for grenade set correctly but I guess not :)02:20
*** nstolyarenko has joined #openstack-infra02:21
timothyb89ianw: I'll make sure to take a look at the devstack-gate bits when I'm back in the office tomorrow02:21
*** tqtran has quit IRC02:22
*** fguillot has quit IRC02:23
*** nstolyarenko has quit IRC02:25
*** thorst has quit IRC02:28
*** edtubill has joined #openstack-infra02:28
*** dimtruck is now known as zz_dimtruck02:30
openstackgerritChangcheng Intel proposed openstack-infra/jenkins-job-builder: update base_email_ext to adapt Email-ext plugin
*** Goneri has quit IRC02:35
*** vhosakot has joined #openstack-infra02:37
*** mriedem has quit IRC02:39
*** jamielennox|away is now known as jamielennox02:41
*** salv-orlando has joined #openstack-infra02:43
*** aeng has joined #openstack-infra02:44
*** armax has quit IRC02:45
*** zz_dimtruck is now known as dimtruck02:47
*** vhosakot has quit IRC02:48
*** Sukhdev has joined #openstack-infra02:48
*** gouthamr has quit IRC02:49
*** salv-orlando has quit IRC02:50
*** amotoki has joined #openstack-infra02:51
*** salv-orlando has joined #openstack-infra02:52
*** salv-orlando has quit IRC02:57
*** dimtruck is now known as zz_dimtruck03:02
*** esp has joined #openstack-infra03:03
*** docaedo has quit IRC03:03
*** armax has joined #openstack-infra03:03
*** cody-somerville has quit IRC03:05
*** amotoki has quit IRC03:08
*** armax has quit IRC03:08
*** cody-somerville has joined #openstack-infra03:11
*** docaedo has joined #openstack-infra03:11
*** esberglu has quit IRC03:12
*** vinaypotluri has joined #openstack-infra03:14
*** cody-somerville has quit IRC03:16
*** yamahata has quit IRC03:17
*** amotoki has joined #openstack-infra03:21
*** tphummel has quit IRC03:23
*** zz_dimtruck is now known as dimtruck03:23
*** thorst has joined #openstack-infra03:26
*** amotoki has quit IRC03:30
amrithdid something just hiccup in zuul?03:32
amrithI had a recheck running on a review
amrithand it seems to have vanished without a trace03:32
*** thorst has quit IRC03:33
*** shashank_hegde has joined #openstack-infra03:33
*** chem|off has quit IRC03:34
clarkbits there... the recheck was after you asked?03:34
clarkbI am confused03:34
clarkbthe reverify 3.5 hours ago reported. then you just rechecked and its queued03:35
*** amotoki has joined #openstack-infra03:35
clarkbfrom ehat I can see its all working as expected03:35
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: Fix devstack subunit output
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: [WIP] Refactor devstack log copying
*** vikrant has joined #openstack-infra03:36
ianwtimothyb89: ^ i'm thinking something like this.  rolling stackviz into the devstack processing part of 364046 would probably remove the confusion03:36
timothyb89ianw: oh, cool, that would be much better03:38
amrithclarkb I see it now03:39
*** sarob has joined #openstack-infra03:39
amrithafter I refreshed my screen a couple of times03:39
amrithI just requested a recheck03:39
amrithsomething weird03:39
amrithfor sure03:39
amrithpetr requested the reverify at 8:0603:39
amrithit failed at 9:5003:39
clarkband that ran and reported fine03:39
amrithbut at 11:30 I saw nothing03:40
amrithjust the previous gate03:40
clarkbthen you did a recheck and it worked fine03:40
clarkbyou didntcomment untill:3303:40
amrithand on zuul nothing03:40
amrithI was refreshing a couple of times03:40
clarkbyes and it was in zuul at that point03:40
amrithon my browser03:40
clarkbso worked fine03:40
clarkbyou were saying ut wasnt queued before you commented03:40
amrithfrom about 10:30 to 11:30, I saw nothing in review.openstack.org03:40
clarkbI think maybe you just didnt get the comment to poat in gerrit as quickly as you thought03:41
amrithwhich is when I posted the question here on IRC :)03:41
amrithwell it is running again03:41
clarkbwhat woukd there have been to see?03:41
amriththat it failed at 9:50pm03:41
amrithall I was seeing was the previous success03:42
amrithso, check passed and it went to gate. and failed03:42
amrithpetr reverified03:42
amriththen it failed in check03:42
amriththat failure was at 9:50pm03:42
clarkbyes and from what I see thats all rwcorded properly03:42
clarkbthen you rechecked and it restarted jobs like asked03:43
amrithyes, except from about 10:30 to 11:30 I've been refreshing my screen, and checking zuul03:43
amrithand saw nothing :)03:43
amrithMy guess is that after I posted the recheck03:43
amrithsomething happened03:43
clarkbI wasnt able to check that but when I did it worked.any chance you have a proxy being overzealous with caching?03:43
amrithnot that I know of, I'm sitting at home03:44
amrithcomcast doesn't typically cache this stuff, I don't think03:44
openstackgerritChangcheng Intel proposed openstack-infra/jenkins-job-builder: [Don't Merge]update base_email_ext to adapt Email-ext plugin
amrithI don't use a proxy at home for sure03:44
amrithbut, zuul appears to be quite a memory hog03:44, the site that is03:44
amrithit makes firefox limp03:44
clarkbbut zuul isnt what we arw talking about we are talking about comments in gerrit03:45
amrithviewed through a browser that maybe the culprit03:45
clarkband yes comcast shouldnt proxy that for you03:45
*** Srinu has joined #openstack-infra03:46
clarkbmaybe you had toggle ci toggled?03:46
clarkbthats actually lrobably the most likely cause03:46
amrithtoggle CI shouldn't impact the middle of the screen03:46
amrithwhich shows the results03:47
amrithit only shows the stuff in the history03:47
clarkbthat I dont kniw. its a mess of hacky js to parse the gerrit03:47
clarkbI tend to rely in the actual comments03:47
amrithbelow the CR +2's, Verified and Workflow is the jenkins check and gate output03:47
amrithand that didn't refresh03:47
amrithI didn't even look at the history03:47
clarkbalso if you were looking at an old patchset that also affects the table03:47
*** akshai_ has quit IRC03:48
clarkbit wont update like you expect03:48
amriththis section:
amrithcircled in the image03:49
amrithoh, an f5 won't update that section?03:49
clarkbif you are on an old patchset it gets weird I think03:49
amriththat I did not know. maybe I should just navigate away from the review and back03:49
amriththat could be03:49
amrithI could, maybe have been on previous patch set03:50
clarkbif you refresh on thr latest patchset it shoukd be fine with a hardrefresh at least03:50
amrithdon't recall03:50
amrithgood to know03:50
amrithin future just navigate away and come back03:50
clarkbI dont know enough about the js details to know if a soft refresh is enough03:50
amrithit was 10:30, nothing much has worked today03:50
amrithsomeone said something about mercury going retrograde and causing all the computrons to spin in the wrong direciton03:50
amrithour stable branches (both) just died inexplicably in the past couple of days; just realized it03:51
Srinuanyone face this issue.
Srinuplease help me.03:51
amrithbut, on the plus side, the helical inclined plane worked and pulled the cork out of the bottle just fine03:51
amrithhi Srinu03:51
Srinuamrith: hi. did you saw my question03:52
clarkbSrinu: that log points you at thr other log files for specificsyou will need to look at them to determine what is happening03:52
clarkbdo you have a link to the job logs?03:52
amrithyes Srinu .. as clarkb says, the error messages are pointing you to the right log file03:52
amrithwhat's the review #?03:52
*** salv-orlando has joined #openstack-infra03:53
amrithdid he go away?03:53
Srinuclarkb,amrith: while the running the block storage tests cases it is killed and then copying logs03:54
amrithwas this in the gate/CI03:54
amrithor on your local machine?03:54
amrithlooks like CI/gate to me03:54
Srinuamrith: ci03:55
amrithok, what's the review #03:55
amrithor as clarkb said, the link to the file where you got the stuff that you put in paste03:56
*** salv-orlando has quit IRC03:56
amrithclarkb, what tz are you in?03:56
amrithwondered if you were a night owl too; it is getting close to being tomorrow now.03:57
Srinuamrith,clarkb: please check this.
* jlvillal thinks clarkb should be off work by now :)03:57
amrithSrinu, that won't help03:57
amrithwhat's the review #03:57
amrithor a link to the logs03:57
amrithjust a URL would do ...03:57
amrithor is this a private CI?03:57
amrithwhich we can't get to?03:58
clarkbamrith screaming toddlers keeping me awake03:58
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: Update bashate to 0.5.0
Srinuamrith: it is not a patch. this error is coming in jenkins ci(private).03:58
EmilienMhi, I'm waiting for this review to release tripleo newton-3:
tristanCttx: can you please confirm the proposed schedule for upcoming election: (rendered from ?03:58
amrithclarkb, it is quiet here, almost midnight03:58
EmilienMif you're project-config core, please look at this patch when you can03:58
amrithsorry Srinu can't tell from that error; something took too long, someone got angry and killed it. there are lots of books and movies with the same story.03:59
clarkbSrinu: I think that means you timed out03:59
amrithsome test had a 600s timeout03:59
clarkbSrinu: try increasing the timeout or make it run faster03:59
amrithand your test didn't run in time ...03:59
amrithgame over03:59
Srinuamrith.clarkb: thank you03:59
*** rlandy has quit IRC03:59
*** amotoki has quit IRC04:01
*** Srinu has quit IRC04:02
*** cody-somerville has joined #openstack-infra04:05
*** spzala has joined #openstack-infra04:05
*** jamielennox is now known as jamielennox|away04:05
openstackgerritMerged openstack-infra/project-config: tripleo-ui: add missing jobs for release management
amrithhmm, clarkb is there a simple way to look at the history of a particular job in the CI? project=openstack/trove-integration, job=gate-trove-functional-dsvm-mysql-mitaka. I looked in openstack-health ( but it seems to only have data through 8/19.04:08
*** Sukhdev has quit IRC04:09
*** mcarden has quit IRC04:09
clarkbamrith: the three places are the health dashboard and elasticsearch04:09
*** changzhi has quit IRC04:09
*** alexey_weyl has joined #openstack-infra04:09
amrithok, thx. let me look at the other two04:09
*** alexey_weyl has left #openstack-infra04:09
*** vikrant has quit IRC04:09
*** alexey_weyl has joined #openstack-infra04:09
alexey_weylHi Guys,04:09
alexey_weylPlease approve this change04:10
*** vikrant has joined #openstack-infra04:10
*** spzala has quit IRC04:10
*** yamahata has joined #openstack-infra04:12
amrithwow! this graphite is cool stuff04:13
*** mcarden has joined #openstack-infra04:13
clarkbamrith: there is also that is a different frontend to the same data04:14
clarkbwe have a grafyaml config somehwere that you can write out cof igs for dashboard in04:14
amrithwow, awesome04:15
amrithdidn't know I could do this ...04:15
amrithdoes one have to sign in with lp credentials or some other?04:16
openstackgerritMerged openstack-infra/project-config: move tripleo scenario jobs to check pipeline, non-voting
*** amotoki has joined #openstack-infra04:16
clarkbno its all public data04:16
clarkbread only04:16
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: Fix devstack subunit output
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: [WIP] Refactor devstack log copying
alexey_weylCan you please check this change:04:18
amrithg'night clarkb .. I just pushed a change to revert to the last point where stable passed, will see what tomorrow (crap: today) brings.04:19
*** alexey_weyl has quit IRC04:20
*** cody-somerville has quit IRC04:20
*** dimtruck is now known as zz_dimtruck04:22
*** yamamoto_ has quit IRC04:23
*** baoli has quit IRC04:24
*** edtubill has quit IRC04:24
*** esp has quit IRC04:25
*** shashank_hegde has quit IRC04:26
openstackgerritMerged openstack-infra/project-config: Vitrage tempests
openstackgerritIan Wienand proposed openstack-infra/project-config: Run bashate test over devstack-gate too
*** thorst has joined #openstack-infra04:30
*** links has joined #openstack-infra04:32
*** asselin__ has joined #openstack-infra04:34
*** krtaylor has quit IRC04:34
*** rajinir has quit IRC04:35
*** jraim has quit IRC04:36
*** maximov has quit IRC04:36
*** jraim has joined #openstack-infra04:36
*** maximov has joined #openstack-infra04:37
*** esp has joined #openstack-infra04:37
openstackgerritMasayuki Igawa proposed openstack-infra/irc-meetings: Update the QA team meeting chair
*** thorst has quit IRC04:37
*** Illumitardi has joined #openstack-infra04:37
*** asselin_ has quit IRC04:37
*** edwarnicke has quit IRC04:37
*** amit213 has quit IRC04:37
*** Ng has quit IRC04:38
*** NobodyCam has quit IRC04:38
*** scottda has quit IRC04:38
*** ogelbukh has quit IRC04:38
*** zhiyan has quit IRC04:38
*** nikhil has quit IRC04:38
*** samueldmq has quit IRC04:38
*** betherly has quit IRC04:38
*** fmccrthy has quit IRC04:38
*** vdrok has quit IRC04:38
openstackgerritMerged openstack/diskimage-builder: Explain difference between two envvars
*** yamamoto_ has joined #openstack-infra04:39
*** zz_ja has quit IRC04:39
*** shashank_hegde has joined #openstack-infra04:39
*** senk has joined #openstack-infra04:39
*** scottda has joined #openstack-infra04:40
*** amit213 has joined #openstack-infra04:40
*** cody-somerville has joined #openstack-infra04:40
*** psachin has joined #openstack-infra04:40
*** nikhil has joined #openstack-infra04:40
*** zhiyan has joined #openstack-infra04:40
*** vdrok has joined #openstack-infra04:40
*** betherly has joined #openstack-infra04:40
*** fmccrthy has joined #openstack-infra04:41
*** edwarnicke has joined #openstack-infra04:41
*** ogelbukh has joined #openstack-infra04:41
*** NobodyCam has joined #openstack-infra04:41
openstackgerritIan Wienand proposed openstack/diskimage-builder: Disabled IPv6 privacy extensions
*** Ng has joined #openstack-infra04:42
*** zz_dimtruck is now known as dimtruck04:42
*** yamahata has quit IRC04:42
*** samueldmq has joined #openstack-infra04:42
*** dtroyer has quit IRC04:44
*** dtroyer has joined #openstack-infra04:44
*** zz_ja has joined #openstack-infra04:45
*** esp has quit IRC04:45
*** pt_15 has quit IRC04:46
*** jbryce has quit IRC04:46
*** sarob has quit IRC04:48
*** Illumitardi has quit IRC04:48
*** yamamoto_ has quit IRC04:49
*** jbryce has joined #openstack-infra04:52
*** maishsk has joined #openstack-infra04:54
*** yamahata has joined #openstack-infra04:54
*** roxanaghe has quit IRC04:56
*** pcaruana has quit IRC04:57
*** maishsk has quit IRC04:58
*** kzaitsev_mb has quit IRC04:59
*** pgadiya has joined #openstack-infra05:02
*** salv-orlando has joined #openstack-infra05:03
*** ggherdov- has quit IRC05:03
*** dougwig has quit IRC05:03
*** asingh has quit IRC05:03
*** ediardo has quit IRC05:03
*** hrybacki has quit IRC05:03
*** agordeev has quit IRC05:03
*** warpc has quit IRC05:03
*** rockstar has quit IRC05:03
*** cargonza has quit IRC05:04
*** kun_huang has quit IRC05:04
*** jamielennox|away is now known as jamielennox05:04
*** ediardo has joined #openstack-infra05:04
*** asingh has joined #openstack-infra05:04
*** ggherdov- has joined #openstack-infra05:04
*** dham1 has quit IRC05:04
*** hrybacki has joined #openstack-infra05:04
*** agordeev has joined #openstack-infra05:05
*** warpc has joined #openstack-infra05:05
*** alexey_weyl has joined #openstack-infra05:05
*** rockstar has joined #openstack-infra05:05
*** cargonza has joined #openstack-infra05:06
*** alexey_weyl has quit IRC05:06
*** dimtruck is now known as zz_dimtruck05:06
*** dham1 has joined #openstack-infra05:06
*** harlowja_at_home has joined #openstack-infra05:06
*** dougwig has joined #openstack-infra05:07
*** kun_huang has joined #openstack-infra05:07
*** mgkwill has quit IRC05:07
*** mgkwill has joined #openstack-infra05:08
*** pgadiya has quit IRC05:09
openstackgerritMerged openstack-infra/project-config: import the dependencies needed for creating stable branches
*** salv-orlando has quit IRC05:10
openstackgerritMerged openstack-infra/project-config: Change ACLs for fuel-qa project
*** roxanaghe has joined #openstack-infra05:11
*** jaranovich has quit IRC05:13
*** dirk has quit IRC05:13
*** rackertom has quit IRC05:13
*** petems has quit IRC05:13
*** puiterwijk|cld has quit IRC05:13
*** teran has quit IRC05:13
*** madhuvishy has quit IRC05:13
*** andrewbogott has quit IRC05:13
AJaegerproject-config cores, could you review the storyboard publishing change so that Zara has documents, please?
*** salv-orlando has joined #openstack-infra05:14
*** jaranovich has joined #openstack-infra05:14
*** madhuvishy has joined #openstack-infra05:14
*** dirk has joined #openstack-infra05:14
openstackgerritMerged openstack/diskimage-builder: add no_proxy when debootstrap trying to use proxy
openstackgerritMerged openstack/diskimage-builder: Update GRUB_MKCONFIG for detecting what's installed
*** rackertom has joined #openstack-infra05:14
*** puiterwijk|cld has joined #openstack-infra05:15
*** teran has joined #openstack-infra05:15
*** andrewbogott has joined #openstack-infra05:15
*** petems has joined #openstack-infra05:15
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: [WIP] Refactor devstack log copying
*** senk has quit IRC05:19
*** nwkarsten has quit IRC05:21
*** nwkarsten has joined #openstack-infra05:22
*** pgadiya has joined #openstack-infra05:22
*** sdake has joined #openstack-infra05:23
openstackgerritMerged openstack-infra/os-loganalyze: Test high-precision timestamps in console log
*** nwkarsten has quit IRC05:26
*** roxanaghe has quit IRC05:26
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Update api-jobs
*** shashank_hegde has quit IRC05:30
*** netsin has quit IRC05:30
*** roxanaghe has joined #openstack-infra05:31
*** yamamoto_ has joined #openstack-infra05:34
*** javeriak has joined #openstack-infra05:34
*** javeriak has quit IRC05:34
*** javeriak has joined #openstack-infra05:35
*** thorst has joined #openstack-infra05:35
openstackgerritIan Wienand proposed openstack/diskimage-builder: Add IMAGE_ELEMENT_YAML
openstackgerritIan Wienand proposed openstack/diskimage-builder: Making element overriding explicit
openstackgerritIan Wienand proposed openstack/diskimage-builder: Convert pkg-map and svc-map copies to explicit variables
*** roxanaghe has quit IRC05:38
*** asselin_ has joined #openstack-infra05:38
*** asselin__ has quit IRC05:40
AJaegerianw, jhesketh, could you review as well, please? I need some debugging logs for a change...05:42
*** ihrachys has joined #openstack-infra05:42
jheskethAJaeger: looking05:42
*** thorst has quit IRC05:42
*** ilyashakhat has joined #openstack-infra05:42
*** zz_dimtruck is now known as dimtruck05:42
*** Genek has quit IRC05:43
jheskethAJaeger: +w, but we should consider reverting that once you're done (not sure how much noise it'll add to the log storage)05:44
*** Genek has joined #openstack-infra05:44
openstackgerritMerged openstack/diskimage-builder: Add tests for building *-minimal images
*** Genek has quit IRC05:44
AJaegerjhesketh: we have that on all tox based jobs - so I would leave it in unless I hear of a different policy...05:44
*** Genek has joined #openstack-infra05:44
*** cody-somerville has quit IRC05:45
jheskeththat's true, I wonder why these were missing..05:45
*** Genek has quit IRC05:45
*** Genek has joined #openstack-infra05:45
AJaegerjhesketh: see also I8ff8bef0aebdfd1a1e3d2e42a9972e9fb438cd0b and Id2ab952a7b1d34e659916bc38b41be48dd4e34f505:45
AJaegerSo, we added some more recently - but forgot these IMHO.05:45
*** Genek has quit IRC05:46
*** Genek has joined #openstack-infra05:46
*** Genek has quit IRC05:47
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Remove fail_required parameter from convert xml
*** Genek has joined #openstack-infra05:47
*** Genek has quit IRC05:47
openstackgerritMerged openstack-infra/project-config: Update storyboard publishing
*** Genek has joined #openstack-infra05:48
*** Genek has quit IRC05:48
*** adriant has quit IRC05:49
openstackgerritMerged openstack-infra/project-config: Update api-jobs
*** harlowja_at_home has quit IRC05:51
*** Genek has joined #openstack-infra05:52
*** binbincong has quit IRC05:52
*** yamahata has quit IRC05:52
*** shashank_hegde has joined #openstack-infra05:54
*** ilyashakhat has quit IRC05:56
*** Genek has quit IRC05:57
*** nstolyarenko has joined #openstack-infra05:57
*** florianf has joined #openstack-infra05:58
*** edmondsw has joined #openstack-infra05:59
*** edmondsw has quit IRC06:00
*** Sukhdev has joined #openstack-infra06:01
*** nstolyarenko has quit IRC06:02
*** binbincong has joined #openstack-infra06:04
openstackgerritIan Wienand proposed openstack-infra/project-config: Switch centos7 job to neutron
*** coolsvap has joined #openstack-infra06:05
*** abregman has quit IRC06:07
*** dteselkin_ has quit IRC06:08
*** rcernin has quit IRC06:08
*** katyafervent_awa has quit IRC06:08
*** asilenkov has quit IRC06:08
*** salv-orl_ has joined #openstack-infra06:08
*** kzaitsev_ws has quit IRC06:08
*** igormarnat has quit IRC06:09
*** pkarikh has quit IRC06:09
*** mikelk has joined #openstack-infra06:09
*** rakhmerov has quit IRC06:10
*** tsufiev has quit IRC06:10
*** penguinolog has quit IRC06:10
*** ilyashakhat has joined #openstack-infra06:10
*** dteselkin has joined #openstack-infra06:10
openstackgerritMerged openstack/diskimage-builder: Disabled IPv6 privacy extensions
*** salv-orlando has quit IRC06:11
*** hashar has joined #openstack-infra06:11
*** penguinolog has joined #openstack-infra06:12
*** pkarikh has joined #openstack-infra06:12
*** hashar is now known as hasharAway06:12
*** _nadya_ has joined #openstack-infra06:12
*** rakhmerov has joined #openstack-infra06:13
*** katyafervent_awa has joined #openstack-infra06:14
*** tsufiev has joined #openstack-infra06:15
*** kzaitsev_ws has joined #openstack-infra06:15
*** asilenkov has joined #openstack-infra06:16
*** binbincong has quit IRC06:16
*** igormarnat has joined #openstack-infra06:16
odyssey4mejhesketh if you have a moment to review & I'd appreciate it06:16
*** lock__ has joined #openstack-infra06:18
*** oanson has joined #openstack-infra06:18
*** Jeffrey4l__ is now known as Jeffrey4l06:19
*** dulek_ has joined #openstack-infra06:20
*** mnencia_ has joined #openstack-infra06:20
*** taron_ has joined #openstack-infra06:20
*** jlk` has joined #openstack-infra06:21
*** ljjjustin has quit IRC06:21
*** dulek has quit IRC06:21
*** mnencia has quit IRC06:21
*** taron has quit IRC06:21
*** hichihar_ has joined #openstack-infra06:21
*** jlk has quit IRC06:21
*** vponomaryov has quit IRC06:21
*** dulek_ is now known as dulek06:21
*** mnencia_ is now known as mnencia06:21
*** ljjjustin has joined #openstack-infra06:21
*** hichihara has quit IRC06:22
*** lock_ has quit IRC06:22
*** gnuoy has quit IRC06:22
*** gnuoy has joined #openstack-infra06:22
*** vponomaryov has joined #openstack-infra06:22
*** hichihar_ has quit IRC06:22
*** hichihara has joined #openstack-infra06:23
*** kzaitsev_mb has joined #openstack-infra06:25
*** javeriak_ has joined #openstack-infra06:26
*** binbincong has joined #openstack-infra06:28
*** javeriak has quit IRC06:29
openstackgerritMerged openstack/diskimage-builder: Generate and use upper-constraints for ironic-agent
openstackgerritMerged openstack/diskimage-builder: Update portage only if needed
openstackgerritMerged openstack/diskimage-builder: Clarify OVERWRITE_OLD_IMAGE docs
openstackgerritMerged openstack/diskimage-builder: Add blurb about communication to docs landing page
*** pcaruana has joined #openstack-infra06:30
*** kzaitsev_mb has quit IRC06:31
rcarrillocruzinfracloud ran jobs during the night:06:32
jheskethodyssey4me: looking06:33
*** ihrachys has quit IRC06:33
*** senk has joined #openstack-infra06:35
jheskethodyssey4me: do you feel like iterating on a nit on 36364006:36
odyssey4mejhesketh can do06:36
jheskethodyssey4me: if it's too nitty though, I'm happy to +W06:36
odyssey4mejhesketh hmm, perhaps I need more coffee - not sure what you mean there06:38
*** javeriak has joined #openstack-infra06:38
*** javeria__ has joined #openstack-infra06:39
*** javeriak_ has quit IRC06:39
jheskethodyssey4me: you've put the job in projects.yaml twice where the only difference is the suffix -nv right?06:39
jheskethbut the voting version isn't currently used in layout.yaml.. so why not just define it the one time and then when you're ready to make the job voting just remove the job-suffix?06:40
*** dteselkin has quit IRC06:40
odyssey4mejhesketh yeah, oh I suppose I could just use one - assuming they all work right06:40
odyssey4meit might end up with a split based on platform06:40
*** thorst has joined #openstack-infra06:40
jheskethodyssey4me: ah, I see... you might make some voting before others..06:41
*** asilenkov has quit IRC06:41
*** rcernin has joined #openstack-infra06:41
odyssey4mejhesketh I'll be doing a folllow up pretty soon to move some of those jobs to voting and can clean it up then06:41
jheskethodyssey4me: I'm fine with how it is, just need to remember to remove the non-voting duplicate once they are all ovitng06:41
*** igormarnat has quit IRC06:41
jheskethokay, sounds good to me06:41
*** tsufiev has quit IRC06:41
*** pkarikh has quit IRC06:41
*** penguinolog has quit IRC06:41
*** kzaitsev_ws has quit IRC06:41
*** rakhmerov has quit IRC06:42
*** katyafervent_awa has quit IRC06:42
jheskethodyssey4me: +w06:42
*** javeriak has quit IRC06:42
*** Sukhdev has quit IRC06:42
odyssey4methanks jhesketh06:42
odyssey4meI have a job ready to test them all - I'll patch again later this morning to move them to voting06:43
*** rcernin has quit IRC06:45
*** rcernin has joined #openstack-infra06:45
*** javeria__ has quit IRC06:46
*** Genek has joined #openstack-infra06:46
*** kaisers_ has joined #openstack-infra06:46
*** javeriak has joined #openstack-infra06:46
*** thorst has quit IRC06:47
openstackgerritMerged openstack-infra/project-config: Move unsuccessful non-voting OSA jobs to experimental
*** _nadya_ has quit IRC06:47
*** javeriak has quit IRC06:48
*** pkarikh has joined #openstack-infra06:48
*** martinkopec has joined #openstack-infra06:48
*** javeriak has joined #openstack-infra06:48
*** rakhmerov has joined #openstack-infra06:49
*** penguinolog has joined #openstack-infra06:49
*** ilyashakhat has quit IRC06:50
*** salv-orl_ has quit IRC06:50
*** katyafervent_awa has joined #openstack-infra06:51
*** dteselkin has joined #openstack-infra06:53
*** tsufiev has joined #openstack-infra06:53
*** asilenkov has joined #openstack-infra06:54
*** igormarnat has joined #openstack-infra06:54
openstackgerritMerged openstack-infra/project-config: Add OSA keystone uwsgi functional tests
*** hferenc has quit IRC06:55
*** kzaitsev_ws has joined #openstack-infra06:58
*** javeriak_ has joined #openstack-infra06:59
*** javeria__ has joined #openstack-infra07:00
*** _nadya_ has joined #openstack-infra07:00
*** javeriak has quit IRC07:00
*** oanson has quit IRC07:00
*** _nadya_ has quit IRC07:00
*** claudiub has joined #openstack-infra07:01
*** vinaypotluri has quit IRC07:02
*** tesseract- has joined #openstack-infra07:03
*** rwsu has joined #openstack-infra07:03
*** esikachev has joined #openstack-infra07:03
*** netsin has joined #openstack-infra07:03
*** javeriak_ has quit IRC07:04
*** aeng has quit IRC07:04
*** andreas_s has joined #openstack-infra07:05
*** spzala has joined #openstack-infra07:06
*** yolanda has joined #openstack-infra07:06
*** abregman has joined #openstack-infra07:06
*** Na3iL has joined #openstack-infra07:08
*** drifterza has joined #openstack-infra07:08
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/project-config: Bump Infracloud vanilla max-servers to 50
*** javeriak has joined #openstack-infra07:10
*** spzala has quit IRC07:10
*** mptacekx has joined #openstack-infra07:11
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Update the QA team meeting chair
*** javeria__ has quit IRC07:13
AJaegerrcarrillocruz: woot! Great to see the progress on infracloud!07:13
mptacekxHi, I am contributing to intel-nfv-ci, currently we are facing an issue with unstable connectivity to OVH file server (,, there are some drop-outs fr e.g. 15mins nearly every second hr producing unstable results. Is it a known issue ? Thanks07:14
AJaegerrcarrillocruz: will you be around to babysit the cloud? Then let's ask others to +2A 36410107:14
*** sshnaidm|afk is now known as sshnaidm07:14
*** javeriak has quit IRC07:14
*** javeriak has joined #openstack-infra07:15
AJaegerrcarrillocruz: btw. shows vanilla and west. Is west really correct as name?07:17
openstackgerritPeter Zhurba proposed openstack-infra/project-config: Add repo for openstack/puppet-glare.
*** jamielennox is now known as jamielennox|away07:17
*** yaume has joined #openstack-infra07:18
*** shashank_hegde has quit IRC07:19
*** bexelbie has quit IRC07:20
*** matbu|afk is now known as matbu07:20
*** taron_ has quit IRC07:21
*** nstolyarenko has joined #openstack-infra07:21
*** dimtruck is now known as zz_dimtruck07:22
*** jlvillal has quit IRC07:22
*** taron has joined #openstack-infra07:22
*** bexelbie has joined #openstack-infra07:23
*** nwkarsten has joined #openstack-infra07:23
*** jlvillal has joined #openstack-infra07:23
AJaegerzigo: please check
*** jpich has joined #openstack-infra07:25
*** kzaitsev_mb has joined #openstack-infra07:27
*** ifarkas_afk is now known as ifarkas07:28
*** nstolyarenko has quit IRC07:28
*** nwkarsten has quit IRC07:29
rcarrillocruzAJaeger: it is not, that is something that should be corrected07:29
rcarrillocruzwest = vanilla07:29
rcarrillocruzeast = chocolate07:29
rcarrillocruzAJaeger: yeah, let's not land 364101 for now07:29
rcarrillocruzi'm off for a couple hours07:30
rcarrillocruzi better wip it, thx07:30
*** Genek has quit IRC07:31
rcarrillocruzi also need to double-check we won't hit quota issues07:31
*** apetrich has quit IRC07:31
AJaegerrcarrillocruz: ok07:31
*** abregman_ has joined #openstack-infra07:32
rcarrillocruzAJaeger: actually, i see vanilla/chocolate names on project-config/grafana07:32
*** abregman has quit IRC07:32
rcarrillocruzwhere do you see that, in the UI of Grafana?07:33
AJaegerrcarrillocruz: see the link - in the UI07:33
rcarrillocruzah, yeah, the combolist07:33
*** apetrich has joined #openstack-infra07:33
AJaegerhistoric data?07:33
rcarrillocruzprobably it needs a restart or something07:33
AJaegerAh, ok...07:33
rcarrillocruzi'll check with pabelanger  later07:33
AJaegeryeah, it's not in project-config - checked there now.07:34
*** dizquierdo has joined #openstack-infra07:34
*** javeriak has quit IRC07:35
openstackgerritYuval Brik proposed openstack-infra/project-config: Rename Smaug to Karbor
ttxtristanC: Election schedule looks good to me. Note that there is no Requirements team in the PTL list (but that may be ok since there was a very recent election for that ?)07:36
*** mptacekxx has joined #openstack-infra07:38
*** mptacekx has quit IRC07:40
openstackgerritMartin André proposed openstack-infra/tripleo-ci: [WIP] Fix scp command with IPv6 addresses
*** ihrachys has joined #openstack-infra07:41
skraynevAJaeger: hi. sorry for the interruption again. could you please add me to the groups,members,members,members according patch
*** mptacekxx has quit IRC07:42
*** mptacekxx has joined #openstack-infra07:43
*** sdake has quit IRC07:43
*** mptacekxx has quit IRC07:43
*** sdake has joined #openstack-infra07:44
*** caowei has quit IRC07:44
*** javeriak has joined #openstack-infra07:44
*** thorst has joined #openstack-infra07:45
*** mptacekx has joined #openstack-infra07:45
tristanCttx: thanks! indeed it's missing Requirements, I've added in the last patchset07:46
ttxtristanC: I take it you're not in Quebec right now07:46
*** esikachev has quit IRC07:47
*** bexelbie has quit IRC07:50
*** bexelbie has joined #openstack-infra07:50
*** matthewbodkin has joined #openstack-infra07:51
*** salv-orlando has joined #openstack-infra07:51
*** thorst has quit IRC07:52
*** javeriak has quit IRC07:54
*** ihrachys has quit IRC07:55
*** yfried has joined #openstack-infra07:55
yfriedHi, looking for help with shade dependencies07:55
yfriedshade requires os-client-config>=1.20.0 and python-magnumclient>=2.1.007:56
yfriedpython-magnumclient: os-client-config!=1.19.0,!=1.19.1,!=1.20.0,!=1.20.1,!=1.21.0,>=1.13.107:56
*** shardy has joined #openstack-infra07:57
*** dtantsur|afk is now known as dtantsur07:58
*** esikachev has joined #openstack-infra07:58
*** zzzeek has quit IRC08:00
*** salv-orlando has quit IRC08:00
*** ljjjustin has quit IRC08:00
*** zzzeek has joined #openstack-infra08:00
*** dchen has joined #openstack-infra08:02
*** _nadya_ has joined #openstack-infra08:03
*** dingyichen has quit IRC08:04
*** jaosorior has joined #openstack-infra08:04
*** sflanigan has quit IRC08:05
openstackgerritMerged openstack-infra/project-config: Clone sources of ironic-inspector and ironic-inspector-client
*** _nadya_ has quit IRC08:08
*** nmagnezi has joined #openstack-infra08:09
*** oanson has joined #openstack-infra08:11
*** matrohon has joined #openstack-infra08:12
dlahngood morning08:15
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: Fix devstack subunit output
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: [WIP] Refactor devstack log copying
skraynevsorry for the interruption again. could you please add me to the groups,members,members,members according patch
AJaegerskraynev: I can't, I don't have those permissions. I hope somebody reads backscroll and gets back to you. If not: ask when more admins are around - or ask on the infra mailing list.08:19
skraynevAJaeger: ok. got it. thank you for advise08:19
*** shashank_hegde has joined #openstack-infra08:22
*** kushal has joined #openstack-infra08:23
*** asettle has joined #openstack-infra08:26
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Shorten job names for OSA os_keystone repo
odyssey4meproject config cores - an urgent review of the above would be appreciated08:26
*** esikachev has quit IRC08:27
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: Update bashate to 0.5.0
AJaegerodyssey4me: +2A08:27
odyssey4methanks AJaeger08:28
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: POC: WIP: oooq undercloud install
ianwthat problem keeps hitting us08:28
*** kzaitsev_mb has quit IRC08:28
openstackgerritMartin André proposed openstack-infra/tripleo-ci: Fix scp command with IPv6 addresses
*** shashank_hegde has quit IRC08:29
openstackgerritAndrey Nikitin proposed openstack-infra/project-config: Add one new Fuel's plugin
*** dchen has quit IRC08:31
*** r-mibu has quit IRC08:33
*** priteau has joined #openstack-infra08:37
*** hasharAway is now known as hashar08:37
*** esikachev has joined #openstack-infra08:38
*** r-mibu has joined #openstack-infra08:38
*** lucas-dinner is now known as lucasagomes08:38
*** Julien-zte has quit IRC08:40
*** berendt has joined #openstack-infra08:41
*** e0ne has joined #openstack-infra08:44
*** bethwhite_ has joined #openstack-infra08:44
openstackgerritEvgeny Antyshev proposed openstack-infra/nodepool: Add missing month argument
*** hichihara has quit IRC08:45
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Change west/east naming for vanilla/chocolate on set quotas playbook
rcarrillocruzjhesketh: if you're still around, do you mind reviewing ^ ?08:48
rcarrillocruzi need to bump quotas on the zuul projects, prior to bumping max-servers on nodepool08:48
*** dizquierdo has quit IRC08:48
rcarrillocruzand i rather do it via playbook, instead with a one-off command that is not documented/tracked anywhere08:48
rcarrillocruzor ianw ^08:49
*** thorst has joined #openstack-infra08:50
ttxproject-config core reviewers: could you please approve so that automatic tagging of libraries is fixed ?08:50
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: POC: WIP: oooq undercloud install
ttxAJaeger already +2ed it08:51
*** javeriak has joined #openstack-infra08:51
ttx(and we have a truckload of libs to process today so that would help)08:51
rcarrillocruzttx: approved08:52
openstackgerritMerged openstack-infra/project-config: Shorten job names for OSA os_keystone repo
ttxrcarrillocruz: awesome, thx08:52
*** sc`_ has joined #openstack-infra08:53
*** dcwangmit01_ has joined #openstack-infra08:53
*** mbacchi_ has joined #openstack-infra08:54
*** hwoarang_ has joined #openstack-infra08:54
*** derekh has joined #openstack-infra08:54
*** jhesketh_ has joined #openstack-infra08:54
*** HenryG_ has joined #openstack-infra08:54
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/project-config: Bump Infracloud vanilla max-servers to 50
*** amotoki_ has joined #openstack-infra08:55
openstackgerritPawel Koniszewski proposed openstack-infra/project-config: Start LM tests with grenade to test rolling upgrades
*** jeblair_ has joined #openstack-infra08:55
*** Ravikiran_K has joined #openstack-infra08:56
*** jlvillal_ has joined #openstack-infra08:57
*** toabctl_ has joined #openstack-infra08:57
*** thorst has quit IRC08:57
*** dtardivel has joined #openstack-infra08:57
*** shardy has quit IRC08:58
*** mptacekx has quit IRC08:58
*** jlvillal has quit IRC08:58
*** drifterza has quit IRC08:58
*** dulek has quit IRC08:58
*** amotoki has quit IRC08:58
*** vikrant has quit IRC08:58
*** skipp has quit IRC08:58
*** afred312 has quit IRC08:58
*** M-docaedo_vector has quit IRC08:58
*** _degorenko|afk has quit IRC08:58
*** lennyb has quit IRC08:58
*** spiffxp has quit IRC08:58
*** wfoster has quit IRC08:58
*** jhesketh has quit IRC08:58
*** darrenc has quit IRC08:58
*** mat128 has quit IRC08:58
*** SotK has quit IRC08:58
*** dcwangmit01 has quit IRC08:58
*** bapalm has quit IRC08:58
*** sc` has quit IRC08:58
*** HenryG has quit IRC08:58
*** anteaya has quit IRC08:58
*** sambetts|afk has quit IRC08:58
*** stevebaker has quit IRC08:58
*** jeblair has quit IRC08:58
*** michaelxin has quit IRC08:58
*** whoops has quit IRC08:58
*** mbacchi has quit IRC08:58
*** antonym has quit IRC08:58
*** Anticimex has quit IRC08:58
*** eliqiao has quit IRC08:58
*** hwoarang has quit IRC08:58
*** toabctl has quit IRC08:58
*** aderyugin has quit IRC08:58
*** andymccr has quit IRC08:58
*** jgrassler has quit IRC08:58
*** toabctl_ is now known as toabctl08:58
*** sc`_ is now known as sc`08:58
*** salv-orlando has joined #openstack-infra08:58
*** bapalm has joined #openstack-infra08:59
*** mat128 has joined #openstack-infra08:59
*** andymccr has joined #openstack-infra08:59
*** toabctl has quit IRC08:59
*** Anticimex has joined #openstack-infra09:00
*** antonym has joined #openstack-infra09:00
*** toabctl has joined #openstack-infra09:00
*** whoops has joined #openstack-infra09:00
*** jlvillal_ is now known as jlvillal09:00
*** _nadya_ has joined #openstack-infra09:01
*** jlvillal is now known as Guest7795609:01
*** esikachev has quit IRC09:01
*** salv-orlando has quit IRC09:01
*** tkelsey has joined #openstack-infra09:01
*** shardy has joined #openstack-infra09:03
*** mptacekx has joined #openstack-infra09:03
*** drifterza has joined #openstack-infra09:03
*** dulek has joined #openstack-infra09:03
*** vikrant has joined #openstack-infra09:03
*** afred312 has joined #openstack-infra09:03
*** M-docaedo_vector has joined #openstack-infra09:03
*** _degorenko|afk has joined #openstack-infra09:03
*** lennyb has joined #openstack-infra09:03
*** spiffxp has joined #openstack-infra09:03
*** wfoster has joined #openstack-infra09:03
*** darrenc has joined #openstack-infra09:03
*** SotK has joined #openstack-infra09:03
*** anteaya has joined #openstack-infra09:03
*** sambetts|afk has joined #openstack-infra09:03
*** stevebaker has joined #openstack-infra09:03
*** eliqiao has joined #openstack-infra09:03
*** aderyugin has joined #openstack-infra09:03
*** skipp has joined #openstack-infra09:03
*** javeriak has quit IRC09:03
*** mhickey has joined #openstack-infra09:03
openstackgerritMerged openstack-infra/project-config: be more careful using setuptools commands in release script
*** jgrassler has joined #openstack-infra09:05
*** salv-orlando has joined #openstack-infra09:05
*** tkelsey has quit IRC09:05
*** asettle has quit IRC09:08
*** abregman_ has quit IRC09:08
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Send notifications to subscribers for worklists
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Make it possible to get worklist/board timeline events via the API
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Don't allow users to subscribe to private worklists they can't see
*** salv-orlando has quit IRC09:10
*** Genek has joined #openstack-infra09:10
*** asettle has joined #openstack-infra09:12
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: WIP - Implement overcloud upgrade job - Mitaka -> Newton
*** HenryG_ is now known as HenryG09:17
*** rossella_s has quit IRC09:18
*** rossella_s has joined #openstack-infra09:19
*** netsin has quit IRC09:21
*** sambetts|afk is now known as sambetts09:23
*** abregman_ has joined #openstack-infra09:23
*** mrunge has quit IRC09:25
*** ihrachys has joined #openstack-infra09:25
*** Genek has quit IRC09:28
*** Genek has joined #openstack-infra09:28
*** HenryG has quit IRC09:29
*** HenryG has joined #openstack-infra09:29
openstackgerritVladyslav Drok proposed openstack-infra/project-config: Add Q_USE_PROVIDERNET_FOR_PUBLIC in multitenant job
*** hwoarang_ is now known as hwoarang09:31
*** netsin has joined #openstack-infra09:32
*** Genek has quit IRC09:32
*** mrunge has joined #openstack-infra09:36
*** flepied has quit IRC09:36
*** flepied1 has joined #openstack-infra09:36
*** darvon has quit IRC09:38
*** darvon has joined #openstack-infra09:40
*** zshuo has quit IRC09:41
openstackgerritAlexander Evseev proposed openstack-infra/puppet-zuul: Replace upstream module by Mirantis' one
rcarrillocruzsweet, after DC folks magic I'm recovering compute00509:43
rcarrillocruzone less to fix09:43
*** _degorenko|afk is now known as degorenko09:43
*** chem|off has joined #openstack-infra09:43
*** chem|off is now known as chem09:43
*** salv-orlando has joined #openstack-infra09:45
*** daemontool has joined #openstack-infra09:45
*** sdague has joined #openstack-infra09:46
*** chem has quit IRC09:49
*** chem has joined #openstack-infra09:49
openstackgerritYuriy Taraday proposed openstack-infra/project-config: Turn off Puppet jobs on old fuel-octane branches
openstackgerritPawel Koniszewski proposed openstack-infra/project-config: LM tests with grenade to test rolling upgrades
*** yanyanhu has quit IRC09:52
YorikSarAny infra cores around? I've forgot to disable job for old branches before making it voting: - it blocks octane team, please review.09:53
openstackgerritMerged openstack-infra/tripleo-ci: Fix scp command with IPv6 addresses
*** matthewbodkin has quit IRC09:55
*** thorst has joined #openstack-infra09:55
*** dizquierdo has joined #openstack-infra09:58
*** flepied1 has quit IRC10:00
*** dizquierdo has quit IRC10:00
*** flepied has joined #openstack-infra10:00
*** Thelo_ has joined #openstack-infra10:01
*** sdake has quit IRC10:01
*** kzaitsev_mb has joined #openstack-infra10:02
*** HeOS has quit IRC10:02
*** yamamoto_ has quit IRC10:02
*** thorst has quit IRC10:03
*** zhurong has quit IRC10:03
*** HeOS has joined #openstack-infra10:04
*** asettle has quit IRC10:05
*** javeriak has joined #openstack-infra10:08
*** matthewbodkin has joined #openstack-infra10:13
rcarrillocruzthx ianw10:15
*** asettle has joined #openstack-infra10:16
HeOSHello, infra-team! Please repeat review of the following request:
*** jhesketh_ is now known as jhesketh10:20
*** tqtran has joined #openstack-infra10:20
*** dizquierdo has joined #openstack-infra10:22
openstackgerritMerged openstack-infra/project-config: Turn off Puppet jobs on old fuel-octane branches
openstackgerritMerged openstack-infra/system-config: Change west/east naming for vanilla/chocolate on set quotas playbook
*** pilgrimstack has quit IRC10:24
*** tqtran has quit IRC10:25
*** pilgrimstack has joined #openstack-infra10:27
*** nwkarsten has joined #openstack-infra10:27
*** pilgrimstack has quit IRC10:31
*** nwkarsten has quit IRC10:32
*** yolanda has quit IRC10:32
*** nstolyarenko has joined #openstack-infra10:32
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Install tripleo-admin ssh keys on CI nodes
openstackgerritBartosz Kupidura proposed openstack-infra/puppet-apps_site: [wip] Glare support for app-catalog
*** sigmavirus|awa is now known as sigmavirus10:37
*** Genek has joined #openstack-infra10:39
openstackgerritMasayuki Igawa proposed openstack/gertty: Add size column to change list view
*** Qiming has quit IRC10:40
*** pilgrimstack has joined #openstack-infra10:41
*** Genek has quit IRC10:43
*** yamamoto has joined #openstack-infra10:43
*** esikachev has joined #openstack-infra10:44
*** Qiming has joined #openstack-infra10:45
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Promote working os_keystone nv jobs to voting
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: [NO MERGE] Test Ceph RadosGW as replacement for Swift
*** Genek has joined #openstack-infra10:48
*** asettle has quit IRC10:48
*** dtantsur is now known as dtantsur|lunch10:49
*** gongysh has quit IRC10:49
openstackgerritBartosz Kupidura proposed openstack-infra/puppet-apps_site: [wip] Glare support for app-catalog
openstackgerritJim Rollenhagen proposed openstack-infra/elastic-recheck: Add query for bug 1619232
openstackbug 1619232 in Ironic "Heartbeat()'s race condition: InvalidState: Can not transition from state 'deploying' on event 'resume' (no defined transition)" [High,New] - Assigned to Lucas Alvares Gomes (lucasagomes)10:50
*** yamamoto has quit IRC10:52
wznoinskhi infra10:52
wznoinskdo you have any troubles with OVH of any kind today?10:52
*** Ravikiran_K has quit IRC10:52
*** esikachev has quit IRC10:56
*** Genek has quit IRC10:57
*** sskripnick has joined #openstack-infra10:57
sskripnickhi all10:57
sskripnickfatal: unable to access '': Failed to connect to port 80: Connection refused10:57
*** Genek has joined #openstack-infra10:57
sskripnickrally-ci got banned? -_-10:57
*** salv-orlando has quit IRC10:58
*** yamamoto has joined #openstack-infra11:00
openstackgerritMerged openstack-infra/elastic-recheck: Add query for bug 1619232
openstackbug 1619232 in Ironic "Heartbeat()'s race condition: InvalidState: Can not transition from state 'deploying' on event 'resume' (no defined transition)" [High,New] - Assigned to Lucas Alvares Gomes (lucasagomes)11:00
*** thorst has joined #openstack-infra11:00
AJaegersskripnick: there were some network problems that clarkb and others debugged yesterday - in the Rackspace datacenter. I don't know whether those are resolved. Is this a recent result? Is it reproduceable?11:00
*** Genek has quit IRC11:01
sdaguewe've got a ton of multinode issues11:03
sdaguewhere the subnode is not responding11:04
*** rhallisey has joined #openstack-infra11:05
sskripnickAJaeger: like 15% attems are successful11:05
openstackgerritBartosz Kupidura proposed openstack-infra/puppet-apps_site: [wip] Glare support for app-catalog
AJaegersskripnick: that's sad ;( Best talk later with rest of team...11:06
*** dtroyer has left #openstack-infra11:06
sskripnickAJaeger: got it. thanks11:06
*** thorst has quit IRC11:08
openstackgerritCsaba Henk proposed openstack-infra/project-config: remove manila's glusterfs xenial jobs
openstackgerritMerged openstack-infra/project-config: Promote working os_keystone nv jobs to voting
*** javeriak has quit IRC11:17
*** rtheis has joined #openstack-infra11:18
sdagueany infra-root up?11:18
*** esikache1 has joined #openstack-infra11:19
*** ramishra_ has joined #openstack-infra11:20
*** ramishra has quit IRC11:21
*** jkilpatr has joined #openstack-infra11:21
*** kushal has quit IRC11:24
rcarrillocruzi am11:25
rcarrillocruzwhat's up11:25
*** thorst has joined #openstack-infra11:26
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: POC: WIP: oooq undercloud install
pkoniszewskihello infra, is there a way to add grenade to existing job? I mean, I'm trying to work on live migration job with grenade, tried this approach but it did not work at all -
*** ldnunes has joined #openstack-infra11:27
sdaguercarrillocruz: something is really weird with multinode11:28
*** nmagnezi has quit IRC11:28
sdaguethe subnode failed to connect rate has gone way up11:28
*** nmagnezi has joined #openstack-infra11:28
sdaguercarrillocruz: it looks like this command hangs for 2 hours -
sdagueand the subnode logs just stop a few minutes in11:32
rcarrillocruzoutput is not superuseful11:32
rcarrillocruzlet me see11:33
sdagueso this is definitely a spike as of 6 hours ago11:33
*** lucasagomes is now known as lucas-hungry11:33
*** Genek has joined #openstack-infra11:34
sdagueit's not isolated to any particular provider11:34
sdaguewhich is weird11:35
tdurakovsdague: I was wrong, it affects all multinode jobs11:35
*** dprince has joined #openstack-infra11:36
*** Hal1 has quit IRC11:36
*** Hal1 has joined #openstack-infra11:36
tdurakovand there is a spike for last 24 hours11:36
*** Hal2 has joined #openstack-infra11:37
sdaguercarrillocruz: any idea if there is an ansible option that would let us fail a bit faster if we lose the subnode?11:38
*** asettle has joined #openstack-infra11:39
sdaguethis is super weird11:41
rcarrillocruzyou could put the task as async and poll for it11:41
*** baoli has joined #openstack-infra11:41
rcarrillocruzi'm on the subnode11:41
rcarrillocruzon a 2-node i just held11:41
*** rfolco has joined #openstack-infra11:41
*** Hal1 has quit IRC11:41
rcarrillocruzand i don't see any logging or anything pointing to a lock or doing much stuff11:41
sdaguercarrillocruz: well, it's not all of them that are dying for sure11:41
*** baoli_ has joined #openstack-infra11:42
rcarrillocruzand the node is pretty much idle, from top output11:42
*** larainema has quit IRC11:42
rcarrillocruzguess i got a 'good' subnode11:42
sdaguethe last thing that runs is -
sdagueor at least the last thing that logs11:42
sdaguercarrillocruz: oh, I have a node that I know is borked11:43
*** dizquierdo has quit IRC11:43
rcarrillocruzeugh, i have no ipv611:44
sdaguethe subnode there is in trouble11:44
rcarrillocruzi was checking ubuntu-xenial-2-node-rax-ord-4034028 , but i don't see anything wrong with it, again, it may be this one is just fine11:44
sdagueyeh, let me see if I can find another trouble node that's not ipv611:44
sdagueI think that one is in the same trouble11:45
*** baoli has quit IRC11:46
sdagueI wonder if something is weird with openstackclient 3.2.011:47
openstackgerritZara proposed openstack-infra/python-storyboardclient: Add basic docs for python client
sdaguethat spews something that breaks the ssh connection11:47
*** markusry has joined #openstack-infra11:48
*** gongysh has joined #openstack-infra11:49
*** sarob has joined #openstack-infra11:50
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Send notifications to subscribers for worklists
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Make it possible to get worklist/board timeline events via the API
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Don't allow users to subscribe to private worklists they can't see
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Limit PyMySQL to < 0.7.7
Zarauh oh, another limit11:51
rcarrillocruzsdague: i don't see anything suspicious on the subnode11:52
rcarrillocruzchecked stack logs11:52
rcarrillocruzand nothing pointing to a problem11:52
sdagueis openstack complete still running?11:52
sdaguecan you give me ssh to the subnode?11:53
sdagueso it is still running11:53
rcarrillocruzsure, give me your pubkey , you ssh to the node and from there you enter the subnode?11:53
*** sarob has quit IRC11:54
*** larainema has joined #openstack-infra11:55
rcarrillocruzsdague: put your key on zuul user, you should be able to sudo su afterwards11:56
rcarrillocruzlet me know if you can get in11:56
sdaguessh zuul@ ?11:57
rcarrillocruzsubnode is
sdaguethat fails me11:57
sdaguePermission denied (publickey).11:57
sdagueyeh, no dice11:58
*** yaume_ has joined #openstack-infra11:59
sdagueany thoughts about why my key is no good there?11:59
*** rfolco has quit IRC11:59
*** rfolco has joined #openstack-infra12:00
rcarrillocruztry know sdague12:00
rcarrillocruzi think it's due to the paste12:00
rcarrillocruzit wrapped the lines12:01
*** yaume has quit IRC12:01
rcarrillocruzinfracloud ran tests fine last night12:01
rcarrillocruzlet's bump it a little further, then we can make the max count for cores/mem quota12:02
amrithsdague, I got the public key. it is useless without the private key, would you paste that too please :) PM is fine ...12:02
sdaguepabelanger / rcarrillocruz - can we please not change any more cloud things until we figure out why multinode is busted?12:02
amrithI would love to do just the same thing, we're having some odd test failures and I'd love to ssh to the host running the tests and look around12:02
sdaguercarrillocruz: ok, so still not working, what do you need from me to move forward?12:03
rcarrillocruzi'm not sure sdague , i pasted 3 times on the zuul authorized_keys user12:04
*** dtantsur|lunch is now known as dtantsur12:04
rcarrillocruzstill doesn't work for you?12:04
sdagueit does not12:04
amrithrcarrillocruz, for review, and job gate-trove-functional-dsvm-mysql-mitaka which I just triggered, I'd like to ssh into the test instance. would that be possible?12:04
amriththe logs are at telnet://
*** tpsilva has joined #openstack-infra12:05
sdaguercarrillocruz: and a ton of things are recheck grinding that need to land for the release12:05
*** psilvad has joined #openstack-infra12:05
sdaguebecause of the multinode fail spike12:06
rcarrillocruzrcarrillocruz: try now?12:06
rcarrillocruzi copied line by line12:06
*** jaosorior has quit IRC12:06
sdaguercarrillocruz: no, take all the line breaks out of it12:06
pabelangersdague: rcarrillocruz: Ya, I'd like to investigate the launch failures in infracloud-vanilla and internap-mtl01 this morning before we bring more nodes online.  There were a few of them over night12:06
*** jaosorior has joined #openstack-infra12:06
pabelangerrcarrillocruz: sdague: Also want to make sure we run at least 10 nodes in both region for a few hours to confirm things are stable too12:06
sdaguepabelanger: sure, it's just we've got this whole other subnode fail spike12:07
amrithrcarrillocruz, if you could do that, here's my public key.
sdagueamrith: that's typically not done unless it's an infra level fail12:07
pabelangersdague: agreed, we need to confirm jobs are stable too12:07
amrithsdague, the issue I'm having is likely that12:07
amrithwhat I have is a guest being spun up12:07
amrithwhich can't seem to talk back to the controller12:07
amrithtrove guest (vm within the infra machine)12:08
amrithand I have no way to debug if it can't get back on the network to the controller (the machine infra spins up)12:08
amrithhence the request12:08
rcarrillocruzsdague: are you really sshing to ? i don't see ssh attemps on auth.log12:08
amrithwe've tried to instrument with code to have the guest ship back logs12:08
amrithbut for all we can tell, it can't seem to get any data back12:08
sdaguercarrillocruz: ssh -i ~/.ssh/id_dsa zuul@
*** pradk has joined #openstack-infra12:09
sdagueoh, your ssh doesn't allow that12:09
sdaguedebug1: Skipping ssh-dss key /home/sdague/.ssh/id_dsa - not in PubkeyAcceptedKeyTypes12:09
rcarrillocruzcan you generate a dummy one with rsa real quick12:10
sdagueecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCkRpGApH9L2PDo+zgwWkuaQ3k5MmPejXu+eWUhvXln5ieAGW7F4dOptBZ+RifJSqFPHD2czwtC7KpNo12:10
sdagueRwzm+aU= sdague@ribos12:10
sdague ?12:10
sdaguehow about that one12:10
amrithuse mind sdague :)12:10
rcarrillocruzsdague: mind doing a paste.openstack, the line breaks and i fear i't going to fial for you12:11
rcarrillocruzfail too12:11
*** yamamoto has quit IRC12:11
*** psilvad has quit IRC12:11
* amrith waits12:12
pabelangerrcarrillocruz: going to look into this failure from this morning Looks like some issues scheduling nodes12:12
rcarrillocruzit's the line wrap i guess12:12
*** yamamoto has joined #openstack-infra12:13
sdagueok, I'm in12:13
sdagueok, how do I get to the subnode?12:13
rcarrillocruzthe subnode is:12:13
sdaguealso, are these things held, so they won't go away on me?12:13
rcarrillocruzyeah, i held12:13
amrithrcarrillocruz, may I request ssh access into the machine running the job gate-trove-functional-dsvm-mysql-mitaka in review
sdaguercarrillocruz: what key do I use to get there?12:14
rcarrillocruzsudo su12:14
rcarrillocruzand from there you can ssh in12:14
rcarrillocruzamrith: which test12:15
rcarrillocruzi read now12:15
amrithno worries, sorry for interrupting you and sdague :)12:15
amrithbut I'm at my wits end after trying to debug this all night12:15
amrithand happened on your conversation with sdague12:15
amrithwhat we're facing is best seen here:12:16
*** yamamoto has quit IRC12:16
amrithno job has successfully passed the tests gate-trove-functional-dsvm-mysql-mitaka andgate-trove-functional-dsvm-mysql-liberty12:17
amrithI tried to revert the last change, that failed too12:17
amriththe issue is that the controller node that is spun up hosts a guest VM for the trove db that gets tested12:17
sdagueok, I think it's the auth prompt12:17
rcarrillocruzamrith: try as 'zuul'12:17
amrithok, one second12:17
*** rfolco has quit IRC12:17
sdagueI think openstackclient is hanging in a password hold12:18
rcarrillocruzanything else ?12:18
rcarrillocruzi was about to go lunch12:18
amrithgot it, thanks12:18
*** rfolco has joined #openstack-infra12:18
mordredsdague: it should only prompt for password if it doesn't have a value for PASSWORD ... any chance you can tell how it was executed?12:19
*** sskripnick has left #openstack-infra12:20
*** oanson has quit IRC12:20
*** lucas-hungry is now known as lucasagomes12:21
sdaguemordred: it's on the subnodes, don't fully know12:21
mordredsdague: zomg. it's the tab complete thing12:21
sdaguemordred: yep, it is12:21
mordredstack     1462 20148  0 10:27 pts/0    00:00:00 /usr/bin/python /usr/local/bin/openstack complete12:21
sdagueoh, I know that bit12:21
sdagueanyway, I think we need to block that release and sort it out later12:22
sdagueright now this is causing a huge fail on multinode jobs12:22
mordredit's an easy devstack fix12:22
mordredif you know where it's being executed12:22
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: WIP - Implement overcloud upgrade job - Mitaka -> Newton
sdagueeasy how?12:22
mordredopenstack --os-cloud=devstack complete12:22
mordredwill not hang12:22
mordredI'm sure the _real_ fix in osc will take more work12:22
sdagueok, so what you are saying is everyone using osc anywhere now has to update their scripts using it otherwise their stuff hangs indefinitely?12:23
mordredI'm saying that anyone anywhere who is running openstack complete, just that command, without auth12:23
mordredwill have to update their scripts12:23
mordredthe number of people who are doin gthat are likely to be done12:23
sdaguethat's really the only thing that breaks here?12:23
mordredit's a hack - but it'll fix the gate without needing to land an exclusion pin12:24
mordredall the _other_ openstack commands already specify cloud becaues they're trying to interact with a cloud12:24
mordredso they'll hav ea password in clouds.yaml12:24
sdaguemordred: ok, but this means that --help is going to hang for people the same way, right?12:24
mordredlemme try12:24
mordredhelp works12:25
*** yolanda has joined #openstack-infra12:25
sdagueyou sure, did you reset your env?12:25
sdagueok, help works12:25
sdagueopenstack ... with no args, prompts for password before you get to the REPL12:25
mordredyah. I would expect it to if there is no OS_PASSWORD env var set12:26
*** esberglu has joined #openstack-infra12:26
mordredthis, btw, is amongst the reasons why I have consistently argued that the password prompt is a bad ui choice12:26
sdaguemordred: ok, I'll take the devstack work around if you register a critical bug somewhere to fix it for real12:26
mordredand should not exist12:26
sdaguemordred: ok, what was the change that made this?12:26
*** yamamoto has joined #openstack-infra12:26
sdaguemordred: and I 100% agree with you that prompts for password are terrible12:26
mordredthere was an osc change recently that changed how password prompt on missing password workded12:27
mordredwe added a hook point in os-client-config for it, and osc started using that hook point12:27
openstackgerritPaul Belanger proposed openstack-infra/puppet-user: Add user to adm group too
mordredI'm guessing osc just missed a use case12:27
sdagueok, show me the bug, I want to update the code to reference it12:28
mordredI have not yet filed one12:28
mordredthat's next on my list12:28
sdagueok, please do that now :)12:28
*** kgiusti has joined #openstack-infra12:28
mordredsdague: how is this not breaking normal devstack runs?12:28
sdaguedon't know, it only seems to trigger on the subnode12:29
sdagueand only some of the time12:29
mordredthat makes no sense to me at all12:29
sdagueme either12:29
*** javeriak has joined #openstack-infra12:29
*** raildo has joined #openstack-infra12:30
sdaguewell, on the subnode might make sense if we don't give the subnode the right osc vars12:31
sdaguebecause we don't actually run any other osc commands there12:31
*** mdrabe has joined #openstack-infra12:31
openstackLaunchpad bug 1619274 in python-openstackclient "openstack complete triggers password prompt" [Undecided,New]12:31
sdagueso we just never setup localrc so that any real commands would work12:31
mordredsdague: ah - that does make sense then12:31
sdaguemordred: your fix assumes a clouds.yaml exists, right?12:33
sdagueoh, I guess this is late enough, it should12:34
*** cody-somerville has joined #openstack-infra12:34
pabelangerrcarrillocruz: mordred: looks like one of our compute nodes in infracloud-vanilla is having some ConnectTimeout issues:
*** zhurong has joined #openstack-infra12:34
rcarrillocruzThx, I will look in a bit12:35
mordredsdague: yah. I mean, I tested the fix on the subnode12:35
pabelangerrcarrillocruz: mordred: I guess we don't have unbound setup? nameserver
*** Hal2 has quit IRC12:36
*** gongysh has quit IRC12:36
*** Genek has quit IRC12:36
pabelangerya, looks like we are hard coding DNS to google in our network interface config12:37
sdaguemordred / rcarrillocruz / pabelanger could we get a gate enqueue and promote on ?12:37
*** Hal1 has joined #openstack-infra12:37
mordredon it12:37
mordredsdague: done12:39
*** cardeois has joined #openstack-infra12:41
*** Hal1 has quit IRC12:42
*** Hal1 has joined #openstack-infra12:44
*** cody-somerville has quit IRC12:45
amrithrcarrillocruz, in what linux user name will devstack get run on the test instance?12:45
amrithis it zuul or jenkins?12:45
openstackgerritPaul Belanger proposed openstack-infra/system-config: Install unbound on infracloud servers
pabelangermordred: rcarrillocruz: ^ add unbound to infracloud controller and compute nodes12:49
*** Hal1 has quit IRC12:49
*** Hal1 has joined #openstack-infra12:50
mordredyolanda: ^^ you're using infracloud puppet not for infra ... will that break you?12:50
*** rlandy has joined #openstack-infra12:50
mordred(I think it won't - but just want to check)12:50
*** markvoelker has joined #openstack-infra12:50
*** nwkarsten has joined #openstack-infra12:50
*** links has quit IRC12:51
*** gouthamr has joined #openstack-infra12:51
*** Goneri has joined #openstack-infra12:54
*** abregman_ has quit IRC12:54
* rcarrillocruz reads backscroll12:56
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add non-voting multi-region functional test to OSA os_swift
rcarrillocruzoh nice12:56
rcarrillocruzsdague: so the strace stuck on the read syscall was due to a pass prompt?12:57
sdaguercarrillocruz: yes12:57
sdagueosc goes into a blocking prompt with no timeout12:57
*** yamamoto has quit IRC12:57
rcarrillocruzpabelanger: cool, i was just aware about unbound on baremetal0012:58
rcarrillocruzapproved the change12:58
rcarrillocruzamrith: let me check , not sure if it was changed from jenkins to zuul12:58
rcarrillocruza job log may show12:58
rcarrillocruzamrith: the tests puts the stuff on /home/jenkins/workspace, so I assume it's jenkins12:59
amrithI found it12:59
amrithit is jenkins12:59
amrithbut I couldn't (initially) get into the trove guest12:59
*** spzala has joined #openstack-infra12:59
amrithbecause it was using a key I dind't expect12:59
amrithlooks like cloud-init is pooched12:59
amrithI can see what is failing12:59
amrithhave to figure out why12:59
amrithI've seen this before12:59
amrithand entered a bug on cloud-init13:00
pabelangerrcarrillocruz: ya, I didn't change baremetal00. But good to know it is running unbound13:00
amrithand I was given a promise that this problem wouldn't hit u14.0413:00
mordredyou could always shift to glean instead of cloud-init ...13:00
amrithonly 15.0413:00
amrithand 16.0413:00
amrithbut this is now on 14.0413:00
rcarrillocruzstack iirc13:00
rcarrillocruzsdague: ^ ?13:00
rcarrillocruzoh sorry, i was scrolled back, responded to the linux username question on devstack13:00
openstackLaunchpad bug 1602373 in cloud-init (Ubuntu Xenial) "cloud-init doesn't always land files that one expects" [High,Fix released]13:01
rcarrillocruzpabelanger: so the intent for putting unbound is to hit the dns cache to avoid , or you pasted it to me for another root cause problem?13:01
*** drawsmcgraw has left #openstack-infra13:01
amrithhi monty13:02
*** sshnaidm is now known as sshnaidm|mtg13:02
amrithwe're using (at the whim of) whatever nova uses under the covers, we just ask to inject a file. how it gets there (USPS, Fedex, UPS, ...) we don't know13:02
mordredI suggest not using file injection at all13:02
*** andreykurilin has left #openstack-infra13:02
*** andreykurilin has joined #openstack-infra13:03
mordredjust put an ssh key on the guest and use ssh to put files where you need them - ansible is a good tool for doing that if an ssh key is there, but obviously you can also script that in a ton of different ways13:03
*** mbacchi_ is now known as mbacchi13:03
mordredI mean, that's not a thing you're going to change today obviously13:04
pabelangerrcarrillocruz: Ya, that was the reason we failed to launch a node with nodepool13:04
mordredbut one of the things we've learned here in infra is to keep the amount of stuff that happens in the nova-metadata to cloud-init/glean interaction to a bare minimum13:04
amrithmordred, the issue is configurations that are injected at runtime13:04
amrithyes, we could get those too13:04
amrithin a bootstrapping manner13:04
mordredI'm not saying don't inject them13:04
sdaguemordred: your patch failed on a subnode issue that is unlogged13:04
mordredjust don't use nova metadata to do so13:04
rcarrillocruzi'm glad is just that and not some net issues on HPE side :D13:04
mordredsdague: YAY!13:04
amriththe issue here is that the injection isn't working13:04
amrithI think I've figured out why13:05
pabelangerrcarrillocruz: Figured, we should setup unbound first and see if we can reproduce, since google will throttle DNS requests13:05
sdaguethere are no subnode logs there13:05
*** binbincong has quit IRC13:05
AJaegermordred: yolanda is on FTO13:05
*** jamesden_ has joined #openstack-infra13:05
mordredsdague: I believe my complete answer is "awlkjevcalskencalserifnasldkcasdf"13:05
sdaguemordred: it's a super weird fail13:06
mordredsdague: although please imagine me banging my head against something whilst saying that13:06
sdaguecan you re promote13:06
*** skraynev_ has joined #openstack-infra13:06
mordredyah. one sec13:06
sdaguethis never even got to your code, I have no idea why zuul just shot the node in the head13:06
*** vikrant has quit IRC13:06
*** drawsmcgraw has joined #openstack-infra13:06
sdagueit starts at 12:4113:07
pabelangermordred: jeblair_ : rcarrillocruz: So we need to remove HPE from spamhaus:
*** salv-orlando has joined #openstack-infra13:07
*** markvoelker has quit IRC13:07
sdague170 minute timeout set at 12:4213:07
pabelangercause I am getting flooded with Mail Delivery System emails13:07
sdaguegrenade starts at 12:5013:07
sdagueat 12:54 zuul says - job failed13:07
rcarrillocruzpabelanger: i have to admit i have no idea how to do that, we fill a form or something?13:07
pabelangerrcarrillocruz: Ya, I think we just request it from the website13:08
pabelangerbut I haven't done it before13:08
sdagueand the last thing grenade is doing is downloading a pip13:08
sdague2016-09-01 12:54:54.323 |   Downloading
*** Julien-zte has joined #openstack-infra13:08
mordredwow: is listed on the Policy Block List (PBL)13:09
* rcarrillocruz clicks the button "begin the removal process'13:10
mordredoh - because those servers are in an HPE address range but we're not sending email through the HPE mail servers13:10
*** abregman_ has joined #openstack-infra13:10
mordredrcarrillocruz: ++13:11
sdague2016-09-01 12:54:50,502 p=21911 u=zuul |  fatal: [node]: FAILED! => {"failed": true, "msg": "Failed to connect to the host via ssh."}13:11
*** caowei has joined #openstack-infra13:11
amrithrcarrillocruz, thx much. I'm done with my test instance for now. I'm going to figure out how to fix it and see what I can do. many many thx (and a great coincidence happening upon your conversations with sdague)13:12
mordredsdague: we're having all the luck this morning13:12
rcarrillocruzhmm, the form doesn't seem to allow a CIDR13:12
*** salv-orlando has quit IRC13:13
*** abregman_ is now known as abregman|mtg13:13
sdaguemordred: yeh, unfortunately I don't even know why zuul was doing that13:13
sdagueit doesn't make any sense13:13
sdagueit should have been heads down waiting for grenade run to return13:14
*** akshai has joined #openstack-infra13:14
sdaguewhich was chugging along13:14
sdaguebut then it made another ssh call in parallel, which died13:14
sdagueand it blew up the job13:14
AJaegersdague: could you review later a change to project-config: - I tend to -1 but would like your review13:14
*** amotoki_ has quit IRC13:14
mordredsdague: that's how async jobs in ansible work13:15
mordredsdague: it spawns the job ,and then periodically checks on it13:15
mordredto make sure it hasn't blown up13:15
sdaguemordred: ok... how often does it false negative there?13:15
*** matt-borland has joined #openstack-infra13:15
sdaguebecause from the logs, there was nothing wrong13:15
mordrednot a ton - there were some problems like a month ago but those got sorted13:15
mordredwe're just extra lucky this morning13:16
sdaguebut it was downloading packages, so it might have saturated the link13:16
sdaguewhat's the timeout on that check?13:16
*** gongysh has joined #openstack-infra13:16
sdagueand is it tunable?13:16
pabelangerI believe we default to 1013:17
rcarrillocruzpabelanger: i'm not a mail master, but would it work if we relayed the mail from those machines to a HPE mail server13:17
rcarrillocruzcos i'm reading spamhaus13:17
sdaguepabelanger: could we up that to 30 or 60?13:17
rcarrillocruzand it seems to remove an entire CDR block we have to have an account or something on spamhaus13:17
*** moravec has quit IRC13:17
rcarrillocruzotherwise is ip by ip, one by one13:17
sdagueI can definitely imagine a 10 second timeout might wedge if we're going full bore on the network13:18
pabelangerrcarrillocruz: that or we setup our own relay13:18
rcarrillocruzi vote the latter13:18
rcarrillocruzputting it on the bifrost machinee or something13:18
*** rossella_s has quit IRC13:18
rcarrillocruzis not going to take much cpu13:18
*** cardeois_ has joined #openstack-infra13:18
openstackgerritMerged openstack-infra/shade: Allow str for ip_version param in create_subnet
pabelangersdague: I'm sure we could bump it, I cannot think of a reason not to13:19
*** rossella_s has joined #openstack-infra13:19
sdaguepabelanger: can you point me to where it is specified?13:19
sdagueI'll submit a patch13:19
*** pgadiya has quit IRC13:19
*** larsks has left #openstack-infra13:19
mordredsdague: line 1106 in zuul/launcher/ansiblelaunchserver.py13:20
*** yamahata has joined #openstack-infra13:20
*** moravec has joined #openstack-infra13:20
pabelangeris the setting13:20
mordredpabelanger: ah - we are talking about different timeouts13:21
mordredsdague: listen to pabelanger - he's righter than me13:21
sdaguewell, is it that, or the watchdog?13:21
*** dizquierdo has joined #openstack-infra13:22
pabelangerI think we need to update ssh_args13:22
*** cardeois has quit IRC13:22
sdague ?13:22
pabelangergive me a second13:22
sdaguepabelanger: ok13:22
rcarrillocruzsdague: are you done with the held node from earlier13:22
sdaguercarrillocruz: yes13:22
openstackgerritMerged openstack-infra/project-config: Add deb-bandit and deb-python-hacking repo
*** markvoelker has joined #openstack-infra13:23
mordredsdague: btw - next week I plan on hacking in this area of zuul13:23
sdaguemordred: ok, cool13:23
pabelangermordred: sdague: Ya, I think is the right setting that we need to change13:24
sdaguepabelanger: ok, where would that be set?13:24
pabelangerthat is our ansible.cfg per test run13:24
mordredpabelanger: I think we should also investigate why controlpersist isn't in effect here13:24
pabelangermordred: maybe we need to bump the value? we only default to 6013:26
*** berendt has quit IRC13:26
*** asettle is now known as asettle-afk13:26
*** yamamoto has joined #openstack-infra13:26
pabelangermordred: we also poll pretty aggressively, every 5 seconds. Maybe we should bump that to 60 seconds13:27
*** yamamoto has quit IRC13:27
*** markvoelker has quit IRC13:28
zigomordred: pabelanger: fungi: Hi, I'm back. Sorry that I was absent since Friday, but unfortunately, my father died, and I went to see familly and such. Now I'm back at work, even though I not really in such a mood... Anyway, did you guys have time to discuss about giving us merge ACLs within packaging-deb?13:28
openstackgerritSean Dague proposed openstack-infra/zuul: bump timeout on ssh commands to 30s
*** eharney has joined #openstack-infra13:29
pabelangermordred: or some percentage of the value of timeout13:30
pabelangerzigo: let me find the log13:30
*** yamamoto has joined #openstack-infra13:30
*** amotoki has joined #openstack-infra13:31
zigopabelanger: Cheers.13:31
pabelangerzigo: that was the discussion on the subject.13:31
*** yamamoto has quit IRC13:32
*** yamamoto has joined #openstack-infra13:33
*** roxanaghe has joined #openstack-infra13:33
*** yamamoto has quit IRC13:34
openstackgerritMerged openstack-infra/system-config: Install unbound on infracloud servers
*** szaher_ has quit IRC13:34
AJaegerzigo: please check - I'd like to see that repo fixed.13:34
*** yamamoto has joined #openstack-infra13:34
*** yamamoto has quit IRC13:34
*** szaher has joined #openstack-infra13:34
pkoniszewskiAJaeger: thanks for pointing this out, I wasn't aware of it13:37
*** roxanaghe has quit IRC13:37
pkoniszewskiAJaeger: so what's the best way to work on such gate? propose new job to experimental queue?13:38
AJaegerpkoniszewski: yes, an experimental job is the only way to do this...13:39
silehtsdague, hi, do you have sometimes to discus about this ? it seems you perhaps have a better solution.13:40
AJaegerpkoniszewski: next time, it would also help if you explain in the commit message what you want to test ;)13:41
pkoniszewskiAJaeger: okay, I will, and I need to start work on experimental job then, thanks a lot! :)13:42
*** rajinir has joined #openstack-infra13:43
*** lock__ is now known as ianychoi13:43
*** amotoki has quit IRC13:43
*** gordc has joined #openstack-infra13:44
*** sshnaidm|mtg is now known as sshnaidm13:44
sdaguesileht: I definitely don't have the brain bw to handle that atm13:45
openstackgerritPeter Zhurba proposed openstack-infra/project-config: Add repo for openstack/puppet-glare.
*** mriedem has joined #openstack-infra13:46
silehtsdague, ok no problem, can I re-ask tomorrow  ? or do you prefer to come back to me when you have the time ?13:46
sdaguesileht: this has to be post freeze once the release has quieted down13:47
*** asettle-afk has quit IRC13:47
sdaguehonestly, this probably is better to be done in a design summit session13:47
sdaguebecause it sounds like there is a broader range of testing that's desired13:47
*** asettle-afk has joined #openstack-infra13:48
openstackgerritPaul Belanger proposed openstack-infra/nodepool: Retry ssh connect when EOFError is received
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: TEST: DONT RECHECK: periodic jobs
silehtsdague, its for projects that have intermediate release and want to test upgrade with previous synced release13:49
*** openstackgerrit has quit IRC13:49
pabelangerjeblair_: please let me know what you think about 364322^, we are seeing random EOFError in nodepool debug logs and think we should retry our SSH connection over aborting.13:49
*** openstackgerrit has joined #openstack-infra13:49
*** amotoki has joined #openstack-infra13:50
*** sarob has joined #openstack-infra13:51
*** spiette has joined #openstack-infra13:52
*** asettle-afk has quit IRC13:52
*** spiette has left #openstack-infra13:52
*** drifterza has quit IRC13:54
sdaguesileht: right, so here is the thing, it's a bunch of new complexity, for a small number of projects. So I'm honestly not all that keen on the approach.13:54
*** Guest77956 is now known as jlvillal13:55
*** jamesden_ has quit IRC13:55
rcarrillocruzpabelanger: i'm going to add a param to 'images' on the cloud launcher, not tied to the os_image module for pulling images from the internet. Meaning, if we have an image with a field "download_file_url", the launcher pulls it to /tmp and uploads it to glance13:55
*** rodrigods has quit IRC13:55
*** rodrigods has joined #openstack-infra13:55
*** sarob has quit IRC13:55
rcarrillocruzeven if that's not a glance v2 feature, that's super useful to have, we could automate the initial images for our cloud providers to bring up the mirrors, etc13:56
rcarrillocruzwithout having to do it manually13:56
*** jamesden_ has joined #openstack-infra13:56
mordredrcarrillocruz: ++13:56
mordredrcarrillocruz: I think that's a GREAT feature13:56
*** xyang1 has joined #openstack-infra13:56
rcarrillocruzi'll show you later mordred , spinning my dsvm local instance (enough infracloud for me , i wanna code!)13:57
*** nwkarsten has quit IRC13:57
*** dtroyer has joined #openstack-infra13:57
*** inc0 has joined #openstack-infra13:57
*** eharney has quit IRC13:58
*** salv-orlando has joined #openstack-infra13:59
*** amotoki has quit IRC13:59
*** eharney has joined #openstack-infra14:00
*** zz_dimtruck is now known as dimtruck14:00
*** yamamoto has joined #openstack-infra14:00
*** ddieterly has joined #openstack-infra14:01
*** jheroux has joined #openstack-infra14:02
*** oanson has joined #openstack-infra14:03
*** hongbin has joined #openstack-infra14:03
*** cardeois has joined #openstack-infra14:03
*** asettle-afk has joined #openstack-infra14:03
*** jaosorior has quit IRC14:04
*** asettle-afk is now known as asettle14:04
YorikSarianw, rcarrillocruz, AJaeger: Thanks a lot for your swift review.14:05
*** cardeois_ has quit IRC14:05
*** kushal has joined #openstack-infra14:06
*** matthewbodkin has quit IRC14:06
*** fguillot has joined #openstack-infra14:06
*** rbrndt has joined #openstack-infra14:06
*** matthewbodkin has joined #openstack-infra14:06
*** pilgrimstack has quit IRC14:07
*** xarses has quit IRC14:08
*** javeriak has quit IRC14:08
rcarrillocruzdoh, spacex falcon 9 has exploded :/14:09
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: Use proxy on OVB environemnt
*** fguillot has quit IRC14:10
*** claudiub has quit IRC14:10
*** sandanar has joined #openstack-infra14:11
pabelangerrcarrillocruz: unbound live on infracloud14:11
pabelangerDNS working too14:11
pabelangerSSHException: No existing session14:13
pabelangerparamiko raises that from time to time14:13
*** Genek has joined #openstack-infra14:14
*** skraynev_ has quit IRC14:14
*** amotoki has joined #openstack-infra14:14
*** pilgrimstack has joined #openstack-infra14:15
*** tongli has joined #openstack-infra14:15
*** itisha has joined #openstack-infra14:15
*** pt_15 has joined #openstack-infra14:17
openstackgerritRob Cresswell proposed openstack-infra/project-config: Add Ironic UI gerritbot to #openstack-ironic
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Nothing to see here
sdagueoh neat when did the web console hit zuul?14:18
sdagueoh, it didn't, just something changed with the way chrome launched it here14:18
openstackgerritMerged openstack-infra/tripleo-ci: Sync the config of the ha JOBTYPE with ha2
*** sputnik13 has joined #openstack-infra14:20
*** coolsvap has quit IRC14:22
*** skraynev_ has joined #openstack-infra14:22
*** tqtran has joined #openstack-infra14:22
*** hsammic has joined #openstack-infra14:22
openstackgerritPaul Belanger proposed openstack-infra/project-config: Revert "Revert "Increase rh1 max-servers to 75""
*** mtanino has joined #openstack-infra14:24
pabelangermordred: clarkb: ^ Now that nodepool was restarted and shade upgraded, I think we can increase the nodes in tripleo-test-cloud-rh1 again.14:24
pabelangerFIP issues appears to be minimal right no14:24
*** mikelk has quit IRC14:24
*** watanabe_isao has quit IRC14:25
pabelangermordred: this seems new:
*** tqtran has quit IRC14:26
*** fguillot has joined #openstack-infra14:27
*** bogdando has quit IRC14:27
pabelanger is the debug from shade14:28
*** vhosakot has joined #openstack-infra14:28
*** skraynev_ has quit IRC14:31
*** signed8bit has joined #openstack-infra14:32
*** jamesden_ has quit IRC14:32
sdaguemordred: patch finally merged14:33
*** gongysh has quit IRC14:33
*** dimtruck is now known as zz_dimtruck14:33
*** zz_dimtruck is now known as dimtruck14:33
*** ddieterly is now known as ddieterly[away]14:34
*** ivyyy has quit IRC14:35
mordredsdague: woot14:36
*** eharney has quit IRC14:36
mordredpabelanger: looking14:36
*** jheroux has quit IRC14:37
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: TEST: DONT RECHECK: periodic jobs
*** abregman|mtg has quit IRC14:41
clarkbmordred: pabelanger the last time we had ssh issues like that control persist was fine we just broke ipv6. Maybe we are affecting the test instamce with some job side effect?14:41
*** oanson has quit IRC14:41
mordredclarkb: nod14:41
mordredpabelanger: I'm seeing nodes be created there - let's consider that one an outlier for the moment14:42
*** xarses has joined #openstack-infra14:42
*** jheroux has joined #openstack-infra14:42
mordredalso - we _definitely_ need to add floating ip batching14:42
mordredwe're slamming the hell out of the neutron api there14:42
*** caowei has quit IRC14:42
*** dimtruck is now known as zz_dimtruck14:43
pabelangerclarkb: Oh, didn't know that. Have a review handy that shows the fix?14:45
jpichIs there a story behind the Dell StorageCenter CI leaving a -1 on every patch in the Gerrit sandbox? Getting a 3rd party build failure notification before you even get the "Welcome, new contributor!" message is a bit rough :-) -
clarkbpabelanger: this was the accept ra thing in devstack14:45
pabelangerclarkb: okay14:45
pabelangerclarkb: I don't know the full back story on the failures, just that sdague wanted to increase the SSH timeout value14:46
*** kien-ha has joined #openstack-infra14:46
zigopabelanger: The packaging repo needs to hold a copy of the code from upstream for multiple reasons. Do I need to explain?14:47
*** ggnel_t has joined #openstack-infra14:47
clarkbpabelanger: right I was just suggesting the last time we had problems with that fiddling with timeout values would have no effect. And that we shouldnt ignore the possibility of another break neyworking scenario in the jobs14:47
*** tonytan4ever has joined #openstack-infra14:47
pabelangerclarkb: agreed, good to know14:48
pabelangerzigo: you might want to sync up with clarkb and fungi.  I didn't really give input on that topic.14:48
zigoclarkb: fungi: Do you have time to discuss this?14:49
pabelangerhowever, I am interested in the discussions14:49
*** jcoufal has joined #openstack-infra14:50
clarkbzigo I am mostly here now. The outcome seemed to be we could just ensure $repo is present to the right version when needed14:50
clarkbrather than force pushing into the deb repo14:50
zigoclarkb: I don't agree with this outcome, that's the point! :)14:51
zigoclarkb: First off, the repo for upstream get EOL very early.14:51
zigoI don't want this to happen in the packaging trees.14:51
zigoWe need to cover the life of Debian Stable for Newton for example.14:51
clarkbwe dont delete repos14:52
zigoBut branches are EOLed.14:52
clarkbthe tags dont14:52
clarkbnor should any sha1 go away14:52
zigoAnyway, the git-buildpackage workflow *IS* to import upstream tag/branch/tarballs into the packaging branch.14:52
*** kien-ha has quit IRC14:53
mordredyes, I concur with that14:53
zigoThere are ways to make it not do that, but then it's going to be over complicated for no reasons.14:53
zigoFor example, managing quilt patches will be horrible.14:53
sdagueclarkb: this is not breaking ipv614:53
*** Thelo_ has quit IRC14:54
*** abregman|mtg has joined #openstack-infra14:54
*** vtapia has quit IRC14:54
*** Sukhdev has joined #openstack-infra14:54
fungizigo: back now. i understand your _manual_ workflow involves merging a copy of nova commits into the pkg-nova repo, and your automated jobs can do precisely the same thing. that doesn't mean that the pkg-nova repo in _gerrit_ needs to carry a complete copy of nova commits14:54
sdagueclarkb: the last bit in the subnode log is downloading pip packages14:54
mordredfungi: I disagree a bit14:54
mordredfungi: because the gate isn't the only place where the packaging repo might be used to create packages14:54
mordredso it should not assume the encompasing environment that is our gate14:55
sdaguein the middle of that download, zuul ansible shoots it in the head14:55
clarkbif this is a reqyirement you should be a branch on the nova repo14:55
zigoRight. Before an upload to Debian, I'll just git clone and rebuild ...14:55
fungimordred: sure. and the manual process (add a remote for the nova repo, pull from it into an upstream branch, merge) is still a valid workflow in that case, right?14:55
*** sshnaidm has quit IRC14:55
*** mhickey has quit IRC14:55
mordredfungi: that's a severe degredation in funcationality over normal git-buldpackage workflow14:55
clarkbsdague: I didnt say it broke ipv6 just consider the nodes can break themselves14:55
sdaguemy theory is that 10s may not be enough time if the node is flat out pulling packages14:55
*** kgiusti has left #openstack-infra14:56
*** ddieterly[away] is now known as ddieterly14:56
sdagueclarkb: sure14:56
mordredfungi: like, gbp exists and is used heavily - I don't think we should break it14:56
openstackgerritSergey Skripnick proposed openstack-infra/project-config: Add python-db-jobs to Glare
*** yfried has quit IRC14:56
mordredzigo: I'd like to make a suggestion that might be a compromise14:56
sdaguehowever, where this is, I don't think that's what happened14:56
fungii'm just concerned that now not only do we have hundreds of separate repos for all packaged software, we have additional copies of all commits for all packaged software14:56
sdagueclarkb: if you have another thought from those logs, please go and look14:56
mordredfungi: right. I don't think we need hundreds of commits in the upstream branch14:56
mordred"zigo | Anyway, the git-buildpackage workflow *IS* to import upstream tag/branch/tarballs into the packaging branch."14:57
mordredtag/branch is one of the options14:57
mordredthe other option is upstream tarballs, which get imported in a single commit and then tagged14:57
mordredthat is in line with how gerrit works14:57
*** krtaylor has joined #openstack-infra14:57
mordredand sohuld not require tons of infra-side automation ... unfortunately, it would be a non-trivial rework of the existing packaging repos I fear14:58
zigomordred: Even if you use tarballs, it imports the code in the packaging branch.14:58
*** kdas_ has joined #openstack-infra14:58
clarkbthere is somethi g a little funny about hating vendored code14:58
mordredzigo: it does14:58
clarkbthen requiring it14:58
mordredzigo: but it's not 100 commits - it's one commit14:58
mordredzigo: whic hmeans you could submit that commit to gerrit using git review like normal14:58
mordredthen land it14:58
fungialso we've come full circle from "you should avoid putting the debian package metadata in your upstream source tree" to "you should put a copy of the upstream source tree in the repo where you have your debian packaging metadata"14:58
*** edtubill has joined #openstack-infra14:58
*** zhurong has quit IRC14:58
mordredfungi: it's not the same thing at all14:58
zigomordred: The point is *also* to be able to use *any* commit upstream (and not just a branch or a tag) and package that.14:59
mordredputting debian package metadata in upstream is problematic because of sequencing issues14:59
zigoCould you explain what the problem is? Do we lack resources?14:59
mordredif you want to release version 1.2 of an upstream software, what state should the packaging be in in upstream tag 1.214:59
zigoIs it just too big?14:59
fungii understand the reasons, i'm asking that we separate _process_ (steps to build a package) from revision control (data we store for building a package)14:59
mordredit necessarily cannot be the packaging that knows how to handle 1.2, because 1.2 doesn't exist until 1.2 exists14:59
*** _nadya_ has quit IRC15:00
zigofungi: If we get an ACL to merge tags, it is the same process, it goes under the Gerrit code review.15:00
zigoI tried, the only issue is the missing ACL.15:00
zigo(at least, it looks like it)15:00
*** vtapia has joined #openstack-infra15:00
*** sshnaidm has joined #openstack-infra15:00
fungiall the data we need to build a debian package of nova exists in a combination of the nova repo and the pkg-deb repo. the steps to build the package can include combining those. the combined result doesn't have to get pushed back into the revision control history though, does it?15:00
zigofungi: If we don't, then it becomes a way harder to do *many* things.15:01
*** kushal has quit IRC15:01
zigoLike adding a quilt patch, or building before uploading to Debian.15:01
fungii don't understand. what prevents you from adding a git remote for the nova repo, pulling from it into an unstream branch and then doing a git merge?15:02
mordredI feel like we're trying ot fight the current established best practice of debian packaging (use gbp with upstream sources included in the packaging repo) in the context of openstack infra, which seems like a very strange thing for us to be in the business of doing15:02
fungiit doesn't seem like we're fighting it at all15:02
*** kushal has joined #openstack-infra15:02
clarkbmordred: do you at least agreeforce pushing noca into a different repo is a bad idea?15:03
*** pcaruana has quit IRC15:03
fungii'm saying it's a step which can be performed locally, the same as it can be performed in our ci. why does the result (a duplicative, generated result) need to be pushed back into revision control history?15:03
*** eharney has joined #openstack-infra15:03
mordredclarkb: I believe we sohuld be importing tarballs using gbp pristine-tarball15:03
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: TEST: DONT RECHECK: periodic jobs
mordredfungi: but it's a manual step which is not how gbp works15:03
*** kien-ha has joined #openstack-infra15:03
fungiyou don'y manually invoke gbp?15:03
mordredif it was a normal operational mode of gbp, then sure15:03
*** annegentle has joined #openstack-infra15:04
mordredyes - but you run git-buildpackage and it does all the thigs15:04
fungiit seems like any manual process involves _at_least_ one manual step15:04
mordredyes. the one step of running git-buildpackage15:04
openstackgerritLukas Bednar proposed openstack-infra/jenkins-job-builder: Builders: Add ansible-playbook builder
pabelangerzigo: mordred: What is the objection to pulling in the upstream git repo or tarball at build time?  When I was rolling packages with git-buildpackages, I would use uscan to fetch them before git-buildpackage. Obviously skipping using the upstream-branch step15:04
fungibefore running gbp, you probably _also_ have a manual step of cloning the repo you're going to run it in, yeah?15:05
mordredif for openstack packages it's clone the repo, now do this other stuff to get another repo added to this repo, then run git-buildpackage, then we've subverted the power of the tool15:05
*** ggnel_t has quit IRC15:05
*** kien-ha has quit IRC15:05
mordredthe steps are "clone the packaging repo, cd to the packaging repo, run git-buildpackage" ... what I'm saying is that if we inject more manual steps into that, then we're doing weird things15:05
zigofungi: pabelanger: 1/ It makes it very hard to manage quilt patch 2/ it's not the usual git-buildpackage workflow, so it will confuse contributors 3/ It will be a lot of pain points for no reasons maintaining scripts which we otherwise wouldn't need to write15:05
*** kdas_ has quit IRC15:05
zigofungi: pabelanger: What's the reason for *NOT* doing it?15:06
pabelangermordred: zigo: I would use overlay, IIFC15:06
*** thcipriani|afk is now known as thcipriani15:06
pabelangerzigo: eg:
mordredpabelanger: right. with overlay you have to maintain local patches in quilt15:06
clarkbzigo: the reason for not pushing all of nova into nova deb is nova code has a home it does not need two homes. its goimg to double the size of our test image cahes and so forth15:06
*** rcernin has quit IRC15:06
fungii guess my point is that i've seen plenty of different gbp workflows. some maintainers import tarballs, some pull from upstream git repos, some use quilt, some use single-patch, some push copies of upstream code back into public revision history and some don't...15:06
mordredthat is very differnt than using git15:06
mordredfungi: right.15:07
clarkbyou are aslo asking to bypass every control on herrit15:07
pabelangermordred: zigo: yes, that is what I would do, quilt15:07
zigopabelanger: Last time we discussed using this in the Python module team, absolutely everyone but a single person liked using overlay.15:07
mordredso ...15:07
openstackgerritJeffrey Zhang proposed openstack-infra/project-config: Increasing the timeout value for kolla jobs
mordredI do not think we should completely rework the 175 packaging repos that have a multi-year packaging history already15:07
pabelangerzigo: hehe, ya, I prefer overlay too15:07
mordredto use quilt instead of git15:07
fungii'm requesting flexibility in choosing a workflow that meshes well with the situation where upstream and packaging repositories are already in the same system and avoid keeping duplicate copies15:07
fungimordred: well, they use quilt _and_ git afaik15:08
pabelangermordred: Ya, now I understand the issue.15:08
*** kien-ha has joined #openstack-infra15:08
fungiquilt packages carried in teh debian/patches dir, committed into the packaging branch15:08
*** mptacekx has quit IRC15:08
mordredfungi: right. but there is no need to introduce quilt into an existing ecosystem that does not already use quilt15:08
pabelangerFWIW: I didn't mind quilt, but I am not the one doing the packaging15:08
mordredthat is one way these _could_ have been done15:08
mordredbut it's not the way they were15:08
fungiwell, i wasn't specifically suggesting to use quilt. i was giving it as an example (and i thought zigo already carried debian/patches in his packages but i'll pull a source package and double-check that)15:09
zigoI do use quilt, I'm not sure what fungi suggests here...15:09
mordredzigo: wait - you're already using quilt?15:10
mordredwhy are we having this argument then?15:10
clarkbanyways my biggest concern is we dont bypass gerrit by allowing force pushes and we dont doubt the size of our test images by literally copy.pasting all our repos15:10
zigomordred: I don't understand ! :)15:10
*** yamamoto has quit IRC15:10
*** zz_dimtruck is now known as dimtruck15:11
*** mhickey has joined #openstack-infra15:11
mordredzigo: importing upstream sources into the packaging repo is a thing you do as an alternative to using quilt to manage local changes15:11
zigomordred: No.15:11
mordredzigo: if you're using quilt, then just pointing at release tarballs should be fine15:11
zigoI use quilt patches to make sure I keep Debian specific patches separated.15:11
zigoI don't see any relationship between using quilt and the current gbp workflow.15:12
pabelangerRight, that is how I understood it too.15:12
mordredright. in a git workflow you can have each of those debian specific patches in the debian packaging branch and gbp will create the patch overlay and packaging build - you don't need the quilt, you have git15:12
pabelangerclarkb: Right,
*** kien-ha has quit IRC15:12
mordredhowever, that's getting into the weeds15:12
zigoNop! :)15:12
zigoThat's not what it is about.15:13
pabelangeris basically pytnon-os-client-config repo, with debian folder added15:13
zigoAt least, that's not how I use quilt.15:13
pabelangerclarkb: so, a lot of duplicated code15:13
zigomordred: What you're talking about is more like if we were using git-dpm or gbp pq.15:13
zigopabelanger is right.15:13
pabelangerI always thought we we're just talking about the debian folder for the packaging repos15:13
fungithe question being floated is, for example, whether we should import a complete copy of nova into the pkg-nova repo15:14
zigoWe're talking about having upstream code/tag within the debian packaging branch, yes.15:14
*** pblaho has quit IRC15:14
zigoWhich is what 99% of package maintainers do.15:14
mordredyah. if we're keeping patches in quilt, I see no point in having the upstream sources in the repo. I _do_ argue in favor of upstream sources in the repo so that quilt can be avoided15:14
clarkbpabelanger: no this is literally copying all history an potentially making more than one copy one for each branch (not sure how its organized)15:15
fungiand my position, which i've yet to see clear evidence indicating it's not viable, is that pulling nova source into the pkg-nova repo could happen at job runtime rather than being a manual process the results of which are pushed back into the repo in gerrit15:15
zigoThen how do you actually manage to run dpkg-source --commit ?15:15
pabelangerclarkb: looking at and it is slightly confusing which one is upstream, since both master branches are the same15:16
openstackgerritMerged openstack-infra/jenkins-job-builder: Add support for Parameterized Scheduler Plugin
zigofungi: I'm not saying it's not possible to do it, I'm saying it's a *WAY* more complicated to do what you're describing.15:17
zigofungi: First, you got to design the code to do the pull / push from here and there.15:17
*** Thelo_ has joined #openstack-infra15:17
*** sandanar has quit IRC15:17
*** dtantsur is now known as dtantsur|afk15:17
fungizigo: we actually do a lot of tat in other jobs anyway15:17
zigofungi: Then, when designing a quilt patch, you need to do many things so that you can finally type dpkg-source --commit15:17
pabelangerI'm not sure it is way more complicated, but it does delay the packaging process15:17
*** sputnik13 has quit IRC15:18
zigofungi: Then, we'd have to do more work to upload to debian as well, instead of just git clone, build, upload.15:18
zigoI've given my argumentation about why I prefer the current workflow, I still don't get why you prefer not using it.15:18
fungii'm suggesting that when developing locally you can pull from the nova repo while writing your quilt patch, but only push the quilt patch back into the pkg-nova repo rather than merging a copy of all of nova into the pkg-nova repo15:18
*** Thelo_ has quit IRC15:18
zigofungi: Yes, that's more manual operations !15:19
*** krotscheck has joined #openstack-infra15:19
*** mlavalle has joined #openstack-infra15:19
zigoTo me, it just looks like you're saying "whaaaaat? We'll have 2 copies?". Is there any other point of argumentation besides that?15:19
fungiand is avoiding that one manual operation worth keeping an entire extra copy of all upstream commits to all packaged projects in gerrit?15:19
zigoAlso, what would be the workflow if I want to package commit 28af82dec ?15:20
mlavallehi is there anybody here who can help me with a logstash question?15:20
*** sputnik13 has joined #openstack-infra15:21
zigoThe version should be something like 1:2.3.4+2016.09.01.git.28af82dec-5+13~bpo8+115:21
zigoDo you suggest that we add some logic to extract the git sha256, and pull the relevant things in the packaging branch, in the build script?15:21
zigoThis looks very hackish to me.15:22
clarkbzigo: I listed my two reasons why I don't like it15:22
pabelangerzigo: right, we'd have to do that15:22
zigoclarkb: I was on the phone at the same time, I'm sorry, I probably missed them.15:22
zigo(now I'm all focussed...)15:22
pabelangerzigo: and we do today, some jobs will use zuul-cloner to fetch specific commits. This could be used in your build process with the Depends-On field15:23
*** jcoufal_ has joined #openstack-infra15:23
*** sdake has joined #openstack-infra15:23
*** sputnik13 has quit IRC15:23
*** Sukhdev has quit IRC15:24
clarkbzigo: I don't want multiple copies of every repo floating around as it puts pressure on already large test images which we can't reliably update. I also don't want a git process that bypasses every control in Gerrit15:24
*** kien-ha has joined #openstack-infra15:24
pabelangerrcarrillocruz: over an hour and no launch failures in infracloud15:25
*** jcoufal has quit IRC15:25
zigoclarkb: There's no such thing as "bypasses every control in gerrit", if I understand correctly.15:25
pabelangerrcarrillocruz: our time to ready is still a little high, but we could still be distributing images15:26
clarkbzigo: what you want is to be able to force push arbitrary commits into your repos15:26
zigoclarkb: I believe a git merge -X <tag-name> would just end up in a normal CR.15:26
clarkbzigo: that is bypassing the controls we have put in place15:26
clarkboh you are wanting to push a merge commit? ok thats better. The previous discussion said you all needed to push the tags straight in15:26
zigoclarkb: That isn't what it seemed when I tried to "git review" a merge commit.15:26
*** armax has joined #openstack-infra15:26
*** jordanP has joined #openstack-infra15:26
*** esp has joined #openstack-infra15:26
clarkbbut its still creates the other problem of doubling (or worse) the size of all the repos in gerrit15:27
clarkbmordred: zigo if we do that I will suggest we not cache any of the packaging repos on our test images15:27
zigoclarkb: Basically, I want to be able to do: git merge -X theirs <TAG_NAME> .... some packaging change .... git commit -a --amend && git review15:27
clarkbbecause it just doesn't scale that way15:27
*** mhickey has quit IRC15:27
zigoclarkb: That, you guys decide, that wouldn't be me.15:28
zigoIf you think it's better to remove the cache, that's an -infra team decision.15:28
*** Thelo_ has joined #openstack-infra15:28
clarkbzigo: not that its better just that I am only interested in carrying one copy of the data in the space constrainted environment15:28
zigoI've been doing git clone on each build for a *very* long time on my jenkins, so that's ok to me.15:28
*** ifarkas is now known as ifarkas_afk15:28
*** vinaypotluri has joined #openstack-infra15:29
zigoclarkb: We're talking about around 1GB of data here.15:29
*** sputnik13 has joined #openstack-infra15:29
zigoDo we really need to save THAT much?15:29
clarkbzigo: its not just 1GB15:30
clarkbits 1GB in nova then 1GB in pkg nova15:30
clarkbthen in six months is 1.5GB and 1.5GB15:30
zigoNop, that's the total for all of the Git in all of my packaging.15:30
clarkb(illustrative only not real numbers)15:30
clarkbzigo: yes trying to illustrate the growth pattern here though15:30
zigoNova is huge, other packages are mostly very small.15:30
clarkband as mroe and more projects happen and you package more and more repos...15:30
clarkbthe git cache on our images is huge right now and accounts for most of the disk use of those15:31
*** rcernin has joined #openstack-infra15:31
pabelangerI think the issue is, we are proposing another method of doing it, which is different then the workflow you are accustomed too. And the potential amount of work needed to switch workflows15:31
zigoFor example, a non-bare repo of deb-python-debtcollector is not even 1MB with all the git history.15:31
zigopabelanger: That's one big issue too. And I'm supposed to have all of OpenStack Newton ready this week too ...15:31
*** senk has quit IRC15:31
pabelangerWe could use openstack-infra/zuul-packaging as a POC, because I always envisioned using git-buildpackage with overlay to produce its package builds.15:32
*** annegentle has quit IRC15:32
zigoCould we agree to discuss this *later*, ie when I'm not in the rush of a release?15:32
*** sputnik13 has quit IRC15:32
pabelangerzigo: yes, this would impact that deadline. As we need to workout the process15:32
clarkbwe turned off the du'ing due to the xenail build issue so I can't easily just look at logs for a disk size number15:32
clarkbbut over 7GB I think15:33
zigoI still hope to release Newton with infra, but if we decide to switch to another workflow, I give up for this release.15:33
*** senk has joined #openstack-infra15:33
*** javeriak has joined #openstack-infra15:33
zigoI don't want to risk having a bad OpenStack release for Debian Stretch (frozen at the end of the year).15:34
openstackgerritPeter Zhurba proposed openstack-infra/project-config: Add repo for openstack/puppet-glare.
zigo(and Mitaka is not an option as Horizon/Mitaka isn't Django 1.10 compatible)15:34
*** markusry has quit IRC15:35
*** Hal1 has quit IRC15:35
*** andreas_s has quit IRC15:35
pabelangerNot sure I follow, your going to upload the builds to debian from debian-openstack repo?15:36
*** dizquierdo has quit IRC15:36
rcarrillocruzpabelanger, mordred :
rcarrillocruzi still need to put a test for it, but you get the idea15:37
rcarrillocruzclarkb: we were also talking about ^ these days15:37
zigopabelanger: I'm saying that I wont have the necessary time to change everything (workflow, build scripts, etc) given the time constraints of the Newton release.15:37
clarkbpabelanger: zigo openstack/ is 6.7 GB15:37
clarkbso if packaging deb is 1GB you represent more than 1/7th of our entire disk use15:37
zigoclarkb: This includes stuff I don't need.15:37
clarkbzigo: yes I am comparing you to the whole15:38
clarkband that will only grow as the rest of openstack/ grows because you are a copy of a significant chunk of it15:38
pabelangerzigo: right, you want to use same build script for both repos.  I was confused you wanted to drive package builds from openstack into debian some how.15:38
* zigo tries now to have actual real numbers15:38
*** rhallisey has quit IRC15:39
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: WIP - Implement overcloud upgrade job - Mitaka -> Newton
*** mhickey has joined #openstack-infra15:39
clarkbI am totally happy to cache those repos and host them. I just don't think they should include entire copies of the other repos we host15:39
pabelangerclarkb: Ya, that was my thought on using overlay option for git-buildpackage15:39
pabelangerotherwise, why not just include debian folder into nova?15:40
pabelangerrcarrillocruz: seems right, can we add test for devstack?15:41
pabelangerrcarrillocruz: Oh, I see you are working on that15:41
zigoclarkb: You do realize that this also includes 3rd party python modules?15:41
rcarrillocruzyep, follow up patch15:41
zigo(so that the reasoning doesn't apply for them...)15:41
clarkbwe also created some of the repos multiple times in gerrit15:41
*** jheroux has quit IRC15:42
*** Swami has joined #openstack-infra15:42
clarkbso my ask is just that we bit a bit more careful and not spam gerrit with a bunch of unnecessary copies of things. AIUI this is possible15:42
zigoCurrently, in all the deb-* are 1.5 GB bare git repos.15:42
*** salv-orlando has quit IRC15:43
zigoclarkb: Would it be possible to install the cache only in the Jessie image?15:43
openstackgerritMerged openstack-infra/jenkins-job-builder: Fix disabled always returning true
*** roxanaghe has joined #openstack-infra15:44
clarkbzigo: ya there is potential tuning we can do around that. However we have in the past tried to treat them all the same so that you don't have to consider such differences when moving among distros. Focus should be on the distro differences15:44
clarkbI think we shoul stick to that goal as much as possible.15:44
*** markusry has joined #openstack-infra15:44
zigoMakes sense.15:45
pleia2good morning15:45
*** esp has left #openstack-infra15:46
*** kaisers_ has quit IRC15:47
openstackgerritPaul Belanger proposed openstack-infra/project-config: Bring osic-cloud8 online with 10 nodes
pabelangerclarkb: rcarrillocruz: fungi: brings osic-cloud8 online^15:49
pabelangerso far both internap-mtl01 and infracloud-vanilla look good15:49
rcarrillocruzhas the multinode issue being solved15:50
rcarrillocruzsdague: ^15:50
rcarrillocruzsorry mr dake :-)15:50
*** matrohon has quit IRC15:50
pabelangeryes, sdague should have included you15:50
sdaguercarrillocruz: yes15:50
rcarrillocruzso, we good to bump now15:50
rcarrillocruzi have a change to bump infracloud to 5015:51
rcarrillocruzand i increased quota this morning on the zuul project15:51
sdagueso, probably, I'd just ramp slowly and make sure to watch zuul like a hawk to look for new fails15:51
*** markvoelker has joined #openstack-infra15:51
sdaguewhat would be super cool is if nodes could be brought into a test run holding pen where some changes are just duped onto them15:52
sdagueso we could get burn in without blowing up real work15:52
pabelangerrcarrillocruz: So, I want to see what is needed to start getting mcollective going on infracloud. Or some other method for statsd15:52
pabelangersdague: Ya, I've often thought of that too15:53
rcarrillocruzsdague: so now it's on 10 nodes, my change is about bumping to 50. I'm cool bumping to less than that15:53
pabelangerwould be a good tool to help debug clouds without affecting production15:53
sdaguebecause this only test live thing is just rough if people don't stay on top of job fails for the hours after15:54
sdagueI seem to be meat nagios for that a bunch15:54
*** slashme has left #openstack-infra15:54
rcarrillocruzpabelanger: i'm confused, statsd is metrics but my understanding is mcollective is puppet orchestration ?15:54
pabelangerrcarrillocruz: haha, collectd15:55
pabelangermy mistake15:55
jeblair_please don't run collectd on just *some* of our systems15:55
jeblair_if you want to do that, replace cacti completely, and everywhere, and make sure that you get data that is correct like cacti15:55
pabelangerI thought that was the tooling for choice from the talks at our last midcycle?15:55
jeblair_we don't need two systems graphing memory in parallel15:56
rcarrillocruzpabelanger , jeblair_ : that seems like a good topic for either mid cycle or summit15:56
rcarrillocruzi agree we should stick to one15:56
pabelangerYa, if we can use the same across everything, that makes things easier15:57
*** _nadya_ has joined #openstack-infra15:57
jeblair_rcarrillocruz: i don't think anyone is opposed to it.  i'm just saying it's a lot of hard work, and you have to understand what every graph is measuring to be sure to get the data right.  our cacti install had *a lot* of work put into it to make sure it's correct.15:58
jeblair_rcarrillocruz: i mean, maybe we should talk about it so i can say this to everyone15:58
*** gyee has joined #openstack-infra15:58
ihrachysdepends-on fails for me lately. somehow zuul forgets to pick up a patch for merge after its dependency lands its repo15:58
jeblair_because this is the second time in the past few days i've given this little speech :)15:58
*** roxanaghe has quit IRC15:58
ihrachyshas anyone noticed?15:58
openstackgerritDoug Hellmann proposed openstack-infra/project-config: do not fail if there are no constraint updates to push
sdagueihrachys: which patches?15:58
pabelangerI'm sure its been me twice asking about statsd too15:58
jeblair_pabelanger: in the mean time, we do run cacti everywhere, including infra cloud15:58
ihrachyssdague: just nudged
rcarrillocruzfwiw, i added all compute hosts to cacti15:59
*** kushal has quit IRC15:59
ihrachyssdague: there were others, but I can't immediately come up with links because I lost context.15:59
jeblair_rcarrillocruz: cool, i'll update the trees later15:59
*** javeriak_ has joined #openstack-infra15:59
*** matthewbodkin has quit IRC16:00
sdagueihrachys: python-neutronclient isn't in a shared pipeline with neutron16:00
ihrachyssdague: another one was
sdagueso I think that's expected16:00
sdaguethey have to share a pipeline for them to auto process together16:00
sdagueihrachys: yep, that would be the same thing16:00
sdaguesame reason it happens for project-config changes16:00
dhellmanninfra folks, it would be good to have some quick attention on to cut down the false failures we're seeing in release jobs today16:00
*** javeriak has quit IRC16:00
ihrachyssdague: oh I see. not that it's the best user experience because honestly I don't know by heart pipeline setups :)16:00
pabelangerjeblair_: Yes. Maybe what I should be asking is how to get data out of cacti to be rendered with grafana.16:01
jeblair_ihrachys: if the two projects are not tested together from source, then the depends-on has a slightly different meaning.  it means "this patch can't be enqueued until the other patch lands".16:02
ihrachysjeblair_: yeah, but even when the first lands, the second does not get anywhere near merge queue16:02
jeblair_ihrachys: (in this case, the depends-on can't be used to incorporate a pending change from another repo)16:02
ihrachysjeblair_: so I need to W+0/W+1 it again to get the job done16:02
ihrachysjeblair_: you mean by client patch has not really fetched server side?16:03
jeblair_ihrachys: yes, that's true.  you could always leave the depends-on header off in that case.16:03
jeblair_ihrachys: correct16:03
sdaguejeblair_: would it be hard to look for those an enqueue them on patch merge?16:04
ihrachysjeblair_: gotcha. it sucks and is confusing, but I guess I just had too many hopes in the magic :)16:04
jeblair_ihrachys: well, if it *did* then the pipeline config is wrong16:04
*** claudiub has joined #openstack-infra16:04
sdagueI've definitely seen it trip people up16:04
jeblair_ihrachys: if it actually *did* pull in changes, then maybe it should be in the same shared queue16:04
ihrachysjeblair_: I guess they were passing fine independently, that's why I haven't spotted anything on check queue16:04
jeblair_sdague: i think that would be possible16:05
*** andrewbogott has quit IRC16:05
*** andrewbogott has joined #openstack-infra16:05
sdaguejeblair_: I think that's the thing that people are expecting, then get confused when it doesn't happen.16:05
ihrachysjeblair_: ok, for example neutronclient gates on neutron code via its functional job. wouldn't it mean that they should have had the same pipeline?16:05
*** kien-ha has quit IRC16:06
sdagueihrachys: only if they share a job name16:06
*** jlk` is now known as jlk16:06
*** jlk has quit IRC16:06
*** jlk has joined #openstack-infra16:06
jeblair_i guess in this case, neutron doesn't gate on neutronclient16:06
sdaguejeblair_: correct16:06
sdaguewhich is intentional16:06
openstackgerritZara proposed openstack-infra/infra-manual: Document commit message syntax for StoryBoard
*** nstolyarenko has quit IRC16:07
openstackgerritKhai Do proposed openstack-infra/system-config: upgrade Gerrit's javamelody plugin
zarofungi, clarkb: for tomorrow ^16:08
jesusaurinfra-root: when you get a chance, I'd like your opinions on and the output at
*** mwhahaha has quit IRC16:10
*** shashank_hegde has joined #openstack-infra16:11
*** ijw has joined #openstack-infra16:11
mgagnepabelanger, clarkb: ready for internap-mtl0116:11
sdagueany other project-config core folks want to do a quick review here - ?16:11
*** yamamoto has joined #openstack-infra16:11
openstackgerritMerged openstack-infra/project-config: [Rally] Make py35 job voting
sdakercarrillocruz no need to call me mr :)16:12
*** salv-orlando has joined #openstack-infra16:12
*** jordanP has quit IRC16:12
openstackgerritDavid Lyle proposed openstack-infra/project-config: Add craton-dashboard repository (Horizon Plugin)
fungisdague: in theory we could do something like that with a nodepool-dev/zuul-dev environment. set them up with a limited project list and a check pipeline and only add quota for new providers to nodepool-dev initially16:14
fungi(the new provider incubation corral)16:15
sdaguefungi: that would be kind of nice16:15
fungiit would be a nontrivial thing to create and maintain, but it's probably worth entertaining16:15
sdaguebecause the onboarding of new providers is great for capacity, but tends to make fail spikes16:15
*** yamamoto has quit IRC16:15
sdagueso is rough during crunch time16:16
clarkbI mean16:17
rcarrillocruzfungi: so, i assume we'll get chocolate on newton , when we discuss  at the mid-cycle no?16:17
rcarrillocruzwhen is newton due?16:17
*** annegentle has joined #openstack-infra16:17
clarkbhave ahve fail spikes in long lived clouds all the time too16:17
clarkbI think its more "cluod" than "new cloud"16:17
clarkb(like the osic ipv6 thing wasn't a new cloud issue, just cloud issue)16:18
pabelangerthere is a lot of overhead for nodepool-dev too, image uploads for example16:18
fungircarrillocruz: worth bookmarking
clarkbthat said I think its a reasonable thing to do I just don't want the expectation to be we will never have a cloud issue again16:18
*** kien-ha has joined #openstack-infra16:19
sdagueclarkb: sure16:19
mordredclarkb: to be fair though ... the ipv6 was a cloud config change that we knew about and was _like_ adding a new cloud16:19
mordredor could have been seen that way - although I will admit I did not think of it that way at the time16:19
*** hashar is now known as hasharAway16:19
sdaguebut we had to drop an internap region, and infra cloud in the last 2 weeks16:19
clarkbmordred: I didn't (and I still don't really getting a new IP address is not liek getting a new cloud imo)16:19
fungircarrillocruz: so the infra sprint is the week before final release candidates for newton16:20
sdagueall of which could be better servered with a holding pen16:20
mordredclarkb: totally16:20
sdagueso those could live debug through issues16:20
sdagueand once got to a good pass rate, get added to the good pool16:20
clarkbsdague: yup I agree. I have always tried to run tests myself on the instances before bringing them into the fold not sure if we managed to do that this time. And having a tool to do that would avoid ah uman needing to think about it. So definitely a reasonable thing to do16:20
fungisdague: clarkb: mordred: though i think having a proving ground shadow ci for new providers in some ways simplifies testing them out when we have to determine appropriate flavors and whatnot16:21
fungier, what clarkb just said basically16:21
*** e0ne has quit IRC16:21
sdaguefungi: yep16:21
*** lucasagomes is now known as lucas-dinner16:21
*** senk has quit IRC16:21
sdagueI am well aware we will always have fails :)16:22
fungiwe hadn't seriously considered it in the past because adding a new provider happened once every year or two16:22
*** ramishra_ has quit IRC16:22
funginow it seems to be something closer to a monthly occurrence16:22
rcarrillocruzheh, yeah16:22
fungiwhat an awesome problem to have, btw16:23
pabelangerrcarrillocruz: fungi: Maybe Newton RC1?16:23
*** ramishra has joined #openstack-infra16:23
fungipabelanger: i'm not opposed to running infra-cloud on bleeding-edge prerelease code as long as it doesn't get in the way of being able to use it. having early feedback to the community on new releases is great, but i think our #1 goal should be making sure we're able to run _something_ and use it consistently16:24
*** coolsvap has joined #openstack-infra16:24
rcarrillocruzwfm, although i think that bringing up all servers, iron out issues, get back to DC with tickets etc, it could very well get past one-two weeks, just in time for the newton release16:24
fungiif sticking with mitaka helps increase our chances of keeping it up and running i'm much more in favor of that16:25
*** krtaylor has quit IRC16:25
pabelangerrcarrillocruz: What is stopping us from launching infra-chocolate now?16:25
fungiand then consider upgrading to newton once newton is releasedc16:25
*** sarob has joined #openstack-infra16:25
fungipabelanger: mostly that there's still work to finish on vanilla, aiui16:26
rcarrillocruzi need to double check the inventory of things16:26
*** berendt has joined #openstack-infra16:26
rcarrillocruzwhat took a long time was to find out what those servers were16:26
rcarrillocruzcos in our inventory we had ilO ips16:26
pabelangerrcarrillocruz: fungi: okay, assign me a task! I'm eager to help16:26
rcarrillocruzno serial numbers16:26
rcarrillocruzno nothing16:26
* mtreinish wants ice cream now16:26
rcarrillocruzbut the DC folks they refer them by serial number16:27
rcarrillocruzso i had to go one by one16:27
rcarrillocruzguessing by macs16:27
rcarrillocruzwhat was what16:27
rcarrillocruzregistering the racks the machine were put16:27
*** martinkopec has quit IRC16:27
rcarrillocruzessetnially, cross-checking two different sources of truth16:27
rcarrillocruzbut sure, we can start doing provisioning on chocolate16:27
*** martinkopec has joined #openstack-infra16:27
rcarrillocruzas a amtter of ract, i plan to use those machines to do the live demo i promised on the meeting a couple weeks ago16:28
rcarrillocruzhow to enroll with bifrost16:28
*** martinkopec has quit IRC16:28
fungiyeah, and doing a demo with software you already have deployed previously increases the chances that it's a viable demonstration rather than getting bogged down in whatever isn't quite right for newton yet16:29
pabelangerokay, is there a list of infracloud-vanilla that needs finishing up?16:29
rcarrillocruzpabelanger: none16:29
rcarrillocruzall that can be deployed, are deployed16:29
rcarrillocruzwe have 3 machines with issues16:29
pabelangerrcarrillocruz: everything is online?16:29
rcarrillocruzthat i have tickets for16:29
*** jpich has quit IRC16:29
rcarrillocruzeverything that is ok, is online16:29
rcarrillocruzcheck oepnstack-dev mailing list, i did a summary of the machines16:29
fungiand i guess the new ansible wheel is churning correctly for them, they're in cacti now, et cetera?16:29
rcarrillocruzi'll link16:30
jeblair_fungi: is there a server without a forward dns record?16:30
*** jeblair_ is now known as jeblair16:30
*** sarob has quit IRC16:30
rcarrillocruzfungi: they are all in cacti16:30
fungijeblair_: it's the result of a --keep i'm trying to work through vcsrepo errors for16:31
fungijeblair: i'll be deleting it shortly and relaunching16:31
jeblairfungi: ok.  pabelanger is not the only one to get NDRs -- i just got one for him for that server :)16:31
fungijeblair: cute16:31
rcarrillocruzi was hoping to get ALL fixed today, the lab said 'all is good', but out of 4 with issues just one i brought it back to life today16:31
rcarrillocruzso we're down to 3 now with issues16:32
pabelangerya, looks to be down right now16:32
jeblairpabelanger, rcarrillocruz: i'm happy to help click through the spamhaus records for all the infra-cloud ips, or if we want to set up a smarthost and just do one or two, that should work too.16:32
rcarrillocruzpabelanger: yeah, that one has a HD broken16:32
fungircarrillocruz: pabelanger: do we have any good numbers on relative job runtimes and nondeterministic failures (if any) in our nodepool project in infra-cloud yet?16:32
*** zul has quit IRC16:32
rcarrillocruzfungi: i think it may be too soon, just one day of real workload16:33
rcarrillocruzi really want to see dsvm runs when there are more noisy neighbours in the computes16:33
*** Thelo_ has quit IRC16:33
jeblairfungi: the 2 graphs at the bottom say 'no datapoints'
rcarrillocruzcos in my initial test, a nova tempest full run took the same as osic, but you know, the  VM has the entire compute for itself16:33
fungilooking in grafana myself now, yes16:33
fungilooks like it's all building and deleting?16:34
fungioh, i guess we have some in use16:34
*** berendt has quit IRC16:34
rcarrillocruzjeblair: in regards to the spamhaus thing, yeah, i think having a smarthost would be good16:34
*** weshay is now known as weshay_lunch16:35
pabelangerfungi: jeblair: So, I'd like some feedback on: The current launch failures in infracloud are related to EOFError from paramiko.  Not sure if that is the right approach16:35
*** sambetts is now known as sambetts|afk16:35
pabelangerfungi: jeblair: the other failure was related to DNS, we were just using google DNS, so we added unbound to infracloud this morning16:36
jeblairpabelanger: i'm suspicious of 36462216:37
jeblairpabelanger: does it happen elsewhere, or just omfra?16:37
jeblairpabelanger: and what's *really* going on?  i mean, it opened a connection, but then it was closed?16:38
pabelangerjeblair: Ya, looks like randomly in rax, ovh, internap and bluebox too16:38
openstackgerritEmilien Macchi proposed openstack-infra/project-config: Revert "tripleo-ui: add missing jobs for release management"
fungiwhat's the failure rate from that? frequent enough we could attempt to recreate it with openssh?16:38
fungiand if it's a problem for paramiko, is it also going to happen to ansible?16:39
pabelanger EOFError from today16:39
fungidoes ansible (or maybe zuul-launcher) already have a similar workaround?16:39
mordredfungi: ansible uses openssh, not paramiko16:40
*** Apoorva has joined #openstack-infra16:40
*** yamahata has quit IRC16:40
*** ijw has quit IRC16:40
fungimordred: sure, which is why i asked whether there's any hope of us finding the underlying cause by testing with openssh16:40
pabelangerjeblair: Yes, I am not sure what the issue is right now. It looks to be on the first ssh_connect attempt from nodepool. And because we don't retry, the server is deleted and we move on16:40
mordredfungi: good point16:40
jeblairright, but i think what's being gotten at here is that some of the choices nodepool makes are about preventing bad hosts from making it to zuul16:40
*** cardeois has quit IRC16:40
*** daemontool has quit IRC16:40
jeblairso is this really a situation where we *want* to help more things get through16:41
fungiagreed, right now _assuming_ this eoferror indicates a problem node, then retry-spamming it into service could be detrimental to whatever job runs on it down the line16:41
jeblairthis is why i asked what's really going on16:42
*** woodster_ has joined #openstack-infra16:42
jeblaircause the other errors in there all have explanations for why we should ignore them (user not created yet, ssh not started yet, etc)16:42
pabelangerright, I cannot answer that. I was hoping retry would better expose the actually issue16:42
fungiyep, figuring out what is causing the eoferror paramiko is raising might help us figure out whether it's safe to press nodes exhibiting this behavior into service16:42
fungirather than papering over the failure with retries16:43
pabelanger is the traceback of the failure16:43
rcarrillocruzk folks, gotta run to catch my son16:43
fungiis there a way to get paramiko to provide more detail on the failure mode?16:43
fungiaha, thanks16:43
rcarrillocruz mordred , pabelanger , it passed tests16:43
pabelangerfungi: I believe we could enabled debug logs for that16:44
*** akshai has quit IRC16:45
*** zul has joined #openstack-infra16:45
*** trown is now known as trown|lunch16:45
*** yamamoto has joined #openstack-infra16:46
openstackgerritEmilien Macchi proposed openstack-infra/project-config: tripleo-ui: use nodejs4-publish-to-npm template
jeblairfungi: yeah, though looking at the code, i'm not sure we'd learn much from that16:47
*** jamesdenton has quit IRC16:47
*** akshai has joined #openstack-infra16:47
fungi maybe?16:47
*** ilyashakhat_mobi has joined #openstack-infra16:48
*** asettle has quit IRC16:49
pabelangerjeblair: was there talk of moving away from paramiko in nodepool?16:49
*** timello has quit IRC16:49
jeblairpabelanger: i don't recall?16:49
*** asettle has joined #openstack-infra16:49
*** ilyashakhat_mobi has quit IRC16:49
*** rhallisey has joined #openstack-infra16:50
*** sarob has joined #openstack-infra16:50
*** sputnik13_ has joined #openstack-infra16:50
*** mhickey has quit IRC16:50
*** drifterza has joined #openstack-infra16:50
*** yamamoto has quit IRC16:50
*** timello has joined #openstack-infra16:50
Shrewspabelanger: mordred: rcarrillocruz: fyi, ansible testing WG meeting happening in 9 min. i keep forgetting16:51
pabelangerShrews: Thanks16:51
*** pradk has quit IRC16:51
*** amotoki has quit IRC16:51
*** ilyashakhat_mobi has joined #openstack-infra16:52
jeblairpabelanger: i think if you want to chase this down, that's fine -- i would recommend you create a new log message for it though so you can track it16:52
openstackgerritMerged openstack-infra/storyboard: Limit PyMySQL to < 0.7.7
fungiworth noting, we're on paramiko 1.17.2 for nodepool.o.o16:53
pabelangerjeblair: okay, I'll update the patch shortly16:53
jeblairpabelanger: and log the exception, so we know where it's coming from.  that way if we see multiple eoferrors from the same host, we'll know if they're all from the same spot16:53
pabelangerjeblair: will do16:53
*** asettle has quit IRC16:54
fungiugh. their repo has tags for 1.17.2 and v1.17.216:54
*** sarob has quit IRC16:54
fungi(that file looks to be the same under the v1.17.2 tag as well)16:56
*** derekh has quit IRC16:56
*** drifterza has quit IRC16:56
*** Thelo_ has joined #openstack-infra16:58
*** markvoelker has quit IRC16:58
pabelangerfungi: ya, looks like we don't even get to the authentication step16:58
*** cardeois has joined #openstack-infra16:59
*** esikache1 has quit IRC16:59
pabelangerdoes ssh-server restart after keys are generated?16:59
*** shashank_hegde has quit IRC16:59
fungipabelanger: so what's interesting about the failures... none seem to be in osic16:59
fungieven though osic is now by far the bulk of our volume17:00
clarkbpabelanger: no it doesn't start at all until keys are generated17:00
*** tesseract- has quit IRC17:01
fungino rax-iad either, but i could chalk that up to lack of a statistically significant sample17:01
fungihowever, if this were consistent across providers, i would expect to see lots in osic too17:01
*** yamahata has joined #openstack-infra17:01
pabelangerrax-iad is currently disabled however17:01
fungihah, that explains that one then17:01
*** kgiusti has joined #openstack-infra17:01
pabelangerYa, but no osic in any logs on disk17:01
pabelangerwould should include ipv417:02
pabelangeroh, maybe not17:02
fungiright, i think we're only connecting from nodepool to osic nodes via ipv6 now17:03
pabelanger2016-08-22 is last log, I think we had ipv6 by then17:03
*** kdas_ has joined #openstack-infra17:03
fungiso the question is whether this is a v4-only issue, or an issue than manifests dirrerently (and raises a different error condition) under v617:03
fungior an issue that doesn't affect osic for some other reasons unrelated to ipv4 vs ipv617:04
*** jerryz has joined #openstack-infra17:04
*** abregman has joined #openstack-infra17:04
*** abregman|mtg has quit IRC17:06
*** edtubill has quit IRC17:06
*** mwhahaha has joined #openstack-infra17:07
*** tqtran has joined #openstack-infra17:07
*** kdas_ is now known as kushal17:08
*** kushal has quit IRC17:08
*** kushal has joined #openstack-infra17:08
*** nstolyarenko has joined #openstack-infra17:08
*** ilyashakhat_mobi has quit IRC17:08
*** krtaylor has joined #openstack-infra17:08
*** weshay_lunch is now known as weshay17:08
*** tqtran has quit IRC17:12
*** ilyashakhat_mobi has joined #openstack-infra17:12
*** tonytan4ever has quit IRC17:16
mgagnepabelanger: let me know when you are ready to enable more in internap-mtl0117:16
*** HeOS has quit IRC17:17
pabelangermgagne: I think we are trying to determine how the cloud is preforming right now.  Average test runs, if anything is failing, etc17:18
mgagnealright, fine with me17:18
clarkbpabelanger: are any of thosefailures on precise or trusty? if not maybe its related to systemd socket activation and the service not starting fast enough for paramiko?17:18
*** rossella_s has quit IRC17:18
mgagneis there any dashboard/link I can read?17:18
clarkbpabelanger: basically with socket activation systemd is going to listen on port22 and accept connections for ssh before sshd is ready. Then when sshd is ready it will hand over control of the socket17:19
pabelangerclarkb: let me extra the node type17:19
*** rossella_s has joined #openstack-infra17:19
clarkbso wondering if maybe tcp handshake happens then we have a long pause long enough to make ssh unhappy17:19
pabelangermgagne: mostly looking in logstash.o.o and nodepool logs atm17:20
*** ilyashakhat_mobi has quit IRC17:21
*** nstolyarenko has quit IRC17:22
*** pradk has joined #openstack-infra17:22
clarkbhrm glance wants to remove -2 perms from their core group?17:22
fungiwhere did we get to with making nodepool logs public? i think we said we'd be comfortable with it once we finished the migration to shade?17:23
fungiand that's done now, afaik17:23
pabelangerclarkb: seems ubuntu-xenial and debian-jessie:
pabelangerso you are on to something17:24
clarkbI think we are still maybe waiting for swift bits to use ksa? I don't recall if that was required for the password sanitizing17:24
clarkbthough now all the swift stuff happens in the nodepool builder we could just not serve those logs17:24
clarkbfungi: ^17:24
fungiahh, right17:24
fungiespecially easy if we move the builder daemon to a separate server17:25
*** niska has quit IRC17:26
pabelangerclarkb: ya, so that goes back to my question about ssh server being restart.  It sounds like what you are describing with systemd could cause issues with the socket17:27
pabelangerif we connect early enough17:27
*** ddieterly is now known as ddieterly[away]17:27
pabelangerSo, once is cleaned up, we should see a failure or 2, and then eventually ssh connection17:28
nikhilheya.. there're a few glance patches in gate that seem stuck verifying?
nikhilbased on the status on zuul17:29
clarkbIn theory it should be fine because systemd just holds the fd but maybe there is a timeout or something more aggressive in paramiko17:29
openstackgerritJames E. Blair proposed openstack-infra/puppet-nodepool: Enable mod_proxy when proxying status commands
nikhilall of them are really really important for us to tag newton-3 today17:29
*** senk has joined #openstack-infra17:29
pabelangeranother option, could be to delay our ssh connections per cloud, with some sort of configuration option.17:29
jeblairclarkb, pabelanger, fungi: ^ i just did that manually on the nodepool server17:29
nikhilany help/pointer would be super useful!17:29
clarkbnikhil: have you pulled up teh console logs/17:29
jeblairclarkb, pabelanger, fungi, mgagne: works now17:29
jeblairas does
clarkbnikhil: says its still running tempest17:30
mgagnethat's one interesting output17:30
*** ramishra has quit IRC17:30
pabelangerjeblair: excellent17:30
AJaegerfnikil 363838 is still running...17:30
*** dteselkin has quit IRC17:30
jeblairsdague: you may find useful17:30
*** igormarnat has quit IRC17:30
AJaegernikhil: sorry for typo ^17:30
nikhilAJaeger: clarkb : that's been running for along time17:30
nikhiland the other two seem done17:30
AJaegernikhil: gate-tempest-dsvm-neutron-full-ubuntu-xenial is running - did you see that?17:30
*** rakhmerov has quit IRC17:30
*** ramishra has joined #openstack-infra17:30
fungi2016-09-01 17:30:47.614538 | {1} tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_stop_start [165.255744s] ... ok17:30
nikhilAJaeger: yeah, waiting for it for last 1-1.5 hrs17:31
*** kzaitsev_ws has quit IRC17:31
*** kzaitsev_ws has joined #openstack-infra17:31
fungithe node seems to be in rax-ord so we can't blame slowness in infra-cloud or anything for this one17:31
*** tsufiev has quit IRC17:31
*** katyafervent_awa has quit IRC17:31
*** penguinolog has quit IRC17:32
nikhilwhat's the best path forward? (recheck won't work) so, wait fora  bit more and bug folks then :)17:32
*** javeriak_ has quit IRC17:32
*** e0ne has joined #openstack-infra17:32
*** tsufiev has joined #openstack-infra17:32
*** akshai has quit IRC17:33
clarkbnikhil: you'll have to debug why the job is slow17:33
clarkbit is still doing stuff though seems like17:33
nikhilclarkb: how do I get on this telnet link telnet:// ?17:33
fungilooks like devstack setup took from 15:17:41 to 15:56:3817:33
clarkbnikhil: use telnet or nc to that ip address and port17:33
*** katyafervent_awa has joined #openstack-infra17:33
nikhil(that's what I get when I hover on that gate link)17:33
*** tphummel has joined #openstack-infra17:34
clarkbnikhil: if you are more adventurous there are ways to have your browser do things automatically but I haven't bothered17:34
*** igormarnat has joined #openstack-infra17:34
*** cardeois_ has joined #openstack-infra17:34
*** rakhmerov has joined #openstack-infra17:35
fungilooks like it's just slow. for example the gap from when test_server_connectivity_rebuild reported and test_server_connectivity_resize reported was on the order of 3 minutes17:35
AJaegernikhil: telnet 19885 (or use nc with same arguments)17:35
nikhilclarkb: gotcha, will use telnet17:35
*** ijw_ has joined #openstack-infra17:35
nikhilAJaeger: ty , just hopped on :)17:35
*** kaisers_ has joined #openstack-infra17:35
fungii don't see any tests that have actually failed in that log (yet)17:36
*** javeriak has joined #openstack-infra17:36
fungibut it's entirely possible this job will continue until it reaches the job timeout and gets killed17:36
*** dteselkin has joined #openstack-infra17:36
AJaegersdague: for we should ask the Neutron folks, shouldn't we? armax, dougwig , please review17:36
* armax looks17:36
nikhilfungi: yeah, it's a change in config &  tests are not expected to fail17:37
nikhilfungi: ok17:37
clarkbI think those tests are the ones that tend to run at the end17:37
clarkbbut I haven't actually looked at the sorting recently17:37
AJaegerZara: did you merge anything yet on python-storyboardclient? Any dummy commit to get content published?17:37
openstackgerritZara proposed openstack-infra/python-storyboardclient: Add due_dates
*** cardeois has quit IRC17:38
*** abregman is now known as abregman|mtg17:38
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Add ipv6 nic-configs
AJaegerdo we want to merge the internap-mtl01 to 150 increase now? ?17:38
ZaraAjaeger: it's not merged yet but there's a patch over here: for docs17:38
fungithe same job that ran on the two glance changes behind the slow one only took around an hour to complete, so there's probably something terribly wrong with the node this job is running on17:38
clarkbjeblair: I went ahead and just approved the mod proxy change since its already been applied by hand17:38
jeblairclarkb: thx17:39
fungisystem load on that node is around 417:39
*** javeriak_ has joined #openstack-infra17:40
*** dkehn has quit IRC17:40
openstackgerritPaul Belanger proposed openstack-infra/nodepool: Retry ssh connect when EOFError is received
*** dkehn_ has quit IRC17:40
nikhilfungi: saw it succeed in 339 secs!17:40
*** kaisers_ has quit IRC17:40
funginot an extreme amount of memory pressure17:41
pabelangerAJaeger: still confirming if the cloud is ready for more nodes. Should know more in the next little bit17:41
clarkbfungi: how many bogomips >_>17:41
*** javeriak has quit IRC17:41
AJaegerpabelanger: Mathieu Gagné commented "We're ready" on the review17:42
fungiclarkb: around 5200 on all 8 processors listed in cpuinfo17:42
clarkbpabelanger: is the current concern the ssh eof error?17:42
mat128mgagne: ^17:42
*** dkehn has joined #openstack-infra17:42
*** rbrndt has quit IRC17:42
clarkbfungi: thats more than my local machine!17:42
mgagneAJaeger: pabelanger said " I think we are trying to determine how the cloud is preforming right now.  Average test runs, if anything is failing, etc"17:42
pabelangerclarkb: for internap-mtl01? Just wanted to confirm job times are in line with what we expect, I haven't actually checked that yet.17:43
fungiclarkb: i think the eoferror seems to be consistent across providers (sans osic) so probably not a concern for ramping up17:43
clarkbpabelanger: gotcha17:43
clarkbpabelanger: oh I remember things re the EOF error17:43
clarkbpabelanger: launch node ran into that too when running the restart command17:43
AJaegermgagne: ah, thanks!17:43
clarkbpabelanger: but in that case its killing the services really fast. So maybe there is a restart of the service happenign that I didn't expect that is closing very earlyconnections17:44
AJaegerproject-config cores, could you review for the release team, please?17:44
funginikhil: well, it finally finished and the two behind it merged so looks like we're in the clear for now but should keep an eye out for more of the same17:44
clarkbpabelanger: you can actually reproduce that pretty easily by sshing into a host with systemd and running `reboot`17:44
nikhilfungi: thanks much!17:44
*** nstolyarenko has joined #openstack-infra17:44
pabelangerclarkb: sdague: AJaeger: Looking at grafana for internap-mtl01, tempest tests are inline with nyj01. So, if everbody is on board, we can up the capacity I think17:44
mordrednikhil: o hai17:44
nikhilfungi: clarkb AJaeger : I noticed they merged after we'd a chat here. you guys have magic vision to make things work just by looking at'em!!17:45
fungizuul's going to be busy for a while processing the dozen or so merges that piled up behind that glance change17:45
nikhilmordred: \o17:45
*** jtomasek is now known as jtomasek|afk17:45
fungiqueue lengths are already starting to rise (events is up to 1k)17:46
AJaegernikhil: we're all part of the magic team ;)17:46
mordrednikhil: rcarrillocruz and I were talking about image import yesterday ... and I just wanted to confirm17:46
clarkbpabelanger: so yes, I am suspecting some systemd behavior we may be tickling there17:46
mordrednikhil: import_from_url is not a thing in v2? or it's a thing but only with tasks?17:46
fungiclarkb: was there a systemd vs non-systemd split on the ssh failures then?17:46
*** SumitNaiksatam has joined #openstack-infra17:47
mordrednikhil: (my understanding is that I should not consider import_from to be a thing anymore, yeah?)17:47
clarkbfungi: pabelanger said its just jessie and xenial17:47
mrhillsmanmordred fungi pabelanger clarkb for osic cloud8 the best solution is the dns suggestion from yesterday17:47
fungiclarkb: in that case, yes, sounds highly likely17:47
pabelangerclarkb: I think we are restart SSH for some reason17:47
pabelangergetting a logs now17:47
mrhillsmanattaching public address to the VM directly looks like it is going to require quite a bit of work17:48
clarkbpabelanger: though in theory restarting sshd doesn't kill existing connections... at least it didn't with upstart maybe this is new and exciting bugs17:48
pabelangerclarkb: that is from an random osic ubuntu-xenail server17:48
mrhillsmani have to discuss with network folks since cloud8 is setup differently than cloud117:48
fungimrhillsman: it's an okay short-term solution, but long term it means that if we lose cloud1 for some reason then cloud8 will also effectively be dead to us17:48
mrhillsmanmakes total sense17:49
nikhilmordred: import from url won't be a generic one. things will be rather predefined
AJaegerarmax: is fine?17:49
pabelangerclarkb: I think it maybe glean that kicks off the stop / start under systemd17:50
clarkbpabelanger: aha it does a stop start after the reload17:50
*** cardeois_ is now known as cardeois17:50
clarkbpabelanger: ya I Think that would explain ti then as I have seen the same behavior with launch node on `reboot`17:50
*** psachin has quit IRC17:50
nikhilmordred: as per that value discovery call, and then the info about the what to provide is here
nikhilmordred: which says, give me the container name if swift local or give me a stream of data is glance-direct17:50
pabelangerSo, maybe we should see how to make glean run before networking is started17:50
clarkbpabelanger: we may be able to just edit glean to say before sshd17:51
mordrednikhil: awesome. thanks. super helpful17:51
nikhilmordred: so, once import refactor merges, that will be the case :)17:51
pabelangerclarkb: right17:51
*** ijw_ has quit IRC17:51
pabelangerclarkb: let me get a new server up and play with it17:51
clarkbpabelanger: kk17:51
*** e0ne has quit IRC17:53
*** dkehn_ has joined #openstack-infra17:54
*** esikache1 has joined #openstack-infra17:55
jeblairsdague, pabelanger: re i share clarkb and mordred's suspicion that there might be something more subtle at play.  ansible uses a persistent ssh connection, so the time/effort required to open a new channel should be greatly diminished.  10 seconds seems more than ample.  it's also the case that we saw this sort of thing with jenkins too, which had a different approach to connectivity.  having said ...17:55
jeblair... that, i don't have any other suggestions for how to make progress on this.  i do think if we decide to do it, we need to check on it after a few days in production and evaluate if it's made a difference.17:55
*** shashank_hegde has joined #openstack-infra17:55
jeblair(also, it's worth noting that zuul automatically restarts jobs that hit that problem)17:55
openstackgerritMerged openstack-infra/devstack-gate: remove old tests
pabelangerI still want to make ansible async ignore ssh failures and keep trying until the timeout is reached. Or some ignore ssh failure limit17:57
dhellmannhey, folks, I think I have some logic wrong in the script that tries to propose upper constraint changes when we release libraries.17:57
dhellmannit gets the branching for the requirements repo wrong17:57
jeblairpabelanger: yeah, i think that would be a nice improvement17:57
*** nstolyarenko has quit IRC17:57
dhellmannin this log, it should be trying to check out the stable/mitaka branch to propose the commit there, but it doesn't find origin/stable/mitaka:
*** harlowja_ has joined #openstack-infra17:58
*** niska has joined #openstack-infra17:58
AJaegerthanks, armax !17:59
openstackgerritMerged openstack-infra/puppet-nodepool: Enable mod_proxy when proxying status commands
openstackgerritMerged openstack-infra/storyboard: Don't allow users to subscribe to private worklists they can't see
pabelangerjeblair: I also think, we maybe be too aggressive with 5s polling for zuul_runner. Maybe we should bump that to every 60 seconds or a percentage (5%) of the timeout value18:00
*** harlowja has quit IRC18:00
openstackgerritZara proposed openstack-infra/python-storyboardclient: Add basic docs for python client
mordredpabelanger: problem with 60s poll18:00
clarkbdhellmann: I think that may be a subtle git behavior where it can't identify a unique thing called that because there may be a file or other item with the same name?18:00
*** ihrachys has quit IRC18:00
mordredpabelanger: is that then there is a 60s lag between a job finishing and ansible knowing that18:00
dhellmannclarkb : maybe? I'm pretty sure there is no file called origin/stable/mitaka though?18:01
mordredpabelanger, jeblair: in the ansible zuul work I want to do next week - which involves forward porting 2.5 to 3 - I also want to investigate our own action plugin18:01
clarkbdhellmann: right above where you linked you can see where it says stable/mitaka -> origin/stable/mitaka and its up to date18:01
*** e0ne has joined #openstack-infra18:01
mordredand that action plugin should be able to be MUCH smarter about how async happens18:01
mordredbecause we know our intent18:01
* clarkb tries to reproduce locally18:01
dhellmannclarkb : I've run into similar issues where the local branch name is not what I expect because of something about how we clone repos that I don't understand.18:01
jeblairmordred: ah, neat.18:01
mordredwhereas the async module from ansible has to be more generic, which means it can't respond18:01
dhellmannclarkb : maybe I should change the script to just look for stable/mitaka and not origin/stable/mitaka? it used to be that the shorter name wouldn't exist locally, though18:02
Shrewsmordred: that's exciting18:02
jeblairpabelanger: well, some zuul_runner things are very fast; i'd hate to have to wait 60 seconds for an 'echo' statement18:02
clarkbdhellmann: it works locally :/18:02
dhellmannclarkb : yeah18:02
pabelangermordred: exciting18:02
pabelangerjeblair: ya, down side18:02
fungidhellmann: the non-remote branch name will generally only exist locally in the clone if the job has requested checking out that branch by name18:03
sdaguejeblair: zuul did not restart this job18:03
pabelangerif only we could have linear increasing polling18:03
jeblairsdague: i did not see an error report for it18:03
dhellmannfungi : ok. I'm looking for the full remote name and not finding it as origin/stable/mitaka. would it have a different name for some reason?18:03
fungidhellmann: by default, a normal clone only gets the default branch indicated in the remote HEAD (or an alternate branch if you specified one explicitly)18:04
mordredoh. crap. I was going to fix a bug for jeblair today18:04
*** _nadya_ has quit IRC18:04
sdaguejeblair: we force repromoted it because it was the critical patch to fix multinode18:04
mordredjeblair: do you remember what the bug was that I was going to track down today?18:04
fungidhellmann: you also won't have an origin/stable/mitaka until after a remote update18:04
*** degorenko is now known as _degorenko|afk18:04
fungipretty sure18:04
fungitesting now18:04
sdagueit had failed that job, reset the gate, taken everything off of it18:04
sdaguebut we needed that patch18:04
dhellmannfungi : this is from the script doing its own call to git clone to check out the requirements repository18:04
dhellmannfungi : and then it does "git fetch -v --tags" and the output from that includes a bunch of branch names, including stable/mitaka and origin/stable/mitaka18:05
AJaegerZara: you're too fast for me - I explained how to use references...18:05
openstackgerritEmilien Macchi proposed openstack-infra/project-config: tripleo-ui: use nodejs4-publish-to-npm template
jeblairsdague: this one?18:05
dhellmannunfortunately that's all in a temporary directory so it's no longer there to examine18:05
fungidhellmann: yeah, just confirmed, if i `git clone ...` the keystone repo, and then cd into it, `git branch -v` only lists "master"18:05
sdaguejeblair: yes18:05
*** claudiub has quit IRC18:05
clarkbfungi: ya but the job does git fetch -v --tags first which seems to populate the things18:06
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci: Add IPv6 network configuration for ipv6 job types
*** trown|lunch is now known as trown18:06
clarkbfungi: at least when I do the same for requriments taht git show works locally18:06
dhellmannfungi , clarkb : I can add a "git remote update" if you think that would help, but I thought the fetch was more or less doing that?18:06
fungiclarkb: well, except git branch -v still only lists my local master after i do that18:06
fungihrm, maybe i'm using git branch incorrectly18:07
clarkbfungi: see git branch -a18:07
fungii wanted git branch -a18:07
ZaraAJaeger: ahaha, thanks. I'll fix it. :)18:07
*** kzaitsev_ws has quit IRC18:07
fungiso even immediately after a git clone, git branch -a actually has all the remote branches for the remote i cloned from18:07
jeblairpabelanger, mordred, sdague: hrm.  ansible did exit with exit code 1, not 3.18:08
*** salv-orl_ has joined #openstack-infra18:08
fungibut no local branches except the one referenced by HEAD18:08
dhellmannclarkb , fungi :
AJaegerZara: and sorry for wrong guideance on ``code``18:08
pabelangerjeblair: yes, because failed=118:08
*** sshnaidm is now known as sshnaidm|afk18:08
clarkbdhellmann: I think I know what the issue is18:08
jeblairpabelanger: why did that end up as a failure?18:09
clarkbdhellmann: its a git repo in a git repo18:09
*** Na3iL has quit IRC18:09
*** tqtran has joined #openstack-infra18:09
clarkbdhellmann: can you confirm that is how the job is setting the tree up? It seems that way from the log18:09
jeblairpabelanger: any chance it's because of the block/rescue thing?18:09
dhellmannclarkb : oh! could be18:09
dhellmannit does do that, yes18:09
fungiahh, right, git has some funky behaviors around git inside git18:10
clarkbdhellmann: so I think the ambiguousness is git not knowing which repo's mitaka to show?18:10
dhellmannthat could well be18:10
dhellmannit should be using our fancy tmp dir stuff, let me see why that isn't18:10
dhellmannclarkb : it does seem to be using a tmpdir:
dhellmannoh, except that temporary directory is inside the workspace18:11
pabelangerjeblair: I don't fully understand why. When I last tried to debug this, I considered our usage of the failed task the issue, but it might also be possible that something is async task is doing it18:11
dhellmannwhich is a git repo18:11
*** maishsk has joined #openstack-infra18:11
*** salv-orlando has quit IRC18:11
clarkbdhellmann: ya18:11
dhellmannok, let me see if I can fix that18:11
pabelangerjeblair: but when we get unreachable=1, we still called the fail task, which leads me to think using fail is not the problem18:11
* clarkb does a quick test18:12
AJaegerpython experts, could you lend me a hand, please? See - and my comment from 5:30 this morning. Why is pbr freeze showing openstackdocstheme==1.5.0 instead of something like openstackdocstheme==1.5.1.dev2 ?18:12
jeblairpabelanger: i'm looking for the cases where we actually get exit=318:12
*** waht has joined #openstack-infra18:12
fungidhellmann: separate (but related) note... we did eventually get /usr/zuul-env/bin/zuul-cloner onto the signing node so that script can likely be simplified again when you're ready to hack on that18:13
jeblairpabelanger: and it looks like they are ones where we get an ssh error in zuul_runner, but then we *also* get an ssh error in the zuul_log in the rescue block....18:13
dhellmannfungi : ok, cool, I'll put that on the ocata list18:13
clarkbhrm at least my local git knows how to handle that (I am running really new git though)18:13
AJaegerfungi, dhellmann : But we don't have a local cache, correct?18:13
fungiAJaeger: correct18:13
jeblairpabelanger: that makes me more suspicious that the block/rescue thing is converting connection errors from inside the block into failures, *unless* there is also a connection failure in the rescue.18:14
jeblairmordred, Shrews: ^18:14
*** senk has quit IRC18:14
openstackgerritMerged openstack-infra/project-config: Add Ironic UI gerritbot to #openstack-ironic
fungiAJaeger: it would help if we had tox logs collected from that job18:15
pabelangerseems to imply a problem with ssh client, 25518:15
*** Jeffrey4l has quit IRC18:15
AJaegerfungi, we have - see the recheck result18:16
*** ijw has joined #openstack-infra18:16
fungiAJaeger: i don't immediately see any indication of you using edit-constraints in that job18:16
clarkbAJaeger: I remember reviewing that change I Think :)18:16
fungiAJaeger: so you're probably running into forcing you to the constrained version?18:16
jeblairpabelanger, Shrews: i'd like to back-burner the ssh failures for a moment and focus on the exit code.18:16
*** senk has joined #openstack-infra18:16
*** tonytan4ever has joined #openstack-infra18:16
*** awayne has quit IRC18:17
AJaegerfungi, but I'm not using constraints in that tox.ini18:17
*** tonytan4ever has quit IRC18:17
jeblairpabelanger, mordred, Shrews: i'm going to see if i can synthetically create an ssh error, and then put it in a block/rescue and see if it gets converted into a failure18:17
AJaegerfungi, I agree, if that repo would use constraints, then we would need edit-constraints...18:18
openstackgerritDoug Hellmann proposed openstack-infra/project-config: put release temporary directories under $TMPDIR
openstackgerritZara proposed openstack-infra/python-storyboardclient: Add basic docs for python client
*** openstackgerrit has quit IRC18:18
fungiAJaeger: yeah, it's not constraints... i see it18:18
mrhillsmanfungi mordred clarkb pabelanger - spoke with network folks and it is possible but will take some time18:18
AJaegerfungi, clarkb , I run tox -e api-ref locally and got "openstackdocstheme==1.5.1.dev2  # git sha 670fbd8" in the freeze...18:18
dhellmannclarkb : I need to test ^^ locally but I don't have a lib that's going to trigger a stable branch requirements update18:18
*** openstackgerrit has joined #openstack-infra18:18
fungiAJaeger: Collecting openstackdocstheme>=1.4.0 (from os-api-ref>=1.0.0->-r /home/jenkins/workspace/gate-openstackdocstheme-api-ref/test-requirements.txt (line 12))18:18
mrhillsmanroll with short-term and i will respond when long-term has been implemented18:19
clarkbdhellmann: you should be able to just invent one and remove it from your repo when done18:19
fungidhellmann: i think you uploaded right when openstackgerrit was restarting for a config update18:19
*** nstolyarenko has joined #openstack-infra18:19
*** senk has quit IRC18:19
clarkbdhellmann: maybe use a temp repo so you don't risk mixing it up with the real world18:19
fungidhellmann: oh, you mean 36448918:19
fungii didn't scroll back far enough, sorry18:20
*** pvaneck has joined #openstack-infra18:20
*** maishsk has quit IRC18:20
AJaegerfungi, so os-api-ref forces the downgrade? ;(18:20
mordredmrhillsman: awesome! thnaks!18:20
AJaegerfungi, how can I avoid that?18:20
dhellmannclarkb , fungi : I tested by re-releasing muranoclient and it did not work. I'll keep tweaking locally18:21
*** maishsk has joined #openstack-infra18:21
jeblairmordred: it was shade caching images and flavors18:21
fungiAJaeger: when test-requirements gets installed by tox, os-api-ref>=1.0.0 gets installed depending on openstackdocstheme>=1.4.0 which triggers a download from pypi18:21
fungiAJaeger: looks like pip install -U may be at fault?18:21
fungii honestly can't remember what our position is now on whether install_command should be pip install with -U or without18:22
* AJaeger removes -U and will test that18:23
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: pingtest: run 'openstack stack failures list' when failure
AJaegerthanks, fungi18:23
dhellmannclarkb, fungi : I found it. I was being bone-headed.18:23
*** piet has joined #openstack-infra18:24
fungiAJaeger: if we want to consider our current cookiecutter templates to be our official advice, then does not add -U18:24
fungiAJaeger: and you're right to be confused, this is a lot of voodoo18:25
AJaegera grep over openstack namespace shows 400 repos with -U and 300 without ;(18:25
AJaegerthanks, fungi18:25
AJaegerI pushed and will check it tomorrow.18:25
clarkbthe erason we had/have -U is pre constraints you needed it to get the requirements updated if things otherwise fit into the reqs ranges18:25
clarkbwith constraints you will install the constraints version regardless every single time18:25
clarkbThe small gap without constraints and without -U is setuptools I think. But updating it during the main install run doesn't actually use the new version of setuptools so thats mostly a noop18:26
AJaegerah. Perhaps I should use constraints instead ;)18:26
*** mtanino__ has joined #openstack-infra18:26
fungiwell, also we had it to deal with the bad old days where lots of projects were using system site-packages in their tox virtualenvs and we had nodes with a bunch of crufty old distro-packaged python libs installed on them18:26
AJaegerthanks, clarkb18:26
openstackgerritMonty Taylor proposed openstack/os-client-config: Split auth plugin loading into its own method
openstackgerritMonty Taylor proposed openstack/os-client-config: Remove validate_auth_ksc
fungiand so without -U we'd end up using ancient versions of dependencies from the distro packages18:27
*** mtanino has quit IRC18:27
*** abregman|mtg is now known as abregman18:28
* AJaeger congratulates Zara on absolving the RST master class ;)18:28
mordredjeblair: thank you!18:28
dhellmannfungi : since the issue is in my cloning function, I'm going to switch it to use zuul-cloner18:29
AJaegerfungi, it did not help, see
AJaeger - is there a version screwup?18:30
fungiAJaeger: yeah, i see it definitely still pulling in an openstackdocstheme 1.5.0 wheel18:31
*** mtanino has joined #openstack-infra18:31
*** harlowja_ has quit IRC18:31
*** harlowja has joined #openstack-infra18:32
*** _nadya_ has joined #openstack-infra18:32
*** mtanino__ has quit IRC18:33
AJaegeryep ;/18:33
openstackgerritDoug Hellmann proposed openstack-infra/project-config: fix branch handling in clone_repo
*** nstolyarenko has quit IRC18:34
*** ddieterly[away] is now known as ddieterly18:34
dhellmannfungi, clarkb : ok, that should do it ^^18:34
fungiAJaeger: weird, it does look like it later installs the git checkout into the venv claiming to be 1.5.1.dev2 but then immediately after that reports "Successfully installed openstackdocstheme-1.5.0"18:34
*** _nadya_ has quit IRC18:34
*** ijw has quit IRC18:35
ZaraAJaeger: =D thanks!18:35
*** Thelo_ has quit IRC18:35
* AJaeger tries now constraints...18:35
fungiAJaeger: to use constraints, you'll almost certainly need to add an edit-constraints call since this is a repo listed in upper-constraints.txt18:36
AJaegerfungi,done already -
*** akshai has joined #openstack-infra18:37
*** nstolyarenko has joined #openstack-infra18:37
fungiAJaeger: i wonder if you just need tox.skipdist=True like at
fungiiirc there are some odd interactions between skipdist and usedevelop18:38
*** annegentle has quit IRC18:39
fungimordred or dhellmann probably remember more clearly18:39
*** shardy is now known as shardy_afk18:40
dhellmannI think we usually set both of those, but I think just because building the dist is a waste of time if you're not going to install from it18:40
mordredI do not - but I do know that the general intent is to set both18:41
AJaegerok, I'll try - thanks.18:41
mordredsetting both is good for speed when running tox. it's bad for verifying that sdist works, but the tradeoff in the past seemed ot be that devs screaming about tox taking too long trumped the things we might learn from building the sdist every time18:41
clarkband then peopl started using symlinks18:42
fungisomething definitely seems to be causing tox to force it to the (cached?) wheel rather than the git checkout18:42
*** salv-orl_ has quit IRC18:42
*** nstolyarenko has quit IRC18:43
fungimordred: the very end of the tox log here is especially confusing...
*** annegentle has joined #openstack-infra18:43
mordredfungi: WOW18:44
jeblairmordred, pabelanger, Shrews: i have confirmed with synthetic testing that it's neither the block/rescue, nor zuul_runner that's causing the ssh errors to be failures (exit code 1) rather than unreachable errors (exit 3).  it seems to be a behavior of the async module -- if it can't connect at the start, it's 'unreachable'.  if it can't connect for one of it's poll checks, it's a failure.18:44
*** kzaitsev_ws has joined #openstack-infra18:45
fungimordred: if you back up to api-ref-1.log.txt you'll see it previously pulled in a wheel of 1.5.0 because there's a circular (test) dependency of openstackdocstheme on itself (via os-api-ref)18:45
fungiso there will be a 1.5.0 wheel in the cache at that point18:46
mordredfungi: but why would that matter :(18:46
fungimordred: i don't think it should18:46
*** Thelo_ has joined #openstack-infra18:46
fungii'm just stretching for odd corner cases that might be exposing a bug we don't normally see18:46
mordredfungi: oh - totally - sorry, it was a rhetorical head-against-desk question18:47
fungiyeah, no idea whether that has anything to do with the problem, but obviously pip shouldn't be resolving setup_requires for 1.5.1.dev2 and then end by claiming to have installed 1.5.018:48
*** annegent_ has joined #openstack-infra18:49
*** david-lyle has quit IRC18:49
AJaegerstill wrong ;(
fungier, install_requires i guess18:49
*** ijw has joined #openstack-infra18:49
*** hasharAway is now known as hashar18:49
fungiAJaeger: was that after switching to constraints, or adding skipdist=true?18:50
AJaegerOh, I might have a typo...18:50
AJaegerskipdist or skip*s*dist?18:50
* AJaeger has two changes ;)18:50
fungimordred: no constraints at play in the log i linked for you18:50
*** david-lyle has joined #openstack-infra18:50
mordredif you do "pip install -c upper-constraints.txt -e . "18:50
openstackgerritMonty Taylor proposed openstack-infra/shade: Batch calls to list_floating_ips
jeblairpabelanger, sdague: i believe we might be able to see a little more of what ansible is doing with the ssh connections and perhaps ascertain the likelihood of a connection timeout increase being effective if we run with verbose logs for a bit.  i will enable that on one of the launchers and see if we can catch an error before we run out of disk.18:51
AJaegerfungi, skipsdist is it, isn't it?18:51
openstackgerritMonty Taylor proposed openstack-infra/nodepool: Unsubvert image and flavor caching
AJaegerthat's what cookiecutter uses18:51
fungimordred: you can see at the top of the log it only ran `pip install -e .`18:51
mordredfungi: yah. that's excessively weird18:51
AJaegerI'm talking about
mordredfungi: I kind of want to invoke our friendly pip human18:52
*** annegentle has quit IRC18:52
fungiAJaeger: sorry, yes, seems it's "skipsdist = True" and i never noticed18:52
AJaegerno worries, fungi18:52
jeblairpabelanger: actually, i'm going to do it on all the launchers to try to catch this faster18:53
*** waht has quit IRC18:54
*** ddieterly has quit IRC18:55
*** abregman has quit IRC18:56
*** mriedem has quit IRC18:56
*** asettle has joined #openstack-infra18:56
pabelangerjeblair: okay18:56
*** akshai has quit IRC18:57
AJaegerusing constraints:
AJaeger"openstackdocstheme==1.5.1.dev3  # git sha f3782e1"18:58
AJaegerthat looks finally ok...18:58
fungiAJaeger: i wonder if we have some sort of strange versioning going on. checkout the tip of master and run `tox -re venv pip list`18:58
AJaegerwill do18:58
*** kien-ha has quit IRC18:59
fungigit describe says i've got a commit newer than 1.5.0 but pip list is saying "openstackdocstheme (1.5.0, /home/fungi/work/openstack/openstack/openstackdocstheme/.tox/venv/lib/python2.7/site-packages)"18:59
fungiAJaeger: was that with constraints or no?18:59
AJaegerthat was master branch19:00
fungiweird. that's not at all what i'm getting19:00
AJaegerclean tree19:00
*** yaume_ has quit IRC19:00
AJaegerstranger and stranger ;(19:00
* AJaeger double checks the tree19:00
AJaegerfungi, you have an old tree - we released 1.5.119:01
*** rbrndt has joined #openstack-infra19:01
AJaegerwrong, other project. 1.5.0 is last tag.19:02
fungiAJaeger: strangely, remote update isn't picking it up for me19:02
AJaegerSorry, mixed up releases19:02
AJaegeryou have pip 8.1.2, I use 7.1.219:02
*** kushal has quit IRC19:03
clarkbits noon already? I haven't even gotten through email :/ where did the day go19:03
fungiAJaeger: yeah, i'm using tox 2.3.1 and virtualenv 15.0.319:04
pabelangerclarkb: removing from glean, openssh-server just reload, not stop / start19:04
pabelangerclarkb: I am going to build an image and see if things still work19:04
clarkbpabelanger: I think it can still happen after if you don't have an explicit before19:04
clarkbpabelanger: I think it might be better to have an explicit Before sshd19:04
AJaegerour test results show pip 8.1.2 and tox 2.3.1 in the actual job19:05
*** ddieterly has joined #openstack-infra19:05
clarkbpabelanger: buit good to know we can manipulate glean's unit file to have it not stop start19:05
pabelangerclarkb: sure, we can do that too19:05
pabelangerclarkb: also removing fixes the dependency cycle that systemd complains about19:05
fungiAJaeger: right, i'm starting to wonder if this is a regression in virtualenv/pip/tox somewhere19:05
clarkbpabelanger: huh, wasn't that what we had to add to the urandom fixer unit file to make ti work?19:06
AJaegerit works with constraints - but I didn't want to enable those here ;)19:06
pabelangerclarkb: FWIW: we do have, which should protect before ssh starts19:06
fungiAJaeger: yeah, i think there's a bug... i'm going to try to bisect a few tools19:06
pabelangerclarkb: yes, and now I thinking more about it, it is also wrong. So we need to patch both19:06
AJaegerfungi, thanks a lot!19:07
fungiAJaeger: downgrading to virtualenv 14.0.2 gets me the (presumably proper) behavior you're seeing locally19:08
clarkbpabelanger: aha gotcha19:08
AJaegercool, fungi.19:08
fungiAJaeger: and 15.0.0 has the broken behavior19:08
AJaegerSo, wouldn't this cause quite some havoc in the gate? We could test wrong things here ;(19:09
openstackgerritPaul Belanger proposed openstack-infra/glean: Remove dependency cycle
pabelangerclarkb: ^19:09
pabelangerlike I said, going to do a quick build and test the image19:10
clarkbpabelanger: cool I will try to test a local build of that too19:10
AJaegerfungi,  thanks a lot for your great help with this! clarkb, mordred, others: Thanks as well!19:10
clarkbpabelanger: do you know if there is an easy way to make simple-init in dib pull that version of glean?19:10
pabelangerclarkb: not sure, haven't tried19:10
AJaegerfungi, will you file a bug? Should we block that virtualenv version?19:10
mordredclarkb: uhm ... I think so19:11
AJaegerSorry, I have to leave in a few minutes and call it a day ;(19:11
clarkbhrm except my lcoal virsh thing appares to have been broken by an update \o/19:11
clarkbqemu-bridge-helper where have you gone19:11
mordredgreghaynes: ^^ how do we build with dib using simple-init from local source dir?19:11
*** ilyashakhat_mobi has joined #openstack-infra19:12
greghaynesexport DIB_REPOLOCATION_glean=/path/to/glean19:12
mordredclarkb: see - greghaynes continues to be magical pony19:12
openstackgerritPaul Belanger proposed openstack-infra/project-config: Remove from initialize-urandom service
*** gyee has quit IRC19:13
pabelangerclarkb: and urandom fix^19:13
mordredpabelanger: nice catch, btw19:13
clarkbpabelanger: cool I am going to attempt to get my local virt setup working again so I can test an image with both of those things in it19:13
greghaynesmordred: clarkb one caveat is I think that will grab master of that repo, so you might also want to set DIB_REPOREF_glean=some_ref19:13
*** harlowja_ has joined #openstack-infra19:13
pabelangermordred: Ya, finally getting to optimize our nodepool launches.19:14
*** sarob has joined #openstack-infra19:14
*** sarob has quit IRC19:15
*** tonytan4ever has joined #openstack-infra19:16
* AJaeger waves good bye19:16
*** tonytan4ever has quit IRC19:16
*** harlowja has quit IRC19:17
*** eggshell has joined #openstack-infra19:18
fungihave a good evening AJaeger19:18
openstackgerritgreghaynes proposed openstack-infra/irc-meetings: Add diskimage-builder meeting agenda
openstackgerritMonty Taylor proposed openstack-infra/nodepool: Unsubvert image and flavor caching
*** sarob has joined #openstack-infra19:21
*** waht has joined #openstack-infra19:21
rcarrillocruzShrews: sigh, my wife had split shift today , could not attend ansible testing meeting19:22
rcarrillocruzi'll check chat logs19:22
*** _sarob has joined #openstack-infra19:22
rcarrillocruznikhil: that's awesome news, we'll keep an eye to implement that in shade + ansible when the refactor happens :-)19:22
sdaguejeblair: cool19:22
sdaguejeblair: it would also be good if the ansible logs got into elastic search19:23
sdagueto help look for patterns19:23
*** salv-orlando has joined #openstack-infra19:23
nikhilrcarrillocruz: aye19:25
*** spzala has quit IRC19:25
*** sarob has quit IRC19:25
*** spzala has joined #openstack-infra19:26
fungiurgh, something in one of the virtualenv 14.0.x/pip 6.0.x versions horked up my wheel cache19:29
fungithis makes bisection decidedly more complicated19:29
fungier, pip 8.0.x i mean19:30
*** ijw has quit IRC19:31
*** ijw has joined #openstack-infra19:32
fungiokay, so the problem seems to have crept in between virtualenv 14.0.5 and 14.0.619:32
*** annegent_ has quit IRC19:32
fungithe release history for 14.0.6 includes "Upgrade setuptools to 20.0" and "Upgrade wheel to 0.29.0"19:34
*** _nadya_ has joined #openstack-infra19:35
fungiso we can consider this as probably either a regression between setuptools 19.6.2 and 20.0 or wheel 0.26.0 and 0.29.019:36
*** tphummel has quit IRC19:37
*** ijw has quit IRC19:37
*** sdague has quit IRC19:37
*** _nadya_ has quit IRC19:40
*** ddieterly is now known as ddieterly[away]19:41
*** vhosakot has quit IRC19:41
fungithough interestingly, virtualenv 14.0.6 causes me to end up with setuptools 26.1.1 in the resulting venv19:42
*** vhosakot has joined #openstack-infra19:42
fungiscarily, the official documentation for setuptools only has up through 25.1.3 in their included changelog19:43
*** nstolyarenko has joined #openstack-infra19:43
fungioh, good, the CHANGES.rst in their git repo is up to date at least19:44
*** tongli has quit IRC19:44
mat128I have the answer to that question19:45
mat128fungi: ^19:45
mat128fungi: for some reason, virtualenv (trying to find it) installs a "bundled" pip with no version cap19:45
mat128and if you have a newer version in your wheel cache, it's going to be used19:45
mat128leading to confusion and virtualenv not acting as it's supposed19:45
mat128I had filed a bug report, trying to find it19:46
*** asettle has quit IRC19:46
fungimat128: yeah, that's why the claimed version of setuptools bundled in virtualenv is irrelevant, agreed19:46
zigoclarkb: fungi: pabelanger: Can we wrap up the discussion we just had a few hours ago?19:47
zigoIf I understand correctly, the major concern is that each image will eat up to 1.5 GB of cache data, which will globally slow down infra. Is this the only problem, or is there anything else?19:47
zigoAlso, since things are the way they are right now, and that Newton release is approaching (technically for me, it's already released as b3, and I'm already late), can we delay switching to overlay mode for after Newton?19:47
zigoLast, can we decide that we will use the overlay mode *only* for when upstream is OpenStack?19:47
*** asettle has joined #openstack-infra19:47
mat128if that line was changed to pip==VERSION19:47
mat128it'd work correctly19:47
fungimat128: so unfortunately this means the problem is somewhere between pip 8.0.2 and 8.1.2 or setuptools 19.6.2 and 26.1.1 or wheel 0.26.0 and 0.29.019:48
mat128in the meantime, flushing your wheel cache seems like the only way out19:48
*** nstolyarenko has quit IRC19:48
*** asettle has quit IRC19:48
*** mriedem has joined #openstack-infra19:49
pabelangerclarkb: okay, I think I have a minimal DIB working with glean and our urandom element, just confirming now19:49
fungizigo: i'm a little worried that we didn't anticipate the nature of the repos you were importing, and that they already include a fair amount of git history from the corresponding upstream project repos. cleaning that up is likely to be complicated19:50
clarkbpabelanger: ok I just got my virsh issue sorted out19:50
zigofungi: That's the case indeed.19:50
fungie.g. looks like it's not just the debian directory but rather an entire nova source code tree19:50
zigofungi: We already imported everything.19:50
clarkbpabelanger: but gonna try to get the buiod started then go eat lunch19:51
zigofungi: Also, I'm really not sure how I will do for keeping the packaging history and changing the hosted format ... :/19:51
zigofungi: I really need answers for my above concerns ASAP. Time is running... :/19:52
fungizigo: yeah, i'm inclined at this point to just go along with what you want because the bulk of the damage is already done and i don't have a good answer for how to go back and fix it19:52
zigofungi: So, would you agree that we delay the transition for after the Newton release?19:53
fungiwhich would probably involve git filter-branch to trim out everything except the debian subtree and forcve-pushing teh result back over the existing repos19:53
zigo(if we decide to do so...)19:53
fungizigo: yeah, in the near term merging a few more upstream tags isn't going to make the existing situation considerably worse19:53
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Replace watch thread with periodic thread
zigofungi: In such case, how can I get the ACL to do merge commits?19:54
*** Goneri has quit IRC19:54
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Replace watch thread with periodic thread
fungizigo: we have it documented at
*** jcoufal_ has quit IRC19:56
*** abregman has joined #openstack-infra19:57
*** asettle has joined #openstack-infra19:59
*** asettle has quit IRC19:59
rcarrillocruzpabelanger: oh, we have cirros or some other image in our mirrors?20:00
rcarrillocruzor which file you refer to20:00
clarkbpabelanger: ok my build is started20:01
*** _nadya_ has joined #openstack-infra20:01
clarkbwill have to see how that goes20:01
pabelangerrcarrillocruz: ~/cache/files is what I was referring too20:01
pabelangerclarkb: rebuilding, I messed up the glean variable20:01
clarkbgonna grab lunch while its going20:01
clarkbpabelanger: what vars do you think you need? I used ehat greghaynes said20:02
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: Use centos CDN repository with periodic jobs
pabelangerclarkb: Ya, I did use it, I had a typo in my path20:02
*** piet has quit IRC20:02
openstackgerritGuillaume Espanel proposed openstack-infra/project-config: Create puppet-cloudkitty repository
*** florianf has quit IRC20:03
fungiconfirmed the pip cache is definitely playing a part in this behavior. testing with various versions i got it into a state where older virtualenv releases that had been working for me started exhibiting the problem behavior until i blew away ~/.cache/pip20:04
*** ijw has joined #openstack-infra20:04
mat128fungi: I tried very hard to find the bug report I remember submitting, but can't find anything20:04
*** ddieterly[away] is now known as ddieterly20:04
mat128fungi: must have had bad dreams about virtualenv and pip.. can't find any trace of my experiment either :(20:05
*** annegentle has joined #openstack-infra20:05
mat128fungi: if you find a way to disable the wheel cache (env variable or smt), that could help as well20:05
fungimat128: i'm less concerned with that to be honest. the current and more troubling issue is that starting with virtualenv 14.0.6 we seem to be installing cached wheels of downloaded releases when we ask it to pip install the current checked out source tree20:05
mat128fungi: are we using explicit versions?20:06
fungii think it's something after wheel 0.26.0 but i'm still narrowing it down20:06
*** derekh has joined #openstack-infra20:06
openstackgerritThomas Goirand proposed openstack-infra/project-config: Add merge commit ACL for packaging-deb
zigofungi: Like this? ^20:06
mat128fungi: can you reproduce it easily?20:06
fungimat128: explicit versions of what?20:06
mat128fungi: pip install package==1.0.020:06
mat128or constraints20:06
fungimat128: absolutely, i'm whittling down versions of things in the toolchain to narrow down whether it's virtualenv, pip, wheel or setuptools at fault20:06
zigo(minus the english mistakes in the commit header... :( )20:06
fungimat128: no constraints in use in this case20:07
mat128fungi: so let me try to understand the issue: you have software==1.1 in your wheel cache, you issue pip install software==1.0 and 1.1 gets installed?20:07
fungimat128: the short version is demonstrated in and
*** _nadya_ has quit IRC20:08
fungimat128: it does a pip install -r test-requirements.txt where one of the packages listed there is a circular dependency back on the project being tested and so pulls down the packaged release, then does a pip install -e . and ends up i think installing the cached wheel instead of the current source tree20:09
*** ijw has quit IRC20:09
mat128what about ``pip install -U -e .``20:09
funginot sure yet, i'm less concerned with workarounds and more with bisecting to a specific version in the toolchain that introduces this regression20:10
fungiit's just a little fiddly since some of teh parts are vendored so there's some chicken-and-egg problems pinning them independently20:11
*** tonytan4ever has joined #openstack-infra20:11
*** coolsvap has quit IRC20:12
*** flepied has quit IRC20:12
fungishort answer for a reproducer is to clone the openstack/openstackdocstheme repo and then `tox -er venv pip list` and look at the version of openstackdocstheme it reports installed20:12
fungicomparing virtualenv 14.0.5 and 14.0.6 behavior (or 14.0.5 and latest release for that matter)20:12
*** gyee has joined #openstack-infra20:13
mat128openstackdocstheme (1.5.0, /Users/mmitchell/projects/openstackdocstheme/.tox/venv/lib/python2.7/site-packages)20:13
rcarrillocruzpabelanger: oh nice, did not know we had cirros at cache/files, thanks!20:13
fungias noted, i've had to introduce a `rm -rf ~/.cache/pip` into the loop because of the wheel cache from other versions influencing the test result20:14
*** kzaitsev_mb has quit IRC20:14
openstackgerritJames E. Blair proposed openstack-infra/infra-specs: Zuulv3: drop variable interpolation and add nodesets
mat128fungi: pip 8.1.2, virtualenv 15.0.220:15
fungii think i've mostly ruled it out as a regression in wheel. seems more like setuptools but i'm not sure where in its history this crops up yet20:15
mat128oh, virtualenv 1420:15
*** maishsk has quit IRC20:15
jeblairfungi: ^ fixed the merge conflict on that, should be ready for merging when fires are out.  or maybe just smoldering.20:15
*** fguillot has quit IRC20:16
fungijeblair: oh, cool!20:16
*** tonytan_brb has joined #openstack-infra20:16
ianwhas the idea of zuul-cloner being a separate thing to zuul been covered before?20:16
*** maishsk has joined #openstack-infra20:16
fungiianw: we've talked about breaking out the cli tools, but the arguments for doing so have been fairly shallow20:16
*** tonytan4ever has quit IRC20:17
fungimat128: getting closer... i think it's around setuptools 2520:17
*** esikache1 has quit IRC20:18
*** spzala has quit IRC20:19
*** markusry has quit IRC20:20
fungimat128: bingo... put setuptools==24.3.1 in requirements.txt and compare against 25.0.020:20
*** flepied has joined #openstack-infra20:21
*** kzaitsev_mb has joined #openstack-infra20:21
fungii wonder if SETUPTOOLS_SYS_PATH_TECHNIQUE=rewrite changes this20:21
mat128fungi: confirmed here too20:22
mat128fungi: yes, that fixes it20:23
fungi"This project hopes that that few if any environments find it necessary to retain the old behavior, and intends to drop support for it altogether in a future release. Please report any relevant concerns in the ticket for this change."20:23
fungii guess we have one :/20:24
mat128installing a project from source doesnt seem like a rare case20:24
fungithis is probably a pip -e case20:24
fungii'm going to fiddle a little more to see if usedevelop=False changes things20:24
mat128fungi: look at the paths20:25
openstackgerritMonty Taylor proposed openstack-infra/shade: Batch calls to list_floating_ips
mat128fungi: one is from site-packages (as you said, pulled via dep resolution)20:25
*** dprince has quit IRC20:26
*** ijw has joined #openstack-infra20:26
clarkbpabelanger: my local.instance never dhcped20:26
fungiokay, so usedevelop=false does indeed cause the problem to go away20:26
fungiwhich means setuptools 25.0.0 basically broke editable installs20:27
*** pvaneck has quit IRC20:27
clarkbpabelanger: so rebuilding to set apassword to logij on console20:27
*** nmagnezi_ has joined #openstack-infra20:27
*** dtardivel has quit IRC20:27
clarkbpabelanger: I think thr ordering for glean might be more subtle20:27
fungimat128: i'll update the bug mentioned in the changelog and see where that gets us20:27
mat128fungi: yeah, probably our best bet20:28
*** ddieterly is now known as ddieterly[away]20:28
*** eggshell has quit IRC20:28
mat128fungi: and the environemnt variable is our temporary workaround20:28
pabelangerclarkb: yup20:28
*** nmagnezi has quit IRC20:28
fungi for the record20:28
fungimat128: i'll come up with a reproducer that doesn't rely on tox, now that i know what the problem is20:29
*** e0ne has quit IRC20:30
*** gordc has quit IRC20:30
*** ddieterly[away] is now known as ddieterly20:30
*** itisha has quit IRC20:30
*** ijw has quit IRC20:31
lifelessfungi: yay quagmire20:32
lifelessfungi: lets all switch to rust :)20:32
mordredlifeless: if only20:32
prometheanfiresomeone mind looking at this for the release process?
prometheanfireit'd help with getting the bot submitted reviews for target branches working for requirements20:33
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Replace watch thread with periodic thread
*** gouthamr has quit IRC20:34
openstackgerritDoug Hellmann proposed openstack-infra/project-config: do not error if a branch already exists
fungilifeless: indeed. who needs editable installs anyway? ;)20:37
*** annegentle has quit IRC20:39
*** yfried has joined #openstack-infra20:40
*** ijw has joined #openstack-infra20:41
mrhillsmanany thoughts on the time it will take to get some workload on cloud8?20:41
ianwlifeless: if around ... <- is it possible to query how you were installed with a various environments?  or is the idea you would just do an import and catch exceptions?20:43
openstackgerritThomas Goirand proposed openstack-infra/project-config: Add merge commit ACL for packaging-deb
*** kgiusti has left #openstack-infra20:44
*** ijw has quit IRC20:46
*** derekh has quit IRC20:47
*** annegentle has joined #openstack-infra20:47
*** piet has joined #openstack-infra20:49
openstackgerritMonty Taylor proposed openstack-infra/shade: Fix up image and flavor by name in create_server
*** maishsk has quit IRC20:50
mordredarmax, dtroyer: neutronclient released. new os-client-config released - the gate does not seem to have completely broken ... so I think we can consider that good!20:51
ianwtimothyb89: around?  I wouldn't mind rolling stackviz cleanup into my devstack-gate cleanup-refactor, just to get it all done.  want to pick your brain on what you think is supposed to be happening under regular & grenade20:51
*** maishsk has joined #openstack-infra20:51
armaxmordred: someone in the neutron channel said something got belly up20:52
timothyb89ianw: sure, happy to help20:52
*** ijw has joined #openstack-infra20:52
mordredarmax: oh no20:52
*** pvaneck has joined #openstack-infra20:52
ianwtimothyb89: so see ... that has old & new and each has their tempest run20:53
*** baoli_ has quit IRC20:53
ianwtimothyb89: do you expect that both would have stackviz output too?20:53
mordredjeblair: so ... interesting edge case for your brainhole wrt zuul20:53
mordredprometheanfire: ^^ (about to mention the neutron thing to jeblair)20:54
prometheanfiremordred: we are handling it in -release20:54
timothyb89ianw: ideally if a *.subunit exists, stackviz should too20:54
mordredprometheanfire: well, there is a thing here which is, I think, potentially worth considering on the zuul side for future development ...20:54
mordredjeblair: release automation released a new version of python-neutronclient (yay!) and then submitted a patch to global requirements bumping the constraints for the release20:55
mordredjeblair: the tests for that change ran before the release artifact had made it to the mirrors20:55
clarkbpabelanger: ok on my local instance I don't see glean or ssh ever start20:56
mordredjeblair: I'm mentioning it because it sounds similar to me to one of the things we've heard from distro folks - sometimes they'd like to run a job in response to an artifact being uploaded, not to a git commit20:56
clarkbpabelanger: now to determine if it is pebcak during build20:56
mordredjeblair: and this seems like a specific case of that for if/when we get around to musing about such a thing20:56
ianwtimothyb89: so if you look at _copy_devstack_logs in -> .  if stackviz is installed, can it use the "tempest.subunit" file in there, like the subunit2html file does?20:57
ianwtimothyb89: would i just "cat tempest.subunit" | stackviz ... ?20:58
timothyb89ianw: there are a few extra steps, but that would work20:59
timothyb89ianw: though specifically on it looks like a permissions issue of some sort?21:00
*** Gibi is now known as gibi21:00
*** nmagnezi_ has quit IRC21:01
pabelangerclarkb: can you see glean in systemctl?21:01
*** ddieterly is now known as ddieterly[away]21:01
clarkbpabelanger: no21:01
*** raildo has quit IRC21:01
clarkbpabelanger: but it is installed21:01
clarkbthe command glean is installed I mean21:01
pabelangerjust bringing my image online now21:01
clarkband its definitely not configuring eth0 which means no dhcp21:02
pabelangerlooks like glean failed to start for me too21:02
pabelangersame with initialize-urandom21:02
ianwtimothyb89: there is that ... i think it's confusion between when grenade uses "new" v the top-level21:03
*** jkilpatr has quit IRC21:03
pabelangerclarkb: you likely see Ordering cycle found, skipping Network21:03
*** ddieterly[away] is now known as ddieterly21:03
ianwtimothyb89: how big is "stackviz"?  if we have old/stackviz and new/stackviz are we duplicating a lot?21:03
jeblairmordred: maybe the constraints update shouldn't come from the release tag job; maybe it should happen in response to the artifact upload job?21:03
ianwtimothyb89: can you have one "stackviz" dir and put two lots of data in?21:04
jeblairmordred: (in other words, look at it as not being the reponsibility of the release pipeline to update constraints; look at it as the responsibility of the project to update constraints when it's released (regardless of *how* the release is triggered).21:04
*** berendt has joined #openstack-infra21:04
clarkbpabelanger: no its more like glean just isn't enabled at all for some reason21:04
clarkbpabelanger: systemctl status glean says Loaded: not-found (Reason: No such file or directory)21:06
timothyb89ianw: stackviz can accept multiple files in one directory, yes21:06
clarkbpabelanger: but the unit file is in /usr/lib/systemd/system21:06
pabelangerclarkb: it will be glean@eth0.service I think21:06
timothyb89ianw: something like, `stackviz-export -f old/testrepository.subunit -f new/testrepository.subunit $log_path/stackviz/data` would work I think21:07
mordredjeblair: I believe that is what the release team is going to do21:07
*** trown is now known as trown|outtypewww21:07
clarkbpabelanger: there is glean@.service21:08
clarkbpabelanger: is your instance working?21:08
ianwtimothyb89: great, i'll try that.  will it work if i do the exports separately, or do they have to be done from the one stackviz-export call?  making two calls would be easier and fit into the existing fn21:08
*** yfried has quit IRC21:08
pabelangerclarkb: still figuring that out21:08
mordredjeblair: I was more just bringing it up as a local example of a thing people who are not in openstack have expressed wanting to be able to do21:09
timothyb89ianw: right now it would need 1 call, but we can always just make 2 copies like we do now if that's easier21:10
*** rvasilets___ has joined #openstack-infra21:10
*** eharney has quit IRC21:10
clarkbpabelanger: hrm I still have an in that file so I may not have gotten glean/simple-init to install properly21:10
timothyb89ianw: actually, 2 separate may still be the best option so dstat is included ... the export CLI only accepts 1 dstat input right now21:11
ianwtimothyb89: ok, cool ... let me get something together and we can see how it looks21:12
timothyb89ianw: sounds good!21:12
*** kaisers_ has joined #openstack-infra21:14
pabelangerclarkb: ya, same. glean was installed by pip for me21:15
openstackgerritIlya Shakhat proposed openstack-infra/project-config: Rename os-failures to os-faults
*** ilyashakhat_mobi has quit IRC21:15
*** ldnunes has quit IRC21:16
clarkbpabelanger: we need to set DIB_INSTALLTYPE_simple_init=repo21:17
clarkbpabelanger: so I rebuilding with that. But I don't know why its otherwise broken for me21:18
clarkbpabelanger: did glean the service run for you?21:18
*** kaisers_ has quit IRC21:18
*** rossella_s has quit IRC21:18
fungimat128: lifeless:
pabelangerclarkb: didn't come up for me, I don't know why21:18
clarkbpabelanger: same problem likely then :)21:18
fungidstufft: ^ that issue is probably of interest to you as well, as pip maintainer21:18
clarkbpabelanger: I used devuser to create a user with password then logged in on console21:18
*** rossella_s has joined #openstack-infra21:19
pabelangerclarkb: ya, going to do the same21:19
*** aeng has joined #openstack-infra21:20
*** tphummel has joined #openstack-infra21:21
fungioh, and now i find and basically already cover this21:21
openstackgerritMatthew Thode proposed openstack-infra/project-config: Pause before submitting the requirements review
clarkbpabelanger: manually running `sudo systemctl start glean@ens3` worked21:24
* prometheanfire likes glean21:24
*** rcernin has quit IRC21:24
pabelangerclarkb: okay, lets hope this DIB works21:25
clarkbnow to figure out why it didn't fire on boot21:25
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: Refactor stackviz run
pabelangerclarkb: did you rebuild?21:25
ianwtimothyb89: ^ that's the idea split out21:25
clarkbpabelanger: also there isn't au nit file for each individual interface instead the glean@ unit file is a template that takes the interface dev name as an argument21:25
fungimat128: AJaeger: so in summary, has already merged to fix the behavior in pip, and will likely be in pip 8.1.3 or whatever the next release is21:25
clarkbpabelanger: still rebuilding. But also debugging on my old broken host21:25
*** Goneri has joined #openstack-infra21:26
*** thorst has quit IRC21:26
fungimat128: AJaeger: oh, not yet merged, i misread21:26
clarkbpabelanger: oh there was an ordering dependency why did that not show up in dmesg?21:26
*** thorst has joined #openstack-infra21:27
clarkbpabelanger: apparently I have to ask journalctl for logs that actually matter :(21:27
timothyb89ianw: that looks a lot better! small issue, though, the '--end' flag copied over is actually an old typo, that should be '--env'21:27
*** matt-borland has quit IRC21:27
clarkbpabelanger: so its possible the fix might actuall fix this21:27
pabelangerclarkb: Ya, that is the original ordering issue21:27
pabelangerfor some reason, the magic make it work today21:28
clarkbI should just alias dmesg to journalctl -xe21:28
timothyb89ianw: apparently that branch was never followed since it should have been spitting out errors for months... whoops21:28
*** jerryz has quit IRC21:28
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: Refactor stackviz run
*** rtheis has quit IRC21:29
clarkbpabelanger: I am to the qemu img convert so hopefully will have answersoon21:29
*** pt_15 has quit IRC21:29
ianwtimothyb89: heh, it was really really hard to follow.  i think too much "organic" growth21:29
timothyb89ianw: that, and probably some poor testing on my part... I guess that explains why dstat hasn't been showing up, though21:30
*** abregman has quit IRC21:31
*** thorst has quit IRC21:31
*** claudiub has joined #openstack-infra21:31
*** abregman has joined #openstack-infra21:31
*** ddieterly has quit IRC21:31
openstackgerritMonty Taylor proposed openstack-infra/shade: Fix up image and flavor by name in create_server
timothyb89ianw: for what it's worth, I'm hoping to remove the stackviz-export step entirely in the near future, hopefully just the single `sudo cp  -r ...` will be required21:32
*** gouthamr has joined #openstack-infra21:32
clarkbpabelanger: ok glean worked but initialize-urandom failed due to no such file or directory. I don't know which file or directory yet21:33
*** abregman has quit IRC21:33
pabelangerclarkb: oh, maybe we are missing haveged21:36
clarkbpabelanger: oh!21:36
pabelangerI think that is a dependency21:36
clarkbits not part of the initialize-urandom element21:36
* clarkb builds another image21:36
pabelangerya, we can added it as a pkg-map21:37
*** _sarob has quit IRC21:37
clarkbI just added it to infra package needs really quick totest but ya would be better in initialize-urandom21:38
pabelangererr, package-install.yaml21:38
pabelangerthat works21:38
*** shardy_afk has quit IRC21:38
pabelangerrebuild started.... again21:38
clarkb:) isn't it fun how fixing things like this ends up being21:39
fungimordred: dhellmann: so... revisiting this apparently intentional behavior change in setuptools path ordering will result in future as-of-yet-unidentified modifications in pip's behavior around editable installs, can you remind me why some projects use them for tox in the first place?21:39
pabelangerI'm hoping at the mid-cycle we can talk about DIB elements for a bit21:39
*** shardy_afk has joined #openstack-infra21:40
clarkbpabelanger: part of the problem here is the normal builds take forever and are massive due to all the caching so I try to avoid that and just build with ubuntu-minimal simple-init growroot devuser infra-package-needs (for ssh) and initialize-urandom21:40
clarkbbut then we find where we have undeclared deps between elements21:40
clarkbbut also undocumented flags in simple-init that need setting aren't fun either21:41
clarkbI should push a patch to dib now to fix that21:41
*** vhosakot has quit IRC21:41
pabelangerYa, don't want to get into it now. But there is some frustrations around it.  I am sure we could make it better, with a little programming21:42
*** adriant has joined #openstack-infra21:42
mordredfungi: because it shortens the iteration cycle21:42
pabelangerokay, will be back shortly, need to walk down to store for something21:42
*** hashar has quit IRC21:42
*** priteau has quit IRC21:42
mordredfungi: for things like nova, needing install to run after each edit before running tox can be a significant delay21:42
mordredfungi: the that nova used before we went tox did editable installs into a virtualenv rather quickly, and the devs were annoyed by how long tox took21:43
*** gouthamr_ has joined #openstack-infra21:44
fungimordred: got it. and specifically they want to do it with tox21:44
*** ijw has quit IRC21:44
fungiso we can't just take usedevelop out. and the SETUPTOOLS_SYS_PATH_TECHNIQUE envvar is targeted for future removal from setuptools21:44
*** adriant has quit IRC21:45
mordredfungi: so - should I go read the bug in question?21:45
mordredfungi: like, why have they decided to break -e ?21:45
fungiit doesn't look like setuptools is interested in changing this back to the old behavior, and potential fixes have been proposed to pip but don't seem to have garnered much interest from pip maintainers21:46
fungimordred: explains the reason for the behavior change21:46
openstackgerritClark Boylan proposed openstack/diskimage-builder: Document source glean installs in simple-init
mordredfungi: cool21:46
clarkbgreghaynes: ianw ^21:46
*** gouthamr has quit IRC21:47
cineramaclarkb: thanks!21:47
mordredfungi: it seems like what we _really_ want to happen in tox21:47
fungimordred: so anyway, i think we have to consider the possibility that usedevelop=true in tox.ini is simply unsafe (and certainly currently broken with latest pip/setuptools)21:47
mordredfungi: is the sequence of "pip install -e . ; pip install -r test-requirements.txt"21:48
fungimordred: other way around21:48
mordredfungi: since that should get the -e install to satisfy the transitive/circular dep21:48
*** Thelo_ has quit IRC21:48
fungimordred: basically tox runs your install_command first, then installs the local tree21:48
*** inc0 has quit IRC21:49
fungiso it's in some cases getting a packaged version of whatever we're testing pulled in, and then does the editable install of the local source tree after21:49
clarkbpbr has this behavior21:49
clarkbwe had a workaround in it involving put . in the deps list or something21:50
clarkb(since pbr bootstraps itself)21:50
*** yamahata has quit IRC21:50
mordreddeps = .21:51
mordred       -r{toxinidir}/test-requirements.txt21:51
ianwclarkb: lg, it might be helpful to give an example of using a upstream review and getting the ref from there.  i've manually talked people through that a couple of times (should have taken the time to document it :)21:52
*** ddieterly has joined #openstack-infra21:53
*** cardeois has quit IRC21:56
clarkbpabelanger: looks ok to me it still reloads (and seems to start with a "I shut down ok" message) but I think thats all fine21:56
clarkbianw: oh thats a good idea ya I can make it more verbose21:56
*** thorst has joined #openstack-infra21:57
*** Goneri has quit IRC21:57
*** berendt has quit IRC21:58
clarkbpabelanger: also initialize-urandom and glean both ran before the ssh things started21:58
clarkb(I also really don't like that its ssh not sshd such muscle memory)21:59
*** adriant has joined #openstack-infra21:59
clarkbpabelanger: you good with me approving the glean change, then we need a release before we can approve the urandom one22:00
*** thorst has quit IRC22:01
*** yamahata has joined #openstack-infra22:02
openstackgerritClark Boylan proposed openstack/diskimage-builder: Document source glean installs in simple-init
clarkbianw: ^ is that better?22:03
*** ijw has joined #openstack-infra22:04
*** fguillot has joined #openstack-infra22:05
*** javeriak has joined #openstack-infra22:06
*** Julien-zte has quit IRC22:07
*** rlandy is now known as rlandy|bbl22:07
ianwclarkb: cool.  you can even do it directly from the review, but people can probably figure that out22:08
ianwi mean upstream git.  the hardest part is finding the pull-down in the top-right corner of gerrit ui22:08
*** javeriak_ has quit IRC22:09
*** Julien-zte has joined #openstack-infra22:09
clarkbianw: I hope my example also shows how to test local edits by explaining it that way22:09
clarkbeg before pushing to gerrit22:10
rcarrillocruzpabelanger: mind reviewing pls? it pulls from file now22:10
*** Swami has quit IRC22:11
*** rlandy|bbl is now known as rlandy22:11
*** rlandy is now known as rlandy|bbl22:12
openstackgerritMerged openstack-infra/shade: Batch calls to list_floating_ips
*** vhosakot has joined #openstack-infra22:13
*** esberglu has quit IRC22:14
*** tphummel has quit IRC22:14
*** xyang1 has quit IRC22:14
*** krotscheck has quit IRC22:15
*** krotscheck has joined #openstack-infra22:15
clarkbmordred: what are your thoughts on making a glean release nowish?22:15
clarkbmordred: we will need to coordinate that and the fix for initialize-urandom because if we don't then boot doesn't work22:16
*** ilyashakhat_mobi has joined #openstack-infra22:16
clarkbmordred: glean can't bring up interfaces properly if we run the current glean against the initialize-urandom fix22:16
clarkbits good fun22:16
*** ddieterly is now known as ddieterly[away]22:17
*** javeriak_ has joined #openstack-infra22:17
*** javeriak has quit IRC22:17
*** Thelo_ has joined #openstack-infra22:17
*** Thelo_ has quit IRC22:19
*** tphummel has joined #openstack-infra22:19
*** spzala has joined #openstack-infra22:20
*** piet has quit IRC22:20
*** ilyashakhat_mobi has quit IRC22:21
mordredclarkb: oh lovely22:21
mordredclarkb: well, I am in support of releasing glean anytime you think is good22:21
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: Refactor stackviz run
clarkbmordred: I want pabelanger to confirm his test build functioned then I think we can get both things in and glean released22:22
clarkbdtroyer: stevemar did that osc neutron https thing get merged and released yet? I can't find the bug anymore and I fail at googling22:22
ianwi don't want to mess things up, but i have a bunch of glean stuff just sitting ->
pabelangerclarkb: back22:23
ianwmostly test cleanups22:23
mordredianw: two of those are merge-conflict now22:23
pabelangerlet me test quickly22:23
pabelangerbuild is done22:23
*** mdrabe has quit IRC22:24
openstackgerritIan Wienand proposed openstack-infra/glean: Refactor side-effect functions
openstackgerritIan Wienand proposed openstack-infra/glean: Patch for unit testing
openstackgerritIan Wienand proposed openstack-infra/glean: Use mock patching for most functions
openstackgerritIan Wienand proposed openstack-infra/glean: Add selinux context manager for writing files
*** spzala has quit IRC22:24
mordredclarkb: you have opinons on ?22:24
clarkbmordred: no I suffer mock where it exists :)22:24
ianwmordred: ^ hmm, rebase *seemed* happy ...22:25
clarkbmock is the python3 happy one right?22:25
clarkbI guess that would be the only thing I would want is to make sure we don't regress like mox22:25
*** adriant has quit IRC22:25
mordredyah. mock is the python3 happy one22:25
*** yolanda has quit IRC22:26
*** ramishra has quit IRC22:27
pabelangerclarkb: blarg, my build actually failed.22:28
pabelangerclarkb: I'm happy to use your tests22:28
pabelangerwhile I figure out why my dib failed22:28
*** ramishra has joined #openstack-infra22:29
clarkbpabelanger: ok, so shoudl I approve the glean chagne then and ask modrred for a release then we can approve the project-config change?22:30
pabelangerclarkb: Ya, lets do that22:30
*** Thelo_ has joined #openstack-infra22:30
clarkbok glean change approved22:31
dtroyerclarkb: that was in the release of the SDK that just got put into g-r (0.9.5 IIRC)22:31
*** Thelo_ has quit IRC22:32
dtroyerI haven't tested it myself yet though22:32
openstackgerritMerged openstack-infra/glean: Remove dependency cycle
clarkbmordred: ^ you want to do the honors of a release?22:35
clarkbor maybe get some of ianw's in first?22:36
clarkbif they are all tests not sure how urgent they are22:36
clarkbianw: any of them make snse to try and get into a release if it happens nowish?22:36
*** ddieterly[away] is now known as ddieterly22:38
ianwi don't think it's super urgent.  the selinux one was to help with, well, selinux.  the others were test-cleanups that feel out of trying to test it22:39
ianwi'm just catching up ... are you sure it's the After?  as described in that's mostly a *shutdown* thing22:39
*** Thelo_ has joined #openstack-infra22:40
ianw" has very little meaning during start-up ... It's primary purpose is for ordering things properly at shutdown:"22:40
*** shardy_afk has quit IRC22:40
*** Thelo_ has quit IRC22:41
*** ddieterly is now known as ddieterly[away]22:42
*** sarob has joined #openstack-infra22:43
*** tonytan_brb has quit IRC22:43
openstackgerritMonty Taylor proposed openstack-infra/nodepool: Unsubvert image and flavor caching
clarkbianw: I know that those two changes to remove the After result in no more ordering cycle22:45
clarkbianw: and it prevents ssh from being stop started22:45
mordredclarkb: let's go ahead and release ... there are a LOT of changes unreleased - I hestitate to land more given the debug cycle22:45
*** adriant has joined #openstack-infra22:45
clarkbianw: there were two interrelated problems. The ordering cycle which just made things messy and ssh was being started then stopped then started22:46
clarkbianw: so if connections came in during the first start they would get killed when the service was stopped22:46
mordredclarkb: 1.6.0 ?22:47
*** sarob has quit IRC22:47
clarkbmordred: uh the change I know of should be backaard compat and it just changes a bug so even a point would be fine22:47
clarkbbut not sure about all the other changes22:47
*** signed8bit is now known as signed8bit_Zzz22:48
jeblairpabelanger, mordred, clarkb, and i guess someone can tell sdague if they see him: here's what i've noticed about those ansible ssh connection failures: 1) they seem *vaguely* time correlated.  they seem to come in batches, and the batches happen on all the zuul launchers.  2) i suspect they may be hitting a small number of jobs; related: they seem to disproportionately happen on multinode jobs.  3) the ssh failure happens 25 seconds ...22:48
jeblair... after the last poll.  our polling interval is 5 seconds, and there is a 10 second ssh connection timeout.  i can't account for the other 10 seconds.22:48
jeblairthere is no additional information from the async module about the ssh error, even with -vvv.  (this is perhaps not surprising at this point)22:49
mordredclarkb: that's the changelog entries since the last release22:49
jeblairi'm turning off verbose mode on the launchers22:50
mordredclarkb: I think bonding additions make it 1.6.022:50
clarkbmordred: ah ya that would be a new feature22:50
mordredok. Im pushing the tag22:51
mordredand done22:51
mordredmay god have mercy on our souls22:51
clarkbmordred: is the corresponding one now that that is done22:52
jeblairpretty sure that ship has sailed22:52
*** Swami has joined #openstack-infra22:52
clarkband yes lets hope new images work :)22:52
mordredjeblair: good point22:52
clarkbgood thing we won't get those until after dhellmann is done22:52
*** Swami_ has joined #openstack-infra22:52
mordredclarkb: that sohuld be safe to land now, yeah?22:52
mordredsince glean will exist next time an image is built22:53
clarkbmordred: ya22:53
mordredk. done22:53
*** gouthamr_ has quit IRC22:53
fungiglad to see there will finally be a fix for that issue22:54
*** ijw has quit IRC22:56
*** javeriak_ has quit IRC22:57
clarkbmordred: pabelanger the good thing is if that breaks xenial/centos/jessie it won't break any jobs22:57
clarkbsince it will just prevent us from sshing in to unbooted hosts22:57
clarkbbut I was able to boot locally and dhcp so I am happy22:58
*** sarob has joined #openstack-infra22:58
mordredclarkb: if you have a sec ... is part of me working through the suboptimal caching in nodepool right now22:59
*** annegentle has quit IRC23:01
openstackgerritMerged openstack-infra/project-config: Remove from initialize-urandom service
*** kaisers_ has joined #openstack-infra23:02
*** hongbin has quit IRC23:03
*** sarob has quit IRC23:03
*** ijw has joined #openstack-infra23:04
*** ddieterly[away] has quit IRC23:05
*** dimtruck is now known as zz_dimtruck23:05
*** zz_dimtruck is now known as dimtruck23:05
pabelangerclarkb: do we want to kick off an image-build tonight for ubuntu-xenial?23:06
*** kaisers_ has quit IRC23:07
openstackgerritClark Boylan proposed openstack-infra/project-config: Run requirements/pbr integration on xenial
*** rbrndt has quit IRC23:08
clarkbmordred: tonyb ^ comments on that welcome. I think it should be relatively safe to do as soon as dhellmann is done making milestone 3 things happen23:08
clarkbpabelanger: we could. Then upload to osic real quick and see if it works23:09
clarkbpabelanger: probably a decent idea. Just make sure that glean hits pypi first23:09
*** ijw has quit IRC23:09
*** gouthamr has joined #openstack-infra23:10
*** salv-orlando has quit IRC23:10
openstackgerritMonty Taylor proposed openstack/os-client-config: Add ability to configure Session constructor
dhellmannclarkb : the plan is to finish tagging by 1400 UTC tomorrow when the release team meeting starts, then branch after that for the libs that don't have branches, then some time later in the day tomorrow there's gerrit downtime IIRC. I can let you know here when we're done with the branches23:10
*** markvoelker has joined #openstack-infra23:10
clarkbdhellmann: sounds good thanks23:11
clarkbmordred: question about that caching change. We have two different cache settings in the one clouds.yaml23:11
mordredclarkb: in the ansible one?23:11
clarkbmordred: yes23:11
mordredclarkb: yah - one is the per-resource cache setting - which is really the batch/poll setting23:11
clarkblooks like one is for the inventory cache?23:11
mordredclarkb: the other is "how long should an entire copy of the inventory be kept around"23:12
clarkband it won't conflate the two?23:12
mordrednope. they're completely different things23:12
clarkbok approved23:12
mordredif there is a valid inventory cache, ansible won't execute any shade calls at all23:12
clarkbah gotcha23:12
clarkbbut while building an inventory it will use the other cache settings23:12
mordredin the fullness of time, I want to get the other cache stuff good enough that the giant-hammer inventory cache is not really a thing23:13
mordredbut we're not there yet23:13
pabelangerclarkb: okay, once nodepool updates, I'll kick off the build23:13
jeblairclarkb, pabelanger, mordred: this is the number of counts of ansible ssh failures per job over the past month:
jeblairwe'd probably need to normalize that by number of jobs run to really pick up a pattern23:14
mordredjeblair: wow:  3939 gate-tempest-dsvm-neutron-full-ubuntu-xenial23:14
*** gouthamr_ has joined #openstack-infra23:14
clarkbmordred: thats the ipv6 issue23:14
*** gouthamr has quit IRC23:14
jeblairit is?23:14
clarkbjeblair: ya it only affected jobs using neutron and it was every job using neutron basically23:14
jeblairand that showed up as the problem i'm looking at?23:15
clarkbjeblair: you should be able to tell reasonably well if it drops off in the last week or so23:15
*** shashank_hegde has quit IRC23:15
*** dimtruck is now known as zz_dimtruck23:15
clarkbjeblair: it showed up as ssh connect failed yes23:15
*** xarses has quit IRC23:15
jeblairclarkb: without setting unreachable?23:15
ianwtimothyb89: so i guess running it for "old" and "new" doesn't quite work to show separate outputs ->
clarkbjeblair: I am not sure what that means23:16
pabelangerI think ipv6 was different, because those jobs were requeue in zuul23:16
clarkbjeblair: pabelanger I think if you compare the wkeely numbers for the last week and the 3 weeks before those jobs should fall off as being that bad23:17
clarkbthe ironic jobs all use neutron too (ironic has never supported nova net)23:17
*** rvasilets___ has quit IRC23:17
openstackgerritMerged openstack-infra/system-config: Add floating-ip cache expiration time settings
jeblairclarkb: i'm not convinced that my methodology is not already excluding those23:19
openstackgerritDoug Hellmann proposed openstack-infra/project-config: fix networking-hyperv release acls
jeblairclarkb: but i don't want to spend any more days on this than i already am23:19
jeblairclarkb: so i will just re-run it for the last week23:19
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Remove support for legacy rh1
clarkbok I think we would probably have noticed with freeze if neutron was failing that hard. We certainly noticed it when ipv6 was broken23:19
jeblairclarkb: most of these are being re-queued by zuul23:20
clarkbyes that was the behavior we saw with ipv6. The job would run then timeout then ansible would rerun it.23:20
clarkbthe impact was in slowing down the throughput of the queues by a lot23:20
jeblairclarkb: these aren't timeouts23:20
pabelangerclarkb: mordred: BTW, I noticed this gem while chasing the ssh-server stop / start issue:
jeblairthey are ssh connection errors23:21
pabelangernot sure now to fix that atm23:21
clarkbjeblair: yes23:21
clarkbjeblair: with ipv6 the existing connections had to timeout then we would get the connection error23:21
clarkbwe saw both things together because the hosts jsut became unroutable23:21
clarkbso tcp thinks its still connected for X amount of time before it gives up and then connection fails23:22
jeblairclarkb: okay, that's similar enough that we'd have to get into the weeds to figure out if i'm picking it up.  so i'll still just do the last week.23:23
*** tpsilva has quit IRC23:24
jeblairso just starting with aug 2623:25
mordredpabelanger: wow23:25
mordredpabelanger: uhm ... what is that?23:25
clarkbpabelanger: mordred looks like a host that has run neutron23:26
jeblairclarkb, pabelanger, mordred:
mordredand that glean just picked up all of the things23:26
clarkband glean is gonna try to configure all those interfaces23:26
mordredand fail23:26
mordredjeblair: oh that's much better23:26
clarkbya thats more like I would expect with multinode being more common for whatever reason23:27
*** annegentle has joined #openstack-infra23:27
jeblairhere's the full list sorted by time:
*** ilyashakhat_mobi has joined #openstack-infra23:29
pabelangermordred: Ya, a neutron job. spotted it in passing23:29
jeblairyou can see them grouping i was talking about23:29
mordredpabelanger: oh - wow - is glean getting triggered by udev/systemd every time neutron creates an interface???23:30
clarkbmordred: yes23:30
pabelangermordred: i think so23:30
mordredthat's amaze23:30
clarkbI think its fine if noisy23:31
mordred clarkb, jeblair: last piece of inefficient nodepool ... it has passed already, which contains the shade fix that enables it (it failed a previous run of that, so the success shows the shade fixed fix it)23:31
jeblair13 jobs failed that way within a second of each other at 21:07 today23:31
mordredjeblair: it's interesting to me that it clusters like that ...23:31
mordredjeblair: but that also makes me think "cloud network hiccup"23:31
jeblairmordred: yes, though it is across all of our launchers, and across clouds23:31
jeblairso it's at least a datacenter-scale hiccup23:32
clarkbone commonality seems to be the job/project?23:32
clarkblike 12:54 today a bunch of multinodes fail23:32
jeblairclarkb: yeah, let me regenerate this with project + change info as well23:32
openstackgerritMerged openstack-infra/shade: Fix up image and flavor by name in create_server
clarkbthere is a clump of cinder fails too23:32
pabelangerclarkb: ubuntu-xenial DIB started23:33
*** chlong has quit IRC23:33
*** ilyashakhat_mobi has quit IRC23:34
*** sflanigan has joined #openstack-infra23:34
ianwhow do we feel about devstack-gate making symlinks in the log directory23:34
*** gongysh has joined #openstack-infra23:35
clarkbianw: is this related to the subunit thing?23:37
*** vhosakot has quit IRC23:37
clarkbianw: its probably fine though we should be careful not to copy them with the logs23:37
clarkber rather we should follow the links when copying23:37
ianwclarkb: yeah, splitting stackviz between "old" and "new" runs in grenade output, but not duplicating the ~3mb of js that's driving it23:38
clarkbianw: I just had to symlink /usr/libexec/qemu-bridge-helper to /usr/lib/qemu-bridge-helper because virsh would not honor the path I set in /etc/libvirt/qemu.conf23:39
clarkbso if symlinks can solve this problem too that seems fine with me23:39
ianwheh, apparently libncurses split itself in two between fedora 23 & 24, so the other day debugging the sfdisk stuff i symlinked half of the new ncurses back to a which had just enough symbols to make it work.  symlinks can do anything :)23:41
pabelangerclarkb: still didn't get my local DIB working, you were able to SSh into your version?23:43
clarkbpabelanger: yup23:43
pabelangerwell, giving up now23:43
pabelangersince nodepool is building it23:43
*** Genek has quit IRC23:43
*** xarses has joined #openstack-infra23:44
mordredclarkb, jeblair: woot. the change finished testing. if you look at you can see the issue that the shade patch we landed fixed. then in the logs for the passing job all is happy23:46
*** zhurong has joined #openstack-infra23:48
jeblairi will be so happy when we have afs on the launchers.  i'm doing all kinds of copying files around between them right now and it's silly23:49
*** thcipriani is now known as thcipriani|afk23:49
mordredjeblair: ++23:50
*** Julien-zte has quit IRC23:50
jeblairi guess i should poke at
*** markvoelker has quit IRC23:51
jeblairclarkb, mordred, pabelanger:
*** dingyichen has joined #openstack-infra23:54
jeblairit looks like the groupings are not simply "all the jobs for a change"23:54
jeblairthat seems pretty spread out too23:54
jeblairare they all bluebox+osic?23:55
*** waht has quit IRC23:56
jeblairno, but mostly.23:56
jeblairmostly osic makes sense.  mostly bluebox does not.23:56
*** markvoelker has joined #openstack-infra23:56
jeblairi wonder if we're looking at a real-time map of internet routing issues :)23:57
mordredjeblair: we tend to expose many issues23:58
mordredjeblair: why not internet routing issues?23:58
*** zhurong has quit IRC23:59
jeblairthe project list is pretty broad too.23:59

Generated by 2.14.0 by Marius Gedminas - find it at!