Thursday, 2016-03-31

jeblairall of the jeblair00:00
clarkbpabelanger: oh did I misundertsand how this works?00:00
clarkbpabelanger: did it start the job after the restart? I may have derped and can stop my process which should remove the delete request00:00
clarkbah yup  Iderped will remove the delete request, it wasnt a stale build it was just a waiting build00:01
pabelangerclarkb: we had a ubuntu-xenail job in the queue, but nodepool-builder didn't know the elements00:01
pabelangerso I think when you restarted, it was happy again00:01
clarkbpabelanger: yup00:01
*** akshai has joined #openstack-infra00:01
clarkband now that my client that is waiting on a delete job to show up is gone the delete job request should be gone too and everything should be happy00:01
*** mylu has quit IRC00:02
clarkbmordred: jeblair where in that stack should a reviewer start?00:02
*** stevemar has quit IRC00:02
mordredclarkb: so ... good question - one sec00:02
mordredclarkb: the shade patches00:03
*** jamesmcarthur has joined #openstack-infra00:03
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Fix on summit event null  https://review.openstack.org/29968400:03
mordredclarkb: actually, the non-shade patches are the next iteration of happy00:03
clarkbmordred: https://review.openstack.org/#/c/299645/ is failing00:04
jeblairso shade patches get us going again; the nodepool (and occ) patches ar for later00:04
openstackgerritMerged openstack-infra/openstackid-resources: Fix on summit event null  https://review.openstack.org/29968400:04
pabelangerclarkb: so, I am hitting: https://github.com/openstack-dev/devstack/blob/master/stack.sh#L18800:04
clarkblooks like jeblairs patches too00:04
jeblairclarkb: i rechecked 645 -- it hit a pathologically slow vexxhost node00:04
clarkbpabelanger: set FORCE=yes in localrc apparently00:04
openstackgerritMonty Taylor proposed openstack-infra/nodepool: Pass extended network information in to occ/shade  https://review.openstack.org/29967900:04
jeblairwas not a legit failure00:04
clarkbpabelanger: you can add declare -x DEVSTACK_LOCAL_CONFIG='FORCE=yes'00:05
clarkbpabelanger: I think that will get you past that00:05
pabelangerok00:05
pabelangertesting00:05
*** Sukhdev has quit IRC00:06
*** fawadkhaliq has quit IRC00:06
*** eil397 has quit IRC00:06
*** mylu has joined #openstack-infra00:07
*** fawadkhaliq has joined #openstack-infra00:07
*** esker has joined #openstack-infra00:08
*** dimtruck is now known as zz_dimtruck00:08
*** salv-orl_ has quit IRC00:09
*** akshai_ has joined #openstack-infra00:09
fungimordred: is there supposed to be a way to get detailed puppet stdout from ansible launch node? here's a snippet of the failure i get http://paste.openstack.org/show/492518/00:09
fungiit's decidedly terse and unhelpful00:10
pabelangerclarkb: better, looks to be running now00:10
clarkbfungi: yes, check syslog on the machine00:10
*** amitgandhinz has joined #openstack-infra00:11
fungiclarkb: followup question... how do i make it not automatically delete the node on failure?00:11
mordredfungi: --keep00:11
clarkbfungi: there is a flag ya that00:11
fungiaha00:11
fungii missed that00:11
fungithanks!00:11
mordredfungi, clarkb: rcarrillocruz has written some good stuff that I think we can use for ansible-launch-node, btw00:11
mordredfungi, clarkb: I'll send some infos about it soon00:11
cody-somerville\o/00:12
*** pvaneck has quit IRC00:12
openstackgerritJames E. Blair proposed openstack-infra/shade: Cache ports like servers  https://review.openstack.org/29967100:12
mordredcody-somerville: yah man. it's good. you should read the blog post00:12
openstackgerritJames E. Blair proposed openstack-infra/shade: Remove conditional blocking on server list  https://review.openstack.org/29967200:12
*** esker has quit IRC00:12
mordredcody-somerville: http://rcarrillocruz.com/deploying-multiple-openstack-clouds-with-ansible-in-a-data-driven-fashion/00:13
* jeblair is preparing to welcome our ansible node launching overlords00:13
*** akshai has quit IRC00:13
*** fawadkhaliq has quit IRC00:13
clarkbmordred: this is what was talked about at the sprint ya?00:13
cody-somervilleSaweet.00:13
*** ZZelle has quit IRC00:14
mordredclarkb: yah00:15
mordredclarkb: I'm going to poke at it in anger in the next couple of days00:15
mordredbut looking at it it seems to meet our needs00:15
jeblairi'm not going to be in the room for that00:15
mordredjeblair: that's what you think00:15
*** esker has joined #openstack-infra00:15
*** markvoelker has joined #openstack-infra00:15
*** amitgandhinz has quit IRC00:16
jeblairclarkb: my shade change passes local unit tests now00:16
*** fawadkhaliq has joined #openstack-infra00:16
mordredclarkb: also, https://review.openstack.org/#/c/298432/00:17
*** nwkarsten has joined #openstack-infra00:17
*** gildub has joined #openstack-infra00:17
*** sridhar_ram has quit IRC00:18
clarkbmordred: ^ should eb safe to merge before any of the othre change smerge ya?00:19
clarkbwill just be ignored?00:19
*** nwkarsten has quit IRC00:21
*** fawadkhaliq has quit IRC00:21
pabelangerHmm, ubuntu-xenial failed to find: openjdk-7-jdk... odd00:22
fungiE: Unable to locate package php5-cli00:22
pabelangercurrently building again00:22
*** markvoelker_ has joined #openstack-infra00:22
clarkbpabelanger: maybe it is on 8?00:22
fungiexcept i was just able to apt-get install it on that machine00:22
*** yamada-h has joined #openstack-infra00:22
clarkbpabelanger: yup00:22
clarkbfor that you will liekly need to edit devstack00:23
clarkbin any case if the fails are in devstack now thats probably a good improvement from our end00:23
*** mriedem has joined #openstack-infra00:23
pabelangerii  openjdk-7-jdk:amd64                   7u95-2.6.4-1                        amd64        OpenJDK Development Kit (JDK)00:23
pabelangerthat is from my local dib00:23
pabelangerso, unless something changes yesterday00:23
*** davideag_ has quit IRC00:23
*** fawadkhaliq has joined #openstack-infra00:23
clarkboh looks like it has both00:24
clarkbodd I dunno then00:24
pabelangerhoping something wrong with mirrir00:24
mordredclarkb: yes00:24
pabelangermirror*00:24
*** annegentle has quit IRC00:24
clarkbmordred: reading jeblairs change it seems to use a hardcoded value not something from oscc?00:24
mordredthe port one?00:25
clarkbI probably need to read all the changes to understand the whole set00:25
clarkbmordred: yes00:25
jeblairit's a default; it gets the value from oscc if its set00:25
mordredwhat jeblair said00:25
*** markvoelker has quit IRC00:25
jeblairclarkb: line 22800:25
mordredclarkb: the port cache is modeled after how the list_servers cache works00:25
mordredso it's doing the same logic, if looking at the other one helps00:25
clarkbhttps://review.openstack.org/#/c/299671/2/shade/openstackcloud.py line 197 why not set it there instead?00:25
jeblairclarkb: that's the dogpile cache path00:26
jeblairwhich we won't hit00:26
clarkbah we are just using a buit in cache thing like nodepool which is the else case?00:27
mordredyes00:27
*** yamada-h has quit IRC00:27
*** _amrith_ is now known as amrith00:28
pabelangerclarkb: Ah, new release today. Likely still syncing to mirrors00:29
*** mtanino has quit IRC00:29
sc`does anyone know what might cause glance to throw a 403 when uploading an image? it's in an integration gate, so i can only really see that glance is throwing the 403, not what it's doing or why00:31
*** jamesmcarthur has quit IRC00:31
sc`it only happens on ubuntu, i should note. centos is fine00:31
clarkbsc`: the gate jobs should have service logs00:32
clarkbincluding those for glance00:32
sc`if they're there, i'm unable to find them. just a console log00:32
clarkbmordred: jeblair how does cache invalidation work here? doesn't look like making a new port updates the cache00:32
clarkbsc`: what job?00:32
clarkbor link to the logs00:32
*** sridhar_ram has joined #openstack-infra00:33
*** Swami has quit IRC00:33
*** Sukhdev has joined #openstack-infra00:33
clarkbsc`: sounds like a misconfigured job if it isn't getting service logs on an integration test00:33
*** Sukhdev has quit IRC00:33
*** Qiming has quit IRC00:34
sc`https://jenkins02.openstack.org/job/gate-ubuntu-trusty-cookbook-openstack-common-chef-rake-integration-nv/9/console is one of the failed builds. it only seems to nab the console log00:34
*** rossella_s has quit IRC00:34
*** sdake has quit IRC00:34
jeblairclarkb, mordred: correct, it does not invalidate the cache.  i wasn't focused on that path since we aren't creating ports.  we could add that; it's essentially an optimization (the cache will timeout).00:35
*** rossella_s has joined #openstack-infra00:35
clarkbyup, so you will wnat to look into how the other integration tests deal with this. Since this is for chef cookbooks I am assuming you don't run the services like devstack and manage the logs directly instead they should log to normal logging locations?00:35
fungii cannot for the life of me figure out why this server won't bootstrap. puppet claims apt-get failed to find the php5-cli and php5-gmp packages, but on that same server i can apt-get install them just dandy00:35
openstackgerritMonty Taylor proposed openstack-infra/nodepool: Start using dogpile caching in devstack tests  https://review.openstack.org/29969000:35
clarkbif that is the case the puppet module folk have a genericish log grabber script that should work if you use semi standard log locations00:36
clarkbjeblair: we do create ports though00:36
pabelangerfungi: I am having issues installing packages too00:36
clarkbjeblair: every time we create a server00:36
openstackgerritMonty Taylor proposed openstack-infra/nodepool: Start using dogpile caching in devstack tests  https://review.openstack.org/29969000:36
pabelangerfungi: think the mirrors are funky atm00:36
sc`fwiw, centos doesn't have service logs either. i suspect an artifact of the recent work to extract logs from a chef-built openstack00:36
*** sdake has joined #openstack-infra00:36
jeblairfungi: any chance it has something to do with our changing the apt repos?00:36
fungipabelanger: i've launched two servers and it complained about the exact same packages each time00:36
sc`so i'll have to dig into that before i can figure out why glance is even erroring out00:36
fungijeblair: i suppose it's possible this is racing that00:36
pabelangerfungi: Ya, I am failing on openjdk-7-jdk currently00:37
jeblairclarkb: we don't *explicitly* create a port, but yes, one (or more) is implicitly created when we create servers.  i added a loop in there so when we're looking for a port to add an ip to a new server, we check at least twice to give the cache a chance to update.00:37
pabelangersc`: Take a look at the work EmilienM did with the puppet integration jobs. They have a good script for collecting logs00:37
clarkbsc`: copy-puppet-logs is the thing puppet uses00:37
clarkbsc`: its mostly not puppet specific though so you can probably adapt it to your jobs00:38
*** Jeffrey4l_ has quit IRC00:38
clarkbfungi: jeblair launch node shouldn't use our apt mirrors though00:38
sc`some stuff has already been merged that's supposed to do that... but clearly it is not :D00:38
clarkbshould use rax's iirc00:38
sc`i'll poke at it some more until my eyes say no more00:38
fungiclarkb: we override that in our manifest00:38
clarkbfungi: oh to use our mirrors?00:38
fungiclarkb: to use us.archive.ubuntu.com and, more importantly, security.ubuntu.com00:39
clarkbsc`: looking at the console log you may just be missing the part of the thing that tells jenkins to copy the files00:39
jeblairand not "give me a kernel from the future repo" :)00:39
*** mylu has quit IRC00:39
fungiclarkb: so it's possible puppet is trying to install packages from rackspace mirrors before we fix our sources00:39
clarkbsc`: the jjb macro for that seems to be 'devstack-logs'00:39
clarkbfungi: that would not surprise me00:40
fungithough the log indicates the package installation is happening after we fix the sources list00:41
sc`i'll take a look at that. see if i can adapt something to our own things. was kind hoping there was a way to avoid the scenic route, but that's fine00:41
*** gildub has quit IRC00:41
*** achanda has quit IRC00:41
clarkbfungi: it may not have updated00:41
*** gildub has joined #openstack-infra00:41
clarkbfungi: er update the apt-get update cahce things00:41
fungiMar 31 00:30:45 openstackid-dev puppet-user[31408]: (/Stage[main]/Openstack_project::Template/File[/etc/apt/sources.list.d/openstack-infra.list]/ensure) defined content as '{md5}0b159a1823cdad18e31cbce892f5d40f'00:42
fungiMar 31 00:30:47 openstackid-dev puppet-user[31408]: (/Stage[main]/Openstackid/Package[php5-gmp]/ensure) change from purged to present failed: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install php5-gmp' returned 100: Reading package lists...00:42
*** nadya has joined #openstack-infra00:42
fungiyeah, maybe it doesn't apt-get update?00:42
pabelangerbasically some thing here with jenkins::slave00:43
pabelangerexcept openjdk-7-jdk00:43
*** annegentle has joined #openstack-infra00:45
*** mylu has joined #openstack-infra00:45
fungipabelanger: clarkb: do we need a before directive or something at http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/template.pp#n19300:46
fungior is there a way to atomically add an exec to those?00:46
*** nadya has quit IRC00:47
clarkbI think you can before => Package meta class thing00:47
*** tiswanso has joined #openstack-infra00:47
*** tiswanso has quit IRC00:47
*** tiswanso has joined #openstack-infra00:48
pabelangerya, I think we are doing apt-get update too late in the process00:48
pabelanger2016-03-30 17:56:08,500 INFO nodepool.image.build.ubuntu-trusty: Notice: /Stage[main]/Apt::Update/Exec[apt_update]: Triggered 'refresh' from 3 events00:48
pabelangerthat is from ubunt-trusty00:48
pabelangerbut way after we install stuff00:49
pabelangerit should be the first thing we do00:49
pabelangerhttp://nodepool.openstack.org/dib.ubuntu-trusty.log00:49
jeblairyeah, this sounds like a whoops that we wouldn't notice except in these edge cases00:49
*** annegentle has quit IRC00:49
sc`ok. i think i see why i don't have service logs00:49
sc`missing devstack-logs in a couple places00:50
sc`it got updated in one place, but not everywhere00:51
sc`however. i do have it for integration, and i don't have service logs00:51
sc`https://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/chef-jobs.yaml seems to confirm it00:52
sc`my integration template has both console-log and devstack-logs as publishers00:52
clarkbmordred: you going to figure out why https://review.openstack.org/#/c/299679/2 exploded?00:52
sc`that was merged back at the beginning of March00:52
openstackgerritMatthew Thode proposed openstack/diskimage-builder: Add Gentoo to the dhcp-all-interfaces element  https://review.openstack.org/29969400:53
pabelangerwish we had a boostrap stage in puppet, then we could do our package manager updates then00:54
clarkbsc`: all that publisher does is copy files if they exist, if they do not exist you don't get any files00:54
prometheanfirenot sure how people will like how I'm doing this, but it's the same method I used in https://github.com/prometheanfire/gentoo-cloud-prep00:54
clarkbsc`: I would double check that you are copying the files to the appropriate location(s)00:54
mordredclarkb: that needs an OCC release00:54
mordredclarkb: it's not urgent for now00:54
prometheanfireI'll wait for ianw to -1 it :P00:54
pabelangerwill have to dive into it tomorrow and figure out how to force apt to run first in main00:54
pabelangerwithout an insane chain00:54
clarkbmordred: that is the fix for tripleo cloud isnt it?00:54
mordredclarkb: it's the _Real_ fix. we have a workaround patch in00:55
clarkbmordred: where is that?00:55
clarkbis that what we are running in production now?00:55
jeblairclarkb: no, production is manually rolled back to pre-shade00:55
openstackgerritMerged openstack-infra/nodepool: Remove unused functions  https://review.openstack.org/29967800:55
sc`clarkb: is $WORKSPACE/logs the appropriate place?00:56
*** esker has quit IRC00:56
jeblairthough we just landed a patch, so it's about to be post-shade again :)00:56
clarkbsc`: yes that looks correct00:56
sc`then yes we should be00:56
clarkbjeblair: woops I approved that patch00:56
clarkbsc`: can you show me where you are copying the logs to that path?00:57
*** gyee has quit IRC00:57
*** esker has joined #openstack-infra00:57
*** stevemar has joined #openstack-infra00:58
jeblairclarkb: np, just something to be aware of while we're in the sketchy on-again-off-again shade zone.  hopefully we can land these updates to shade and restart and stick with it; of course if it drags out, we can just revert shade and try again later00:58
sc`clarkb: https://github.com/openstack/openstack-chef-repo/blob/master/Rakefile#L153-L160 is the meat of it. down at the bottom it issues a _save_logs with an arg of log_dir00:59
sc`so perhaps it's not picking up the env variable properly?00:59
clarkbsc`: or it isn't running when your glance thing fails first01:00
clarkbsc`: its odd to me that the log grabbing isn't the last thing you do01:01
sc`perhaps. _save_logs might never get a chance to fire for ubuntu due to the initial chef run failing, i believe01:01
clarkband I do not know if an exception can trap and bypass that01:01
clarkb_run_nova_tests presumably wants to have its logs recorded too01:01
sc`someone else wrote this. i've been wrapping everything in conditionals to make it work for both centos and ubuntu01:01
*** jamielennox is now known as jamielennox|away01:01
sc`centos works without issue, and ubuntu fails at uploading to glance. the main problem with being able to see is i don't have logs01:02
clarkbyes so fix the logs01:02
clarkbI think your issue is how you are collecting the logs 1) it isn't the last thing you do and 2) it may be skipped on error01:02
*** esker has quit IRC01:02
sc`perhaps so01:02
*** stevemar has quit IRC01:03
*** shashank_hegde has quit IRC01:03
sc`i don't know what was in the author's head at the time, so at best we can speculate01:03
openstackgerritPaul Belanger proposed openstack-infra/system-config: [WIP] Add bootstrap stage to puppet  https://review.openstack.org/29969701:03
clarkbif it were me I would stop reinventing the wheel and just reuse what everything else does01:03
pabelangerfungi: ^ will test that tomorrow, but that should give us the ability to force apt to run first01:04
clarkbpossibly rename copy_puppet_logs to copy_system_logs, add in the chef specific stuff and use that01:04
sc`chef is a different kind of wheel, so... yeah01:04
pabelangerclarkb: devstack-gate still running, will post the results in the morning01:04
clarkbpabelanger: does that mean devstack got past the packge woes?01:04
clarkbpabelanger: is it running tempest?01:05
pabelangerclarkb: yes01:05
sc`i mean, i see your point, but it _is_ a slightly differently shaped wheel01:05
pabelangertempest running.... slowly01:05
clarkbthe xenial dib build appears to be failing in a loop01:05
fungipabelanger: i'm excited, though in the meantime tonight i either need to wait for the mirror situation to clear up or put together an uglier temporary workaround01:05
clarkbsc`: I am not sure why or how it could be so different when openstack is pretty consistent on these platforms01:05
pabelangerclarkb: Yup, that is the openjdk missing package01:05
clarkbsc`: the chef logs itself may be differnetb but nova and keystone and glance are writing to the same files01:05
clarkbpabelanger: ah01:05
pabelangerfungi: right, waiting for the mirror to clean up too01:06
sc`it shouldn't be so vastly different, i'm in agreement01:06
fungii'm booting another right now to see if it's improved01:06
*** tphummel has quit IRC01:06
sc`making integration work shouldn't be a contortion of code01:06
*** mylu has quit IRC01:07
*** mylu has joined #openstack-infra01:08
*** aysyd has quit IRC01:08
*** esker_ has joined #openstack-infra01:10
clarkbianw: re debian are you saying that while they use systemd they have not fully bought into it and are just using pieces of it?01:10
*** jamesmcarthur has joined #openstack-infra01:11
clarkbianw: I think we can worry about htat if/when it becomes a problem but I worry that if debian is doing that then maybe xenial is too?01:11
clarkbpabelanger: ^on your xenial box is journalctl all functional and happy for logging?01:11
*** amitgandhinz has joined #openstack-infra01:11
*** bpokorny_ has joined #openstack-infra01:12
*** Goneri has quit IRC01:12
openstackgerritMerged openstack-infra/shade: Workaround multiple private network ports  https://review.openstack.org/29964501:12
*** apoorvad has quit IRC01:13
pabelangerclarkb: I didn't test your patch. But journalctl is working01:13
clarkbpabelanger: if journalctl works then my patch should be fine01:14
*** fawadkhaliq has quit IRC01:14
clarkbthe owrry is in cases where journalctl is installed but non functional01:14
*** fawadkhaliq has joined #openstack-infra01:14
*** esker_ has quit IRC01:14
*** bpokorny has quit IRC01:15
*** EricGonc_ has joined #openstack-infra01:16
*** bpokorny_ has quit IRC01:16
*** jamesmcarthur has quit IRC01:16
*** amitgandhinz has quit IRC01:17
ianwclarkb: yes, people have worried about systems like that when doing similar stuff in devstack.  personally, i think make a choice and stick with it01:18
*** EricGonczer_ has quit IRC01:19
clarkbya if a system does that then they can figure out how to get their syslog :)01:19
*** resker has joined #openstack-infra01:20
*** fawadkhaliq has quit IRC01:20
*** smarcet has quit IRC01:22
openstackgerritPaul Belanger proposed openstack-infra/system-config: [WIP] Add bootstrap stage to puppet  https://review.openstack.org/29969701:22
sc`interesting. moved _save_logs down in the for loop to the end and it doesn't try to kick off _setup_tempest anymore before trying and failing to scp logs. not sure what to make of that01:22
*** fawadkhaliq has joined #openstack-infra01:23
sc`no, my eyes are crossing. that's the failure from _setup_tempest01:23
openstackgerritMerged openstack-infra/shade: Re-allow list of networks for FIP assignment  https://review.openstack.org/29878901:23
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Set cache parameters like we do in production  https://review.openstack.org/29970201:23
*** jamesmcarthur has joined #openstack-infra01:23
*** fawadkhaliq has quit IRC01:24
*** resker has quit IRC01:24
*** esker has joined #openstack-infra01:24
*** annegentle has joined #openstack-infra01:24
jeblairfungi: what about adding the apt-get update ?01:24
openstackgerritMonty Taylor proposed openstack-infra/nodepool: Start using dogpile caching in devstack tests  https://review.openstack.org/29969001:25
*** sridhar_ram has quit IRC01:25
*** fawadkhaliq has joined #openstack-infra01:25
*** resker has joined #openstack-infra01:26
jeblairclarkb: ^ i set 702 to depend-on my shade port caching patch, so that code path should be executed in 702 (so we can get test results)01:26
fungijeblair: yeah, adding apt-get update will in theory work, i just need to continue figuring out the syntax for making sure that happens before any apt-get install operations (e.g. before any package resources are ensured)01:27
*** fawadkhaliq has quit IRC01:27
jeblairfungi: ok cool.  was worried that idea got abandoned (it sounded good to me)01:27
*** zhurong has joined #openstack-infra01:27
funginot abandoned, just magical puppet fantasy land stuff01:28
fungiso learning in progress01:28
*** esker has quit IRC01:29
*** annegentle has quit IRC01:29
openstackgerritPaul Belanger proposed openstack-infra/system-config: [WIP] Add bootstrap stage to puppet  https://review.openstack.org/29969701:29
fungithe recommended solution does indeed appear to be pabelanger's patch there01:30
fungi(e.g., ensure ordering by using stages)01:30
fungithough there's also this...01:31
fungiExec["apt-update"] -> Package <| |>01:31
openstackgerritMatthew Thode proposed openstack/diskimage-builder: Add Gentoo to the dhcp-all-interfaces element  https://review.openstack.org/29969401:31
*** resker has quit IRC01:31
*** jamesmcarthur has quit IRC01:31
fungiwhich is some bizarre syntax to be sure01:31
pabelangerfungi: right, that could work too01:31
*** sputnik13 has quit IRC01:31
fungiapparently that will cause the exec to fire on every run though01:32
pabelangerokay, calling it a night.  eyes on fire01:32
pabelangerwill hack more on stages tomorrow01:32
funginight pabelanger01:32
*** jamesmcarthur has joined #openstack-infra01:32
fungiand thanks again!01:33
pabelangernp01:33
*** sputnik13 has joined #openstack-infra01:33
*** baoli has joined #openstack-infra01:37
*** esker_ has joined #openstack-infra01:37
*** sputnik13 has quit IRC01:38
*** baoli_ has joined #openstack-infra01:39
*** stevemar has joined #openstack-infra01:39
*** EricGonc_ has quit IRC01:41
*** jamesmcarthur has quit IRC01:41
*** achanda has joined #openstack-infra01:41
*** esker_ has quit IRC01:42
*** baoli_ has quit IRC01:42
*** baoli has quit IRC01:42
*** Daisy has joined #openstack-infra01:43
*** achanda has quit IRC01:46
*** Qiming has joined #openstack-infra01:47
*** esker has joined #openstack-infra01:47
*** kzaitsev_mb has quit IRC01:47
*** jamesmcarthur has joined #openstack-infra01:48
*** sdake_ has joined #openstack-infra01:49
*** resker has joined #openstack-infra01:50
*** sdake has quit IRC01:50
*** nwkarsten has joined #openstack-infra01:50
*** Daisy has quit IRC01:51
*** Daisy has joined #openstack-infra01:51
*** hichihara has joined #openstack-infra01:51
*** esker has quit IRC01:51
*** esker has joined #openstack-infra01:53
*** tqtran has joined #openstack-infra01:53
*** nwkarsten has quit IRC01:55
*** esker_ has joined #openstack-infra01:55
*** resker has quit IRC01:55
*** jamesmcarthur has quit IRC01:55
*** Daisy has quit IRC01:55
*** Daisy has joined #openstack-infra01:55
*** achanda has joined #openstack-infra01:56
*** esker has quit IRC01:57
*** tqtran has quit IRC01:57
*** jamesmcarthur has joined #openstack-infra01:59
*** esker_ has quit IRC01:59
*** Daisy has quit IRC02:00
*** mtanino has joined #openstack-infra02:00
*** Daisy has joined #openstack-infra02:00
*** Daisy has quit IRC02:03
*** Daisy has joined #openstack-infra02:03
*** zz_dimtruck is now known as dimtruck02:04
*** tphummel has joined #openstack-infra02:06
*** SumitNaiksatam has joined #openstack-infra02:09
*** nwkarsten has joined #openstack-infra02:11
*** mylu has quit IRC02:11
sc`with the logs saving at the bottom of the loop, my centos gate works and properly copies logs, but ubuntu does not copy logs upon failure02:11
*** amitgandhinz has joined #openstack-infra02:12
clarkbsc`: it is probably short circuiting when things fail earlier02:12
clarkbI am not super familiar with ruby/rake but you may need a try: finally: type setup02:13
*** thorst has quit IRC02:13
sc`mmm... perhaps. it is tired and i am late, however02:13
*** thorst has joined #openstack-infra02:13
*** jerryz has quit IRC02:14
*** Goneri has joined #openstack-infra02:14
*** esker_ has joined #openstack-infra02:15
*** jamesmcarthur has quit IRC02:16
*** amitgandhinz has quit IRC02:17
*** Jeffrey4l has joined #openstack-infra02:17
*** yolanda has quit IRC02:18
*** jamesmcarthur has joined #openstack-infra02:18
*** yamamoto_ has joined #openstack-infra02:19
*** esker_ has quit IRC02:20
anteayayonglihe: hi02:20
yongliheanteaya:  hi,02:21
*** nwkarsten has quit IRC02:21
*** shashank_hegde has joined #openstack-infra02:21
*** salv-orlando has joined #openstack-infra02:21
anteayayonglihe: do you know if any of teh intel ci accounts comment on nova?02:22
*** nwkarsten has joined #openstack-infra02:22
*** thorst has quit IRC02:22
anteayayonglihe: I saw your reply to the mailing list thread02:23
*** Goneri has quit IRC02:23
anteayayonglihe: the nova ptl is trying to understand if any of the intel ci accounts are testing nova02:23
anteayayonglihe: who would know the answer to that question?02:23
yonglihei talked to him,  and i know the SRIOV/PCI02:24
yongliheand we escalate the question internally, so we get the answer quickly i think.02:24
anteayayonglihe: wonderful, thank you02:25
yonglihewelcome, anteaya02:25
*** mriedem has quit IRC02:25
*** xyang1 has quit IRC02:26
*** nwkarsten has quit IRC02:26
*** esker has joined #openstack-infra02:27
*** resker has joined #openstack-infra02:29
*** markvoelker_ has quit IRC02:31
*** salv-orlando has quit IRC02:32
*** esker has quit IRC02:32
*** markvoelker_ has joined #openstack-infra02:32
*** esker has joined #openstack-infra02:32
*** bpokorny has joined #openstack-infra02:32
*** jamesmcarthur has quit IRC02:34
*** resker has quit IRC02:34
*** markvoel_ has joined #openstack-infra02:36
*** mylu has joined #openstack-infra02:36
*** asselin has joined #openstack-infra02:36
*** salv-orlando has joined #openstack-infra02:36
*** esker has quit IRC02:37
*** jamesmcarthur has joined #openstack-infra02:38
*** asselin__ has quit IRC02:38
*** yamahata has quit IRC02:38
*** markvoelker_ has quit IRC02:40
*** resker has joined #openstack-infra02:41
*** jamesmcarthur_ has joined #openstack-infra02:43
*** kzaitsev_mb has joined #openstack-infra02:43
*** jamesmcarthur has quit IRC02:43
*** jamesmcarthur_ is now known as jamesmcarthur02:43
*** mylu has quit IRC02:44
*** mylu has joined #openstack-infra02:44
*** esker___ has joined #openstack-infra02:45
*** resker has quit IRC02:46
*** resker has joined #openstack-infra02:47
*** kzaitsev_mb has quit IRC02:48
*** mylu_ has joined #openstack-infra02:48
*** mylu has quit IRC02:49
*** esker___ has quit IRC02:50
*** resker has quit IRC02:51
*** jamesmcarthur has quit IRC02:52
*** salv-orlando has quit IRC02:52
*** amrith is now known as _amrith_02:53
*** smarcet has joined #openstack-infra02:53
*** tqtran has joined #openstack-infra02:54
*** esker_ has joined #openstack-infra02:54
*** esker__ has joined #openstack-infra02:56
*** dingyichen has joined #openstack-infra02:57
*** tqtran has quit IRC02:58
*** resker has joined #openstack-infra02:59
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Fix on schedule null events  https://review.openstack.org/29972502:59
*** esker_ has quit IRC03:00
openstackgerritMerged openstack-infra/openstackid-resources: Fix on schedule null events  https://review.openstack.org/29972503:00
*** esker__ has quit IRC03:00
*** markvoelker has joined #openstack-infra03:01
*** bpokorny has quit IRC03:01
*** esker has joined #openstack-infra03:02
*** fbo has quit IRC03:02
*** achanda has quit IRC03:03
*** markvoelker_ has joined #openstack-infra03:04
*** resker has quit IRC03:04
*** markvoel_ has quit IRC03:04
*** resker has joined #openstack-infra03:05
*** fbo has joined #openstack-infra03:06
*** esker has quit IRC03:06
*** hichihara has quit IRC03:06
*** sdake has joined #openstack-infra03:07
*** fawadkhaliq has joined #openstack-infra03:08
*** markvoelker has quit IRC03:08
*** sdake_ has quit IRC03:10
*** resker has quit IRC03:10
*** apoorvad has joined #openstack-infra03:11
*** amitgandhinz has joined #openstack-infra03:13
*** kzaitsev_mb has joined #openstack-infra03:13
*** esker has joined #openstack-infra03:14
*** amitgandhinz has quit IRC03:18
*** kzaitsev_mb has quit IRC03:18
*** esker has quit IRC03:18
*** apoorvad has quit IRC03:19
*** thorst has joined #openstack-infra03:19
fungismarcet: since it looks like you're still awake, the new openstackid-dev running 14.04 is showing some strange output at https://openstackid-dev.openstack.org/03:20
*** yuanying has quit IRC03:21
fungismarcet: like the proxy isn't happening or something03:21
fungii'm heading to sleep shortly, but can look at it more tomorrow03:21
smarcetfungi: yes looking at it :) tnx03:22
smarcetfungi: have a good nite !03:22
fungiyou too, thanks!03:22
*** sputnik13 has joined #openstack-infra03:22
*** pkoniszewski has quit IRC03:24
*** yamada-h has joined #openstack-infra03:24
*** simonmcc_ has joined #openstack-infra03:25
*** fawadkhaliq has quit IRC03:26
*** bryan_att_ has joined #openstack-infra03:26
*** kshort_away has joined #openstack-infra03:26
*** mgkwill_ has joined #openstack-infra03:26
*** Daviey_ has joined #openstack-infra03:26
*** fawadkhaliq has joined #openstack-infra03:26
*** warpc_ has joined #openstack-infra03:26
*** ajo_ has joined #openstack-infra03:27
*** jeblair_ has joined #openstack-infra03:27
*** thorst has quit IRC03:27
*** waynr1 has joined #openstack-infra03:27
*** cbader_ has joined #openstack-infra03:28
*** aspiers_ has joined #openstack-infra03:28
*** jraim_ has joined #openstack-infra03:29
*** yamada-h has quit IRC03:29
*** xnox_ has joined #openstack-infra03:29
*** arif-ali_ has joined #openstack-infra03:30
*** krotscheck has joined #openstack-infra03:30
*** wfoster_ has joined #openstack-infra03:30
*** zaro has quit IRC03:30
*** zxiiro_ has joined #openstack-infra03:30
*** hamzy_ has joined #openstack-infra03:31
*** mwhahaha_ has joined #openstack-infra03:31
*** ogelbukh_ has joined #openstack-infra03:31
*** dmellado_ has joined #openstack-infra03:31
*** scottda_ has joined #openstack-infra03:31
*** pkoniszewski has joined #openstack-infra03:31
*** zaro has joined #openstack-infra03:31
*** alkhodos has quit IRC03:31
*** fawadkhaliq has quit IRC03:32
*** rpodolyaka_ has joined #openstack-infra03:32
*** fawadkhaliq has joined #openstack-infra03:32
*** simonmcc has quit IRC03:32
*** bryan_att has quit IRC03:32
*** aspiers has quit IRC03:32
*** mgkwill has quit IRC03:32
*** kevinshort has quit IRC03:32
*** kshort_away is now known as kevinshort03:32
*** Daviey has quit IRC03:32
*** warpc has quit IRC03:32
*** xnox has quit IRC03:32
*** dougwig has quit IRC03:32
*** jeblair has quit IRC03:32
*** ajo has quit IRC03:32
*** fkautz has quit IRC03:32
*** wfoster has quit IRC03:32
*** madorn has quit IRC03:32
*** waynr has quit IRC03:32
*** cbader has quit IRC03:32
*** ogelbukh has quit IRC03:32
*** asettle has quit IRC03:32
*** jraim has quit IRC03:32
*** mwhahaha has quit IRC03:32
*** hamzy has quit IRC03:32
*** xnox_ is now known as xnox03:32
*** zxiiro has quit IRC03:32
*** krotscheck_dcm has quit IRC03:32
*** scottda has quit IRC03:32
*** dmellado has quit IRC03:32
*** arif-ali has quit IRC03:32
*** andymccr has quit IRC03:32
*** cargonza has quit IRC03:32
*** arif-ali_ is now known as arif-ali03:32
*** onovy_ has joined #openstack-infra03:33
*** asettle has joined #openstack-infra03:33
*** aorourke_ has joined #openstack-infra03:33
*** warpc_ is now known as warpc03:33
*** peterstac_ has joined #openstack-infra03:33
*** ogelbukh_ is now known as ogelbukh03:33
*** mgkwill_ is now known as mgkwill03:33
*** peterstac has quit IRC03:33
*** fmccrthy has quit IRC03:33
*** rvba has quit IRC03:33
*** rpodolyaka has quit IRC03:33
*** andrewbogott has quit IRC03:33
*** serverascode has quit IRC03:33
*** onovy has quit IRC03:33
*** maximov_ has quit IRC03:33
*** sshnaidm has quit IRC03:33
*** lindsayk has quit IRC03:33
*** electrical has quit IRC03:33
*** mancdaz has quit IRC03:33
*** aorourke has quit IRC03:33
*** adreznec has quit IRC03:33
*** fbo has quit IRC03:33
*** Adri2000 has quit IRC03:33
*** Hunner has quit IRC03:33
*** mancdaz has joined #openstack-infra03:33
*** onovy_ is now known as onovy03:33
*** fbo_ has joined #openstack-infra03:33
*** peterstac_ is now known as peterstac03:33
*** simonmcc_ is now known as simonmcc03:33
*** bryan_att_ is now known as bryan_att03:33
*** Adri2000 has joined #openstack-infra03:33
*** Adri2000 has quit IRC03:33
*** Adri2000 has joined #openstack-infra03:33
*** andymccr has joined #openstack-infra03:33
*** madorn has joined #openstack-infra03:33
*** rpodolyaka_ is now known as rpodolyaka03:34
*** rvba has joined #openstack-infra03:34
*** adreznec has joined #openstack-infra03:34
*** fmccrthy has joined #openstack-infra03:34
*** cargonza has joined #openstack-infra03:34
*** jraim_ is now known as jraim03:34
*** rvba has quit IRC03:34
*** rvba has joined #openstack-infra03:34
*** Jeffrey4l_ has joined #openstack-infra03:34
*** waynr1 has quit IRC03:34
*** mwhahaha_ is now known as mwhahaha03:34
*** waynr has joined #openstack-infra03:34
*** scottda_ is now known as scottda03:34
*** sshnaidm has joined #openstack-infra03:34
*** zxiiro_ is now known as zxiiro03:35
*** Hunner has joined #openstack-infra03:36
*** Hunner has quit IRC03:36
*** Hunner has joined #openstack-infra03:36
*** esker has joined #openstack-infra03:36
*** maximov has joined #openstack-infra03:36
*** Jeffrey4l has quit IRC03:37
*** electrical has joined #openstack-infra03:37
*** andrewbogott has joined #openstack-infra03:38
*** serverascode has joined #openstack-infra03:38
*** fkautz has joined #openstack-infra03:39
*** dougwig has joined #openstack-infra03:39
openstackgerritsebastian marcet proposed openstack-infra/puppet-openstackid: Fix Vhost (Apache2.4)  https://review.openstack.org/29973203:40
*** esker has quit IRC03:40
*** bpokorny has joined #openstack-infra03:40
*** esker has joined #openstack-infra03:40
smarcetfungi: if u are still awake https://review.openstack.org/#/c/299732/1 fix the issue03:41
smarcetfungi: nite!03:41
*** nwkarsten has joined #openstack-infra03:42
*** nwkarsten has quit IRC03:43
*** smarcet has quit IRC03:43
*** nwkarsten has joined #openstack-infra03:43
*** links has joined #openstack-infra03:44
*** esker has quit IRC03:45
*** nwkarsten has quit IRC03:48
*** ramishra has quit IRC03:49
*** esker has joined #openstack-infra03:50
*** ramishra has joined #openstack-infra03:51
*** salv-orlando has joined #openstack-infra03:52
*** salv-orlando has quit IRC03:55
*** esker has quit IRC03:55
openstackgerritsebastian marcet proposed openstack-infra/puppet-openstackid: Fix Vhost (Apache2.4)  https://review.openstack.org/29973203:55
*** smarcet has joined #openstack-infra03:57
*** smarcet has quit IRC03:57
*** mylu_ has quit IRC03:58
*** fawadkhaliq has quit IRC03:58
*** yamamoto_ has quit IRC03:59
*** fawadkhaliq has joined #openstack-infra03:59
*** sputnik13_ has joined #openstack-infra04:00
*** fawadkhaliq has quit IRC04:00
*** fawadkhaliq has joined #openstack-infra04:01
*** fawadkhaliq has quit IRC04:02
*** fawadkhaliq has joined #openstack-infra04:02
*** achanda has joined #openstack-infra04:04
*** yuanying has joined #openstack-infra04:07
*** bpokorny has quit IRC04:08
*** achanda has quit IRC04:09
*** achanda has joined #openstack-infra04:09
*** baoli has joined #openstack-infra04:10
*** eil397 has joined #openstack-infra04:13
*** amitgandhinz has joined #openstack-infra04:14
*** kzaitsev_mb has joined #openstack-infra04:14
*** baoli has quit IRC04:14
*** salv-orlando has joined #openstack-infra04:16
*** mylu has joined #openstack-infra04:17
*** amitgandhinz has quit IRC04:18
*** eil397 has quit IRC04:19
*** kzaitsev_mb has quit IRC04:21
*** markvoelker_ has quit IRC04:21
*** harlowja_at_home has joined #openstack-infra04:23
prometheanfireianw: before I was using diskimage-builder to build gentoo images it's what I used for network setup for openstack guests04:23
prometheanfireif that makes you feel better :P04:23
*** salv-orlando has quit IRC04:23
*** bpokorny has joined #openstack-infra04:24
*** sridhar_ram has joined #openstack-infra04:25
*** sdake_ has joined #openstack-infra04:28
*** sridhar_ram1 has joined #openstack-infra04:28
*** fawadkhaliq has quit IRC04:29
*** fawadkhaliq has joined #openstack-infra04:30
*** sdake has quit IRC04:30
*** esker has joined #openstack-infra04:30
*** sridhar_ram has quit IRC04:30
*** baoli has joined #openstack-infra04:31
*** dimtruck is now known as zz_dimtruck04:32
*** mylu has quit IRC04:34
*** Jeffrey4l_ has quit IRC04:34
*** esker has quit IRC04:35
*** fedexo has joined #openstack-infra04:36
*** baoli has quit IRC04:36
*** maishsk has quit IRC04:38
*** yamada-h has joined #openstack-infra04:39
*** yamamoto_ has joined #openstack-infra04:39
*** camunoz has joined #openstack-infra04:40
ianwprometheanfire: i believe you04:40
*** mylu has joined #openstack-infra04:42
*** nadya has joined #openstack-infra04:43
prometheanfireif it helps even more it's listed here https://github.com/prometheanfire/gentoo-cloud-prep/blob/master/02-catalyst-that-shit.sh#L7004:47
*** bpokorny has quit IRC04:47
prometheanfireand here https://github.com/prometheanfire/gentoo-cloud-prep/blob/master/files/prep.sh#L32-L3504:47
*** IlyaG has quit IRC04:47
*** yamahata has joined #openstack-infra04:48
openstackgerritMerged openstack-infra/project-config: midonet: Make tempest jobs voting  https://review.openstack.org/29814204:56
*** abregman has joined #openstack-infra04:58
*** rcernin has joined #openstack-infra04:59
*** harlowja_at_home has quit IRC05:05
*** lindsayk has joined #openstack-infra05:05
*** yamada-h has quit IRC05:08
*** yamada-h has joined #openstack-infra05:09
*** nikhil has quit IRC05:13
*** yamada-h has quit IRC05:13
*** sputnik13_ has quit IRC05:13
*** nikhil has joined #openstack-infra05:14
*** sdake_ is now known as sdake05:14
*** jaosorior has joined #openstack-infra05:14
*** amitgandhinz has joined #openstack-infra05:14
*** fawadkhaliq has quit IRC05:15
*** fawadkhaliq has joined #openstack-infra05:16
*** kzaitsev_mb has joined #openstack-infra05:18
*** amitgandhinz has quit IRC05:19
*** nadya has quit IRC05:20
*** salv-orlando has joined #openstack-infra05:23
*** kzaitsev_mb has quit IRC05:23
sc`clarkb: thanks for reminding me about begin / rescue / end. finally got some service logs. if only the error i finally have wasn't so nonspecific :)05:23
*** jerryz has joined #openstack-infra05:25
*** sridhar_ram1 has quit IRC05:26
*** Sukhdev has joined #openstack-infra05:26
*** salv-orlando has quit IRC05:28
*** sputnik13_ has joined #openstack-infra05:29
*** annegentle has joined #openstack-infra05:29
*** mtanino has quit IRC05:31
*** sputnik13_ has quit IRC05:32
*** baoli has joined #openstack-infra05:32
*** thorst has joined #openstack-infra05:32
openstackgerritMerged openstack-infra/tripleo-ci: Use Fedora 23 atomic in container gate  https://review.openstack.org/29310805:33
*** esker has joined #openstack-infra05:33
*** sputnik13_ has joined #openstack-infra05:33
*** annegentle has quit IRC05:34
*** baoli has quit IRC05:37
*** thorst has quit IRC05:37
*** fawadkhaliq has quit IRC05:37
*** yamada-h has joined #openstack-infra05:39
*** Jeffrey4l_ has joined #openstack-infra05:39
*** mylu has quit IRC05:39
*** fawadkhaliq has joined #openstack-infra05:39
*** sputnik13_ has quit IRC05:40
*** esker has quit IRC05:42
*** esikache1 has joined #openstack-infra05:46
*** pcaruana has quit IRC05:49
*** alexey_weyl has joined #openstack-infra05:50
alexey_weylHello,05:50
*** mtanino has joined #openstack-infra05:50
*** esikache1 has quit IRC05:50
alexey_weylI have an issue with jenkins in gerrit.05:50
alexey_weylI have unskipped my test in the Vitrage project, and it passes tox, but zuul returns an error for py27.05:51
alexey_weylThe error message doesn't have any useful information about the error, and I would like to ask you what can I do about it?05:52
alexey_weylYou can find it here:05:52
alexey_weylhttps://review.openstack.org/#/c/299408/05:52
alexey_weylhttp://logs.openstack.org/08/299408/1/check/gate-vitrage-python27/97c7a1a/05:52
*** oanson has joined #openstack-infra05:53
alexey_weylp.s. I am starting another thread in test, and I think that what causes it, but still my tox passes the test and it looks fine.05:53
*** Jeffrey4l_ has quit IRC05:54
*** k4n0 has joined #openstack-infra05:54
*** k4n0 has quit IRC05:54
*** e0ne has joined #openstack-infra05:56
*** ihrachys has joined #openstack-infra05:57
*** Sukhdev has quit IRC05:57
*** flepied has joined #openstack-infra05:58
*** Swami has joined #openstack-infra05:59
*** fawadkhaliq has quit IRC05:59
*** fawadkhaliq has joined #openstack-infra06:00
*** torandu has quit IRC06:02
*** mtanino has quit IRC06:03
*** aeng has quit IRC06:03
*** torandu has joined #openstack-infra06:04
*** fawadkhaliq has quit IRC06:04
*** ramishra has quit IRC06:05
*** binbincong has quit IRC06:06
*** ramishra has joined #openstack-infra06:07
*** esikache1 has joined #openstack-infra06:07
AJaegeralexey_weyl: "Build timed out (after 50 minutes). Marking the build as failed."06:07
*** fawadkhaliq has joined #openstack-infra06:08
AJaegerthat can be a problem of the node or some indeterminism in your tests/repo06:08
*** roxanaghe has joined #openstack-infra06:08
*** nadya has joined #openstack-infra06:08
*** ihrachys has quit IRC06:11
*** amitgandhinz has joined #openstack-infra06:15
*** woodster_ has joined #openstack-infra06:18
*** fawadkhaliq has quit IRC06:19
*** kzaitsev_mb has joined #openstack-infra06:19
*** fawadkhaliq has joined #openstack-infra06:19
*** roxanaghe has quit IRC06:19
*** binbincong has joined #openstack-infra06:19
*** amitgandhinz has quit IRC06:20
*** aeng has joined #openstack-infra06:21
*** mikelk has joined #openstack-infra06:21
*** daemontool has joined #openstack-infra06:22
*** yamahata has quit IRC06:23
*** mohankumar has joined #openstack-infra06:23
*** e0ne has quit IRC06:24
*** Swami has quit IRC06:24
*** kzaitsev_mb has quit IRC06:25
openstackgerritSachi King proposed openstack-dev/pbr: Sort 'pbr freeze' output  https://review.openstack.org/29976606:26
AJaegerNakato: nice! ^ Thanks!06:28
*** tphummel has quit IRC06:28
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/29976906:28
*** nadya has quit IRC06:28
*** asettle has quit IRC06:28
*** mikelk has quit IRC06:29
*** annegentle has joined #openstack-infra06:30
alexey_weylAJaeger: What kind of indeterminsm? Tox passed this test every time for 20-30 times.06:31
alexey_weylThe only thing I can think of is that the thread that I am opening is not being started, which can cause the indeterminsm. What can I do to check if the thread is running?06:32
*** fedexo has quit IRC06:32
openstackgerritMehdi Abaakouk (sileht) proposed openstack/requirements: Bump gnocchiclient version to 2.2.0  https://review.openstack.org/29977006:33
*** nadya has joined #openstack-infra06:33
*** esikache1 has quit IRC06:33
*** thorst has joined #openstack-infra06:34
*** mikelk has joined #openstack-infra06:34
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints  https://review.openstack.org/28884906:35
*** annegentle has quit IRC06:35
*** nadya has quit IRC06:35
*** nadya has joined #openstack-infra06:35
NakatoAJaeger: Thanks. :)06:36
*** sdake_ has joined #openstack-infra06:37
*** sdake has quit IRC06:40
*** thorst has quit IRC06:42
*** kzaitsev_mb has joined #openstack-infra06:43
*** sdake has joined #openstack-infra06:45
*** yamada-h_ has joined #openstack-infra06:45
*** abregman_ has joined #openstack-infra06:45
*** aeng has quit IRC06:46
AJaegeralexey_weyl: Just looking at the timeout, there are these two explanations for me - I didn't check code.06:46
*** abregman has quit IRC06:46
AJaegeralexey_weyl: if it's reproduceable in the gate, one way is to hold a node and access it. We can also check which cloud this runs in and see whether there is a corrolation if that happens more often...06:48
*** salv-orlando has joined #openstack-infra06:48
*** sdake_ has quit IRC06:48
*** yamada-h has quit IRC06:48
alexey_weylWhat I am trying to understand is how can I check or print log in the ZUUL \ Jenkins environment06:48
*** fawadkhaliq has quit IRC06:48
alexey_weylYes, it occurs all the time in the gate, and never in my devstack environment06:48
*** fawadkhaliq has joined #openstack-infra06:49
*** tesseract has joined #openstack-infra06:49
AJaegeralexey_weyl: wait for yolanda to show up - or later another infra-root (during US time) and ask them to hold a node for you. Then you can ssh into such a node and investigate...06:49
*** tesseract is now known as Guest1103706:49
AJaegerBut perhaps somebody else has a better idea, you might want to ask in #openstack-qa as well.06:50
*** baoli has joined #openstack-infra06:50
alexey_weylsound great! Thank you very much06:50
*** salv-orlando has quit IRC06:51
openstackgerritShamail Tahir proposed openstack-infra/irc-meetings: Adding AsiaPac Regional PWG Meeting  https://review.openstack.org/29977606:51
*** esikache1 has joined #openstack-infra06:52
*** shashank_hegde has quit IRC06:53
*** alexey_weyl has quit IRC06:54
*** baoli has quit IRC06:55
openstackgerritMerged openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/29976906:56
*** esikache1 has quit IRC06:56
*** ihrachys has joined #openstack-infra06:58
*** binbincong has quit IRC07:03
openstackgerritVasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW  https://review.openstack.org/29927307:03
*** daemontool has quit IRC07:04
*** abregman_ is now known as abregman07:05
*** torandu has quit IRC07:06
*** Daisy_ has joined #openstack-infra07:07
*** yamada-h_ has quit IRC07:08
*** yamada-h has joined #openstack-infra07:09
*** akaszuba has joined #openstack-infra07:10
*** Daisy has quit IRC07:10
*** |-paul-| has joined #openstack-infra07:10
*** maishsk has joined #openstack-infra07:11
*** Daisy has joined #openstack-infra07:11
*** yamada-h has quit IRC07:11
*** Daisy__ has joined #openstack-infra07:12
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Fix image caching logic  https://review.openstack.org/29953707:12
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Pre install packages on the instack image  https://review.openstack.org/29953807:12
*** Daisy___ has joined #openstack-infra07:12
*** zeih has joined #openstack-infra07:13
*** Daisy_ has quit IRC07:14
*** torandu has joined #openstack-infra07:14
*** e0ne has joined #openstack-infra07:15
*** Daisy has quit IRC07:16
*** Daisy___ has quit IRC07:16
*** amitgandhinz has joined #openstack-infra07:16
*** Daisy__ has quit IRC07:16
*** Daisy has joined #openstack-infra07:16
*** mrmartin has joined #openstack-infra07:16
*** binbincong has joined #openstack-infra07:17
*** Daisy_ has joined #openstack-infra07:18
*** e0ne has quit IRC07:18
*** Daisy has quit IRC07:18
*** Daisy has joined #openstack-infra07:20
*** amitgandhinz has quit IRC07:20
*** esikache1 has joined #openstack-infra07:21
*** fhubik has joined #openstack-infra07:22
*** esikache1 has quit IRC07:23
*** esikachev has joined #openstack-infra07:23
*** Daisy_ has quit IRC07:23
*** nmagnezi has joined #openstack-infra07:24
*** Daisy has quit IRC07:24
*** jlanoux has joined #openstack-infra07:26
*** jlanoux_ has joined #openstack-infra07:28
*** jlanoux has quit IRC07:28
*** arxcruz has joined #openstack-infra07:30
*** annegentle has joined #openstack-infra07:31
*** mrmartin has quit IRC07:32
*** stevemar has quit IRC07:33
*** annegentle has quit IRC07:35
*** kzaitsev_mb has quit IRC07:38
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: Run tempest tests with non-HA jobs  https://review.openstack.org/29703807:39
*** jtomasek has joined #openstack-infra07:40
*** thorst has joined #openstack-infra07:41
*** pcaruana has joined #openstack-infra07:41
*** thorst has quit IRC07:47
*** permalac has joined #openstack-infra07:48
*** shardy has joined #openstack-infra07:48
*** armax_ has joined #openstack-infra07:49
*** hashar has joined #openstack-infra07:49
*** jerryz has quit IRC07:51
*** annegentle has joined #openstack-infra07:52
*** armax has quit IRC07:52
*** armax_ is now known as armax07:52
*** ihrachys has quit IRC07:53
*** jistr has joined #openstack-infra07:53
*** yaume has joined #openstack-infra07:53
*** matrohon has joined #openstack-infra07:54
*** dizquierdo has joined #openstack-infra07:56
*** annegentle has quit IRC07:56
*** sflanigan has quit IRC07:57
*** fawadkhaliq has quit IRC07:59
*** jordanP has joined #openstack-infra08:01
*** fawadkhaliq has joined #openstack-infra08:01
openstackgerritLuigi Toscano proposed openstack-infra/project-config: WIP Revert "[sahara-tests] create and merge, disable tests"  https://review.openstack.org/29965208:03
openstackgerritLuigi Toscano proposed openstack-infra/project-config: [sahara-tests] create and merge, disable tests  https://review.openstack.org/29940708:03
*** fawadkhaliq has quit IRC08:03
*** fawadkhaliq has joined #openstack-infra08:04
*** permalac has quit IRC08:06
*** salv-orlando has joined #openstack-infra08:07
*** sshnaidm has quit IRC08:09
*** salv-orlando has quit IRC08:09
*** zhurong has quit IRC08:11
*** zhurong has joined #openstack-infra08:11
*** briancli1e has quit IRC08:12
*** kzaitsev_mb has joined #openstack-infra08:13
*** briancline has joined #openstack-infra08:13
*** Daisy has joined #openstack-infra08:14
*** Daisy has quit IRC08:15
*** jcoufal has joined #openstack-infra08:15
*** Daisy has joined #openstack-infra08:15
*** sdake_ has joined #openstack-infra08:17
*** amitgandhinz has joined #openstack-infra08:17
*** ifarkas has joined #openstack-infra08:17
*** salv-orlando has joined #openstack-infra08:18
*** sdake has quit IRC08:19
*** maishsk has quit IRC08:20
*** amitgandhinz has quit IRC08:21
*** sdake has joined #openstack-infra08:22
*** salv-orlando has quit IRC08:22
*** sdake_ has quit IRC08:24
*** esikachev has quit IRC08:25
*** permalac has joined #openstack-infra08:25
*** woodster_ has quit IRC08:27
*** Daisy_ has joined #openstack-infra08:27
*** e0ne has joined #openstack-infra08:28
*** e0ne has quit IRC08:29
*** ihrachys has joined #openstack-infra08:30
*** Daisy has quit IRC08:30
*** achanda has quit IRC08:30
*** Daisy_ has quit IRC08:31
*** sfinucan has joined #openstack-infra08:33
*** stevemar has joined #openstack-infra08:33
*** achanda has joined #openstack-infra08:34
*** rossella_s has quit IRC08:34
*** e0ne has joined #openstack-infra08:34
*** rossella_s has joined #openstack-infra08:35
*** e0ne has quit IRC08:35
*** akshai_ has quit IRC08:37
*** cody-somerville has quit IRC08:37
*** fawadkhaliq has quit IRC08:37
*** stevemar has quit IRC08:39
*** fawadkhaliq has joined #openstack-infra08:40
openstackgerritVincent Fran├žoise proposed openstack-infra/project-config: Watcher multinode - disable plugin in subnode  https://review.openstack.org/29980908:41
*** daemontool has joined #openstack-infra08:41
*** esikachev has joined #openstack-infra08:42
*** gildub has quit IRC08:43
*** kushal has quit IRC08:44
*** yamada-h has joined #openstack-infra08:44
*** thorst has joined #openstack-infra08:45
*** daemontool has quit IRC08:48
*** achanda has quit IRC08:48
*** achanda has joined #openstack-infra08:49
*** yamada-h has quit IRC08:50
*** baoli has joined #openstack-infra08:51
*** zeih has quit IRC08:51
*** thorst has quit IRC08:52
*** cody-somerville has joined #openstack-infra08:52
*** achanda has quit IRC08:52
*** baoli has quit IRC08:55
*** daemontool has joined #openstack-infra08:58
*** achanda has joined #openstack-infra08:59
*** dizquierdo has quit IRC09:00
*** abregman has quit IRC09:00
*** Daisy has joined #openstack-infra09:00
*** Daisy_ has joined #openstack-infra09:01
*** Daisy_ has quit IRC09:03
*** fawadkhaliq has quit IRC09:05
*** sshnaidm has joined #openstack-infra09:05
*** Daisy has quit IRC09:06
*** torandu has quit IRC09:06
*** torandu has joined #openstack-infra09:07
*** rlandy has quit IRC09:09
*** dtantsur|afk is now known as dtantsur09:09
*** fawadkhaliq has joined #openstack-infra09:09
*** asettle has joined #openstack-infra09:10
*** asettle has quit IRC09:10
*** Daisy has joined #openstack-infra09:10
*** ihrachys has quit IRC09:11
*** Daisy has quit IRC09:11
*** dtardivel has joined #openstack-infra09:14
*** fhorvath has joined #openstack-infra09:14
*** esikachev has quit IRC09:15
*** abregman has joined #openstack-infra09:15
*** zeih has joined #openstack-infra09:16
*** [1]Thelo has joined #openstack-infra09:16
*** amitgandhinz has joined #openstack-infra09:17
*** mrmartin has joined #openstack-infra09:18
*** derekh has joined #openstack-infra09:19
*** eliqiao has quit IRC09:19
*** Thelo has quit IRC09:19
*** [1]Thelo is now known as Thelo09:19
openstackgerritAnton Arefiev proposed openstack-infra/project-config: Add Ironic Inspector basic tempest job  https://review.openstack.org/29982109:22
*** amitgandhinz has quit IRC09:22
*** salv-orlando has joined #openstack-infra09:23
*** esikachev has joined #openstack-infra09:24
openstackgerritRossella Sblendido proposed openstack-infra/reviewday: Add a link for Neutron Gerrit dashboard  https://review.openstack.org/29877909:25
*** fawadkhaliq has quit IRC09:25
*** placko_ has joined #openstack-infra09:25
*** electrofelix has joined #openstack-infra09:26
*** achanda has quit IRC09:26
*** Jeffrey4l_ has joined #openstack-infra09:26
openstackgerritAparna proposed openstack/diskimage-builder: Support to add self-signed certificate in ironic-agent  https://review.openstack.org/29520309:26
*** fawadkhaliq has joined #openstack-infra09:26
*** Daisy has joined #openstack-infra09:29
Zaraanteaya: I'm around now09:30
*** e0ne has joined #openstack-infra09:30
*** sdake_ has joined #openstack-infra09:32
*** sdake has quit IRC09:32
*** sdake has joined #openstack-infra09:33
*** skolekonov|away is now known as skolekonov09:34
*** exploreshaifali has joined #openstack-infra09:36
*** mrmartin has quit IRC09:36
*** sdake_ has quit IRC09:36
*** asettle has joined #openstack-infra09:37
*** e0ne has quit IRC09:37
*** salv-orlando has quit IRC09:39
*** Daisy has quit IRC09:40
*** Daisy has joined #openstack-infra09:40
*** fawadkhaliq has quit IRC09:41
*** e0ne has joined #openstack-infra09:42
*** fawadkhaliq has joined #openstack-infra09:42
*** daemontool has quit IRC09:43
*** daemontool has joined #openstack-infra09:44
*** fawadkhaliq has quit IRC09:44
*** Daisy has quit IRC09:44
*** tosky has joined #openstack-infra09:45
*** nadya has quit IRC09:45
*** _degorenko|afk is now known as degorenko09:47
*** thorst has joined #openstack-infra09:50
*** fawadkhaliq has joined #openstack-infra09:50
*** nmagnezi has quit IRC09:50
*** Qiming has quit IRC09:52
openstackgerritSean M. Collins proposed openstack-infra/project-config: Make grenade-dsvm-neutron-multinode voting  https://review.openstack.org/29984309:54
*** cody-somerville has quit IRC09:54
*** thorst has quit IRC09:56
*** kzaitsev_mb has quit IRC09:56
openstackgerritSean M. Collins proposed openstack-infra/project-config: Move gate-grenade-dsvm-neutron-dvr-multinode into check queue  https://review.openstack.org/29984409:56
*** gnuoy_ has joined #openstack-infra09:59
*** gnuoy_ has quit IRC09:59
*** gnuoy_ has joined #openstack-infra09:59
*** esikachev has quit IRC10:00
*** cdent has joined #openstack-infra10:01
*** eliqiao has joined #openstack-infra10:01
*** kzaitsev_mb has joined #openstack-infra10:02
*** verdurin has quit IRC10:03
*** sdake has quit IRC10:03
*** sdague has joined #openstack-infra10:03
*** verdurin has joined #openstack-infra10:04
*** nmagnezi has joined #openstack-infra10:04
*** kzaitsev_mb has quit IRC10:07
*** electrofelix has quit IRC10:08
*** electrofelix has joined #openstack-infra10:09
*** fawadkhaliq has quit IRC10:09
*** ihrachys has joined #openstack-infra10:10
*** fawadkhaliq has joined #openstack-infra10:10
*** gnuoy_ has quit IRC10:12
*** electrofelix has quit IRC10:13
*** electrofelix has joined #openstack-infra10:13
*** e0ne has quit IRC10:15
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: ssh less  https://review.openstack.org/29597310:16
*** e0ne has joined #openstack-infra10:17
*** fawadkhaliq has quit IRC10:18
*** amitgandhinz has joined #openstack-infra10:19
*** fawadkhaliq has joined #openstack-infra10:19
*** zeih has quit IRC10:19
*** e0ne has quit IRC10:20
*** e0ne has joined #openstack-infra10:21
*** zeih has joined #openstack-infra10:22
*** e0ne has quit IRC10:22
*** e0ne has joined #openstack-infra10:23
*** amitgandhinz has quit IRC10:23
*** e0ne has quit IRC10:24
*** fawadkhaliq has quit IRC10:25
*** skraynev has quit IRC10:27
*** skraynev has joined #openstack-infra10:27
*** ociuhandu has quit IRC10:30
*** electrofelix has quit IRC10:31
*** electrofelix has joined #openstack-infra10:31
*** zeih has quit IRC10:33
*** asettle has quit IRC10:34
bookwarhi, is it possible to disable e-mails for a certain gerrit user? We have third-party ci user to comment on reviews and it gets notifications for all changes in all those reviews10:34
*** asettle has joined #openstack-infra10:35
*** stevemar has joined #openstack-infra10:35
*** dizquierdo has joined #openstack-infra10:37
*** fawadkhaliq has joined #openstack-infra10:39
*** stevemar has quit IRC10:39
csabaAJaeger: can you please help me out with a strange issue about glusterfs manila jobs (gate-manila-tempest-dsvm-glusterfs, gate-manila-tempest-dsvm-glusterfs-native)? They are ending up in UNSTABLE because scp-ing the log files fails. This can be observed with any recent manila change, eg. https://review.openstack.org/#/c/299322/10:44
*** permalac has quit IRC10:44
*** yamamoto_ has quit IRC10:47
*** Qiming has joined #openstack-infra10:49
*** fawadkhaliq has quit IRC10:49
*** tpsilva has joined #openstack-infra10:51
*** vgridnev has joined #openstack-infra10:51
*** salv-orlando has joined #openstack-infra10:51
*** doug-fish has joined #openstack-infra10:51
*** baoli has joined #openstack-infra10:52
*** dmellado_ is now known as dmellado10:52
*** thorst has joined #openstack-infra10:54
openstackgerritAnton Arefiev proposed openstack-infra/project-config: Add Ironic Inspector basic tempest job  https://review.openstack.org/29982110:55
*** baoli has quit IRC10:57
*** esker has joined #openstack-infra10:57
*** esikachev has joined #openstack-infra10:58
*** resker has joined #openstack-infra10:58
*** abregman is now known as abregman|brb11:01
*** ociuhandu has joined #openstack-infra11:01
*** esker_ has joined #openstack-infra11:01
*** thorst has quit IRC11:01
*** esker has quit IRC11:01
esikachevhi! infra-cores need merge https://review.openstack.org/#/c/299407/11:02
*** esker has joined #openstack-infra11:02
*** resker has quit IRC11:02
openstackgerritBeth Elwell proposed openstack-infra/storyboard-webclient: Added comments to dashboard.html  https://review.openstack.org/29985611:03
*** resker has joined #openstack-infra11:03
*** cdent has left #openstack-infra11:05
*** esker_ has quit IRC11:06
*** esker has quit IRC11:07
*** esker has joined #openstack-infra11:08
*** resker has quit IRC11:08
*** resker has joined #openstack-infra11:10
*** _amrith_ is now known as amrith11:11
*** smarcet has joined #openstack-infra11:12
*** esker has quit IRC11:12
*** fhorvath has quit IRC11:12
*** esker_ has joined #openstack-infra11:13
*** kgiusti has left #openstack-infra11:13
*** esker__ has joined #openstack-infra11:14
*** cdent has joined #openstack-infra11:14
*** resker has quit IRC11:15
*** resker has joined #openstack-infra11:15
*** esker___ has joined #openstack-infra11:16
*** e0ne has joined #openstack-infra11:17
*** kzaitsev_mb has joined #openstack-infra11:17
*** esker_ has quit IRC11:17
*** julim has quit IRC11:18
*** esker has joined #openstack-infra11:18
*** e0ne has quit IRC11:18
*** esker__ has quit IRC11:19
openstackgerritMerged openstack-infra/tripleo-ci: Clarify location of large output log files  https://review.openstack.org/29676711:19
*** esker_ has joined #openstack-infra11:19
*** amitgandhinz has joined #openstack-infra11:19
*** resker has quit IRC11:20
*** resker has joined #openstack-infra11:20
*** esker___ has quit IRC11:21
*** dkehn has quit IRC11:21
*** abregman|brb is now known as abregman11:21
*** dkehn has joined #openstack-infra11:21
*** julim_ has joined #openstack-infra11:22
*** dstanek has quit IRC11:22
*** nibalize1 has joined #openstack-infra11:22
*** jlvillal_ has joined #openstack-infra11:22
*** adreznec has quit IRC11:22
*** rkukura has quit IRC11:22
*** nibalizer has quit IRC11:22
*** _ody has quit IRC11:22
*** redrobot has quit IRC11:22
*** andymccr has quit IRC11:22
*** fabo has quit IRC11:22
*** jlvillal has quit IRC11:22
*** hashar has quit IRC11:22
*** bigjools has quit IRC11:22
*** dhellmann has quit IRC11:22
*** _ody_ has joined #openstack-infra11:22
*** fabo_ has joined #openstack-infra11:22
*** electrichead has joined #openstack-infra11:22
*** dhellmann_ has joined #openstack-infra11:22
*** andymccr_ has joined #openstack-infra11:23
*** esker has quit IRC11:23
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: ssh less  https://review.openstack.org/29597311:23
*** adreznec has joined #openstack-infra11:23
*** e0ne has joined #openstack-infra11:23
*** dstanek has joined #openstack-infra11:23
*** andymccr_ is now known as andymccr11:23
*** dizquierdo has quit IRC11:23
*** jlvillal_ is now known as jlvillal11:23
*** bigjools has joined #openstack-infra11:24
*** bigjools has quit IRC11:24
*** bigjools has joined #openstack-infra11:24
*** amitgandhinz has quit IRC11:24
*** esker_ has quit IRC11:24
*** jlvillal is now known as Guest8757811:24
openstackgerritVasyl Saienko proposed openstack-infra/project-config: Increase OS_TEST_TIMEOUT for ironic-multitenant  https://review.openstack.org/29856911:24
*** claudiub has joined #openstack-infra11:25
*** hashar has joined #openstack-infra11:25
*** annegentle has joined #openstack-infra11:25
*** resker has quit IRC11:25
*** e0ne has quit IRC11:26
*** dhellmann_ is now known as dhellmann11:26
*** resker has joined #openstack-infra11:26
openstackgerritAnton Arefiev proposed openstack-infra/project-config: Add Ironic Inspector basic tempest job  https://review.openstack.org/29982111:28
*** esker has joined #openstack-infra11:29
*** annegentle has quit IRC11:29
*** esker_ has joined #openstack-infra11:31
*** jiggalanator has joined #openstack-infra11:31
*** resker has quit IRC11:32
*** resker has joined #openstack-infra11:32
*** esker has quit IRC11:33
*** mrmartin has joined #openstack-infra11:34
*** netsin has quit IRC11:34
*** esker has joined #openstack-infra11:34
*** e0ne has joined #openstack-infra11:35
*** esker___ has joined #openstack-infra11:35
*** esker_ has quit IRC11:35
*** resker has quit IRC11:37
*** resker has joined #openstack-infra11:37
*** nmagnezi has quit IRC11:38
*** esker has quit IRC11:39
*** cdent has left #openstack-infra11:39
*** esker has joined #openstack-infra11:39
*** esker___ has quit IRC11:40
*** esker_ has joined #openstack-infra11:40
*** thorst has joined #openstack-infra11:41
*** e0ne has quit IRC11:41
*** esker__ has joined #openstack-infra11:42
*** resker has quit IRC11:42
*** ldnunes has joined #openstack-infra11:42
*** gordc has joined #openstack-infra11:43
*** esker has quit IRC11:44
*** esker_ has quit IRC11:45
*** esker has joined #openstack-infra11:46
*** esker__ has quit IRC11:47
*** resker has joined #openstack-infra11:47
*** yamamoto_ has joined #openstack-infra11:48
*** nmagnezi has joined #openstack-infra11:48
*** esker_ has joined #openstack-infra11:48
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: Run tempest tests with non-HA jobs  https://review.openstack.org/29703811:49
*** esker has quit IRC11:51
*** flepied has quit IRC11:51
*** resker has quit IRC11:52
*** esker_ has quit IRC11:53
*** rfolco has joined #openstack-infra11:53
*** weshay has joined #openstack-infra11:53
AJaegercsaba: this might be related to a change from our Jenkins update a week (or two?) ago.11:53
AJaegercsaba: I see "(Permission denied)" - old Jenkins ignored files it cannot access, new one is unhappy as you notice11:53
AJaegercsaba: So, we need to find a way to change permissions on these files - or not upload them. Better talk with the rest of the team on best solution. I think clarkb has been looking in some of this.11:54
*** flepied has joined #openstack-infra11:56
*** dizquierdo has joined #openstack-infra11:58
*** esker has joined #openstack-infra11:58
*** e0ne has joined #openstack-infra11:59
*** asettle has quit IRC11:59
*** e0ne has quit IRC12:00
*** resker has joined #openstack-infra12:00
*** annegentle has joined #openstack-infra12:00
*** esker_ has joined #openstack-infra12:02
*** esker has quit IRC12:03
*** kgiusti has joined #openstack-infra12:04
*** |-paul-| has quit IRC12:05
*** esker has joined #openstack-infra12:05
*** annegentle has quit IRC12:05
*** resker has quit IRC12:05
*** resker has joined #openstack-infra12:06
*** esker_ has quit IRC12:06
*** nadya has joined #openstack-infra12:07
*** yamamoto_ has quit IRC12:07
*** esker_ has joined #openstack-infra12:08
*** esker has quit IRC12:09
*** aysyd has joined #openstack-infra12:09
*** esker has joined #openstack-infra12:10
*** resker has quit IRC12:10
*** ayoung has quit IRC12:10
*** sdague_ has joined #openstack-infra12:11
*** e0ne has joined #openstack-infra12:11
*** aysyd has quit IRC12:11
*** sfinucan has quit IRC12:11
*** dprince has joined #openstack-infra12:12
*** EricGonczer_ has joined #openstack-infra12:12
*** e0ne has quit IRC12:13
pabelangermorning12:13
*** esker_ has quit IRC12:13
AJaegermorning, pabelanger !12:14
*** resker has joined #openstack-infra12:14
*** esker has quit IRC12:14
*** esker has joined #openstack-infra12:15
*** aysyd has joined #openstack-infra12:16
openstackgerritJim Rollenhagen proposed openstack-infra/project-config: Ironic: always install tempest plugin from master  https://review.openstack.org/29988812:16
*** esker_ has joined #openstack-infra12:16
*** pradk has joined #openstack-infra12:16
*** esker__ has joined #openstack-infra12:17
*** |-paul-| has joined #openstack-infra12:17
*** sfinucan has joined #openstack-infra12:18
*** yamamoto_ has joined #openstack-infra12:18
*** resker has quit IRC12:18
*** resker has joined #openstack-infra12:19
*** esker has quit IRC12:19
*** amitgandhinz has joined #openstack-infra12:20
*** esker has joined #openstack-infra12:20
*** esker_ has quit IRC12:21
*** annegentle has joined #openstack-infra12:21
*** esker__ has quit IRC12:22
*** resker has quit IRC12:24
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Fix releasenotes translation  https://review.openstack.org/29989212:25
AJaegerproject-config cores, this change is needed to fix syncing of translations for openstack-manuals, please approve ^12:25
*** resker has joined #openstack-infra12:25
*** amitgandhinz has quit IRC12:25
*** esker has quit IRC12:25
*** annegentle has quit IRC12:26
*** baoli has joined #openstack-infra12:27
*** baoli has quit IRC12:27
*** nadya has quit IRC12:27
*** baoli has joined #openstack-infra12:27
*** esker_ has joined #openstack-infra12:28
*** zeih has joined #openstack-infra12:29
*** esker__ has joined #openstack-infra12:29
*** fabo_ is now known as fabo12:29
*** nadya has joined #openstack-infra12:30
*** resker has quit IRC12:30
*** moravec has quit IRC12:30
*** esikachev has quit IRC12:30
*** annegentle has joined #openstack-infra12:32
*** esker_ has quit IRC12:33
*** resker has joined #openstack-infra12:33
*** baoli_ has joined #openstack-infra12:34
*** esker__ has quit IRC12:34
*** esker_ has joined #openstack-infra12:34
*** rossella_s has quit IRC12:34
*** |-paul-| has quit IRC12:35
*** rossella_s has joined #openstack-infra12:35
*** stevemar has joined #openstack-infra12:36
*** annegentle has quit IRC12:37
*** _ari_ has joined #openstack-infra12:37
*** baoli has quit IRC12:37
*** esker has joined #openstack-infra12:37
*** resker has quit IRC12:38
*** _ody_ has quit IRC12:38
*** _ody has joined #openstack-infra12:38
*** resker has joined #openstack-infra12:39
*** daemontool_ has joined #openstack-infra12:39
openstackgerritJan Hruban proposed openstack-infra/zuul: Support GitHub PR webhooks  https://review.openstack.org/16311712:39
openstackgerritJan Hruban proposed openstack-infra/zuul: Merge pull requests from github reporter  https://review.openstack.org/24325012:39
openstackgerritJan Hruban proposed openstack-infra/zuul: Fix job hierarchy bug.  https://review.openstack.org/19245712:39
openstackgerritJan Hruban proposed openstack-infra/zuul: Encapsulate determining the event purpose  https://review.openstack.org/24748712:39
openstackgerritJan Hruban proposed openstack-infra/zuul: support github pull reqeust labels  https://review.openstack.org/24742112:39
openstackgerritJan Hruban proposed openstack-infra/zuul: Better merge message for GitHub pull reqeusts  https://review.openstack.org/28066712:39
openstackgerritJan Hruban proposed openstack-infra/zuul: Add 'push' and 'tag' github webhook events.  https://review.openstack.org/19120712:39
openstackgerritJan Hruban proposed openstack-infra/zuul: Add 'pr-comment' github webhook event  https://review.openstack.org/23920312:39
openstackgerritJan Hruban proposed openstack-infra/zuul: Support for dependent pipelines with github  https://review.openstack.org/24750012:39
openstackgerritJan Hruban proposed openstack-infra/zuul: Configurable SSH access to GitHub  https://review.openstack.org/23913812:39
openstackgerritJan Hruban proposed openstack-infra/zuul: GitHub file matching support  https://review.openstack.org/29237612:39
openstackgerritJan Hruban proposed openstack-infra/zuul: Allow github trigger to match on branches/refs  https://review.openstack.org/25844812:39
openstackgerritJan Hruban proposed openstack-infra/zuul: Log GitHub API rate limit  https://review.openstack.org/29237712:39
openstackgerritJan Hruban proposed openstack-infra/zuul: Allow list values in template parameters.  https://review.openstack.org/19120812:39
openstackgerritJan Hruban proposed openstack-infra/zuul: Add basic Github Zuul Reporter.  https://review.openstack.org/19131212:39
openstackgerritJan Hruban proposed openstack-infra/zuul: Support for github commit status  https://review.openstack.org/23930312:39
*** esker_ has quit IRC12:39
*** burgerk has joined #openstack-infra12:39
*** esker_ has joined #openstack-infra12:40
*** stevemar has quit IRC12:40
*** esker__ has joined #openstack-infra12:41
*** esker___ has joined #openstack-infra12:42
smarcetfungi: morning . i have this patch ready to go https://review.openstack.org/#/c/299732/12:42
*** esker has quit IRC12:42
*** dkehn has quit IRC12:42
*** daemontool has quit IRC12:42
*** esker has joined #openstack-infra12:43
*** annegentle has joined #openstack-infra12:43
*** dkehn has joined #openstack-infra12:43
*** resker has quit IRC12:43
*** resker has joined #openstack-infra12:44
*** exploreshaifali has quit IRC12:44
*** esker_ has quit IRC12:45
*** ZZelle has joined #openstack-infra12:45
anteayaZara: morning12:45
*** esker__ has quit IRC12:45
*** esker___ has quit IRC12:46
*** esker_ has joined #openstack-infra12:46
anteayaZara: I won't be going away after all, my basement flooded in the night within 5 hours so I have to cancel my vacation and take care of the house12:46
anteayaZara: so cheer me up with great questions please and I'll do my best to answer them12:46
*** esker has quit IRC12:47
*** zhurong has quit IRC12:48
*** gtmanfred has quit IRC12:48
*** resker has quit IRC12:48
AJaegeranteaya: That's sad to hear ;( I had hoped the flooding was over...12:48
anteayaAJaeger: no it just started, this is the third wave12:49
*** daemontool__ has joined #openstack-infra12:49
*** rlandy has joined #openstack-infra12:49
openstackgerritPaul Belanger proposed openstack-infra/system-config: [WIP] Add bootstrap stage to puppet  https://review.openstack.org/29969712:49
anteayayeah I'm really disappointed, was looking forward to seeing Germany again12:50
anteayaI haven't told the folks I was going to viist yet12:50
*** e0ne has joined #openstack-infra12:50
anteayaI hope they didn't make large plans12:50
*** dmellado is now known as dmellado|lunch12:50
pabelangerclarkb: devstack-gate results: http://paste.openstack.org/show/492570/12:51
Zaraanteaya: Oh no! :012:51
*** e0ne has quit IRC12:51
*** tiswanso has quit IRC12:52
anteayayeah, I cleaned out the sump at 3am and figured I was good, but at 8:30am water was all over, no way I can leave and ask someone else to stay up all night dealing with waater while i go away12:52
*** daemontool_ has quit IRC12:52
*** tiswanso has joined #openstack-infra12:52
AJaegeranteaya: Third? ;(12:52
Zara;_;12:52
ZaraI'm not sure I can promise any questions that'll be good enough to make up for that...12:53
AJaegerpabelanger, could you review https://review.openstack.org/299892 , please?12:53
anteayaAJaeger: yeah it started raining yesterday been raining all night, it was slowing down before that12:53
anteayaZara: just you company is lovely thank you12:54
anteayaZara: so storyboard and gerrit, what do you understand currently?12:54
*** daemontool__ has quit IRC12:55
*** gtmanfred has joined #openstack-infra12:55
AJaegeranteaya: Next weeks weather forecast for GErmany isn't too good ;) But really sad that you have to cancel :(12:55
*** exploreshaifali has joined #openstack-infra12:56
*** sdague_ has quit IRC12:57
AJaegerbbl12:57
*** edmondsw has joined #openstack-infra12:57
*** thiagop has joined #openstack-infra12:58
*** links has quit IRC12:58
openstackgerritPaul Belanger proposed openstack-infra/system-config: [WIP] Add bootstrap stage to puppet  https://review.openstack.org/29969712:58
anteayaAJaeger: yeah, thanks12:59
*** amrith is now known as _amrith_12:59
*** jsavak has joined #openstack-infra12:59
annegentleanteaya: ohh that sounds awful to deal with plus missing vacay!13:00
*** daemontool__ has joined #openstack-infra13:00
*** daemontool__ is now known as daemontool13:00
*** achanda has joined #openstack-infra13:00
*** Goneri has joined #openstack-infra13:01
*** Daisy has joined #openstack-infra13:01
openstackgerritMarton Kiss proposed openstack-infra/system-config: Add missing mysql client to groups-dev and groups  https://review.openstack.org/29991713:02
*** esikachev has joined #openstack-infra13:03
Zaraanteaya: almost nothing... I'm looking generally in the field of getting storyboard talking to other things, review tools, etc, and gerrit is the most promising.13:03
*** denisra has quit IRC13:04
anteayaannegentle: it is a semi-annual event, so the flooding is just a whole lot of loss of sleep and mess but used to it, mostly so sad I have to cancel and disppoint the folks that had asked me to go to munich13:04
anteayathey are such lovely people and was looking forward to seeing them again13:04
anteayaannegentle: thanks for understanding13:05
Zaraanteaya: so I've looked a little at the python-storyboardclient, and I *think* the next step is probably looking at the launchpad integration to see how that works, and how mcuh can be copied13:05
annegentleanteaya: definitely have sympathy for ya13:05
anteayaZara: that would be a far assessment13:05
anteayaannegentle: thanks, I appreciate it13:05
openstackgerritAnton Arefiev proposed openstack-infra/project-config: Add Ironic Inspector basic tempest job  https://review.openstack.org/29982113:05
*** yolanda has joined #openstack-infra13:05
*** jaosorior has quit IRC13:05
*** achanda has quit IRC13:06
*** Daisy has quit IRC13:06
*** jaosorior has joined #openstack-infra13:06
anteayaZara: I think that looking at the launchpad gerrit bits for how launchpad is able to pick up commit message triggers about bugs would be very helpful13:06
anteayaZara: gerrit has a rest api as well as the ability to communicate via ssh13:06
anteayaZara: I'm supportive of using the rest api13:07
*** torandu has quit IRC13:08
*** zul has quit IRC13:08
Zaraanteaya: okay, I'll start there. In general, I don't have much python, and haven't done anything much in this field (working with rest apis and so on), so any tips on where to start, or good tutorials, etc, is appreciated. I think I have a fuzzy handle on the theory of it but have a lot of gaps in my knowledge.13:08
openstackgerritPaul Belanger proposed openstack-infra/system-config: [WIP] Add bootstrap stage to puppet  https://review.openstack.org/29969713:08
anteayasorry I think I want Biennial not semi-annual13:08
anteayaZara: understood, me too13:08
anteayamy best tip for python would be get to know dhellmann if you don't already :)13:09
Zarahehe, and I normally pester SotK but he's off holidaying.13:09
*** torandu has joined #openstack-infra13:10
*** akshai has joined #openstack-infra13:10
anteayayup13:11
*** rfolco has quit IRC13:11
openstackgerritPaul Belanger proposed openstack-infra/system-config: [WIP] Add bootstrap stage to puppet  https://review.openstack.org/29969713:11
*** yamahata has joined #openstack-infra13:14
*** baoli_ has quit IRC13:16
openstackgerritAnton Arefiev proposed openstack-infra/project-config: Add Ironic Inspector basic tempest job  https://review.openstack.org/29982113:17
*** salv-orlando has quit IRC13:18
*** rfolco has joined #openstack-infra13:19
*** salv-orlando has joined #openstack-infra13:19
*** exploreshaifali has quit IRC13:19
*** mtanino has joined #openstack-infra13:20
*** electrofelix has quit IRC13:21
*** electrofelix has joined #openstack-infra13:21
*** electrofelix has quit IRC13:21
*** Goneri has quit IRC13:22
*** salv-orl_ has joined #openstack-infra13:24
*** nwkarsten has joined #openstack-infra13:25
*** salv-orlando has quit IRC13:26
*** zeih has quit IRC13:26
*** amitgandhinz has joined #openstack-infra13:26
openstackgerritJohn Trowbridge proposed openstack-infra/project-config: Add tripleo-quickstart project to tripleo team  https://review.openstack.org/29993213:27
*** zul has joined #openstack-infra13:27
*** dprince has quit IRC13:28
openstackgerritJohn Trowbridge proposed openstack-infra/project-config: Add tripleo-quickstart project to tripleo team  https://review.openstack.org/29993213:28
*** dprince has joined #openstack-infra13:29
*** akshai has quit IRC13:29
pabelangerfungi: So, I have the bootstrap stage working in puppet. However, we need to have a discussion if we want to move forward with it.  Since it will require updates to other puppet modules using apt13:29
pabelangerfungi: it might just be easier to have DIB run apt-get update again before we run puppet13:30
*** akshai has joined #openstack-infra13:30
*** knikolla has joined #openstack-infra13:32
esikachevyolanda: hi! can you merge https://review.openstack.org/#/c/299407/ please13:32
*** e0ne has joined #openstack-infra13:32
*** tqtran has joined #openstack-infra13:34
yolandahi esikachev, i'm not really around today, but will take a look at the review queue in short13:35
esikachevyolanda: thanks, this small change13:36
*** stevemar has joined #openstack-infra13:36
*** Goneri has joined #openstack-infra13:37
*** mriedem has joined #openstack-infra13:37
yolandaesikachev, why commenting the jobs instead of removing those? is that something very very temporary?13:37
*** salv-orl_ has quit IRC13:38
esikachevyolanda: yes, is very-very temporary change (see patches with same topic)13:38
*** tqtran has quit IRC13:39
yolandaok approved13:39
*** tonytan4ever has joined #openstack-infra13:39
*** abregman has quit IRC13:39
esikachevthanks!13:39
*** moravec has joined #openstack-infra13:40
*** mylu has joined #openstack-infra13:40
toskyyolanda: thanks (I'm the author); the commit has a reference to a (long-ish) email which explain the process13:41
*** stevemar has quit IRC13:41
*** abregman has joined #openstack-infra13:41
*** salv-orlando has joined #openstack-infra13:43
*** burgerk has quit IRC13:44
*** permalac has joined #openstack-infra13:45
*** cdent has joined #openstack-infra13:46
mordredinfra-root, Shrews: when the two shade patches in the gate queue land I'm going to cut another shade release and we'll be ready to try shade-nodepool again13:46
mordredyolanda, pabelanger: https://review.openstack.org/#/c/299702 if you get a sec13:47
*** regXboi has joined #openstack-infra13:47
dmsimardWhere can I see the output of devstack gate for a particular job ? Trying to look if fix_disk_layout from functions.sh is doing some wrong stuff again13:48
*** _amrith_ is now known as amrith13:49
Shrewsmordred: i have concerns about jeblair_'s port cache13:49
mordredShrews: yeah?13:49
Shrewsmordred: if we create or delete a port, would we not want that to show up immediately in the cache? it isn't invalidated in the create/delete apis13:50
mtreinishdmsimard: for fix_disk_layout that'll be in the setup workspace log:  http://logs.openstack.org/06/283806/8/check/gate-tempest-dsvm-ironic-agent_ssh-nv/a91a361/logs/devstack-gate-setup-workspace-new.txt.gz#_2016-02-27_15_18_42_82013:50
*** ayoung has joined #openstack-infra13:50
*** cbader_ has quit IRC13:50
mtreinishthat was just a log I had open, it's kinda old though13:50
*** baoli has joined #openstack-infra13:50
openstackgerritMerged openstack-infra/project-config: [sahara-tests] create and merge, disable tests  https://review.openstack.org/29940713:50
*** cdent has left #openstack-infra13:50
*** cbader_ has joined #openstack-infra13:51
mordredShrews: yah, so the code that polls for has a timeout double the time of the server cache time13:51
*** xyang1 has joined #openstack-infra13:52
dmsimardmtreinish: hmm, does a job need to do something special to get that log there ? I don't have it picked up in our logs http://logs.openstack.org/19/299719/1/check/gate-packstack-integration-scenario003-tempest-dsvm-centos7/90813ab/logs/13:52
mordredShrews: so that the code in _get_free_fixed_port won't return until it's had a chance to get it13:52
openstackgerritPaul Belanger proposed openstack-infra/puppet-jenkins: Ubuntu xenial supports openjdk-8-jdk  https://review.openstack.org/29994813:52
*** e0ne has quit IRC13:52
*** salv-orlando has quit IRC13:52
*** mtanino has quit IRC13:52
Shrewsmordred: oh, i see that now. that's a curious way to handle that13:53
*** Daisy has joined #openstack-infra13:53
pabelangerclarkb: fungi: So, apparently openjdk-7 was deprecated at the last moment for xenial ^13:53
*** permalac has quit IRC13:53
mtreinishdmsimard: does packstack use devstack-gate? it doesn't run devstack so I wouldn't think it would13:53
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Use cached instack qcow image for all ci tests  https://review.openstack.org/29953713:53
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Pre install packages on the instack image  https://review.openstack.org/29953813:53
*** maishsk has joined #openstack-infra13:53
mordredShrews: if you're using the batch-the-list-operations version of caching and touching ports directly it's a thing you'd have to do... but this is a thing that really shouldn't be needed by many people that are not doing nodepool levels of parallel operations13:53
openstackgerritPaul Belanger proposed openstack-infra/puppet-jenkins: Ubuntu xenial supports openjdk-8-jdk  https://review.openstack.org/29994813:53
Shrewsmordred: if we're waiting for the cache to invalidate, why not just go ahead and make it so?13:53
mordredShrews: I'm going to take a node to write up some docs on it13:53
mordredShrews: hrm13:54
*** e0ne has joined #openstack-infra13:54
mordredShrews: still coffeeing - lemme think about that for a sec13:54
dmsimardmtreinish: it uses dsvm so I guess it would ?13:54
Shrewsmordred: just not seeing the advantage of waiting13:54
*** dmellado|lunch is now known as dmellado13:54
mordred(there was a reason yesterday)13:54
dmsimardmtreinish: https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/packstack.yaml#L3 (devstack-{ostype})13:55
*** e0ne has quit IRC13:56
mordredShrews: I can't come up with a good reason ... probably would be good to dig in and see if we can prove out the active invalidation approach as a follow up? (landing it this way then imrpoving it shouldn't be an interface break, yeah?)13:56
*** mriedem has left #openstack-infra13:56
*** mriedem has joined #openstack-infra13:57
mtreinishdmsimard: right, it uses a devstack node type. But where is devstack-gate called?13:57
dmsimardmtreinish: does it need to be called explicitely ? (I'm a noob with this, thanks for bearing with me)13:57
*** mohankumar has quit IRC13:57
*** zaro has quit IRC13:58
mtreinishdmsimard: look at a regular devstack-gate job defined in the devstack-gate.yaml you'll see that's what's actually run by the job13:58
mordredShrews: I think the invalidation would be "acquire port lock in blocking mode, zero out port list" yeah?13:58
*** zaro has joined #openstack-infra13:58
pabelangermordred: done13:58
Shrewsmordred: yes, the same way i just modified the network cache14:00
mordredShrews: oh - also, we're not creating the port14:00
Shrewsmordred: but if you want to put it in now, that's fine. it could be improved i think14:00
mordredShrews: it's a thing that happens when create server happens - so we'd have to invalidate the cache in the server loop too14:00
*** zhurong_ has joined #openstack-infra14:00
*** Hal has joined #openstack-infra14:01
dmsimardmtreinish: hmm, they all look to *actually* run devstack ? All I really want is that fix_disk_layout from functions.sh (https://github.com/openstack-infra/devstack-gate/blob/master/functions.sh#L333-L394) runs on the VMs packstack jobs run on.14:01
*** kushal has joined #openstack-infra14:01
mordredShrews: I agree on it being able to be improved - I just have a hunch we're going to find a few edge cases that might take a while to make sure we're solid on14:01
*** Hal is now known as Guest6760114:01
*** eharney has joined #openstack-infra14:02
Shrewsmordred: ah, i see. i think my argument about invalidating in create/delete still holds though14:02
dmsimardmtreinish: apparently I might have wrongly assumed that ran on every VM that was provided by nodepool..14:02
*** kushal has quit IRC14:02
dmsimardas a pre-setup before handing the VM out14:02
*** salv-orlando has joined #openstack-infra14:02
mordredShrews: totally14:02
Shrewsmordred: to be clear, in create_port/delete_port14:02
*** achanda has joined #openstack-infra14:02
mordred++14:03
*** cdent_ has joined #openstack-infra14:03
*** electrofelix has joined #openstack-infra14:03
*** electrofelix has quit IRC14:03
pabelangermordred: do you mind helping with https://review.openstack.org/#/c/299948/ fixed ubuntu-xenial builds14:04
*** cdent_ has quit IRC14:04
pabelangeryolanda: ^ if you are still around14:04
mordredpabelanger: looking14:04
*** cdent has joined #openstack-infra14:04
mordredpabelanger: and jenkins works on java 8 I'm guessing?14:05
*** sigmavirus24_awa is now known as sigmavirus2414:05
pabelangermordred: yup. tested it a few days ago14:05
*** dprince has quit IRC14:06
*** eharney has quit IRC14:06
*** dprince has joined #openstack-infra14:07
*** placko_ has quit IRC14:07
*** sdague has quit IRC14:08
*** achanda has quit IRC14:08
*** sdague has joined #openstack-infra14:08
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Use cached instack qcow image for all ci tests  https://review.openstack.org/29953714:09
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Pre install packages on the instack image  https://review.openstack.org/29953814:09
*** abregman has quit IRC14:09
*** cdent has quit IRC14:09
fungipabelanger: having dib run apt-get update doesn't solve the race i have, which is that puppet is updating teh sources list and then tries to install packages without running apt-get update in between. if we can ensure that the sources list and ensuing apt-get update exec occur before it ever tries to install any packages then that should fix it. from what i was reading, run stages were the preferred14:09
fungisolution to deal with that14:09
*** akshai has quit IRC14:10
pabelangerfungi: Ya, with my old manifests, I always did package updates in a different stage then main.  Worked well14:10
pabelangerIf we are going that route, we should talk about it more, since this is the first time adding a new stage to puppet.14:11
pabelangercrinkle and nibalize1 likely have suggestions too14:11
*** akshai has joined #openstack-infra14:11
*** rfolco has quit IRC14:12
openstackgerritMerged openstack-infra/shade: Cache ports like servers  https://review.openstack.org/29967114:12
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Fix image caching logic  https://review.openstack.org/29953714:14
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Pre install packages on the instack image  https://review.openstack.org/29953814:14
*** sigmavirus24 is now known as sigmavirus24_awa14:14
sdaguepabelanger: could we get a couple xenial nodes put into the pool and an experimental devstack with it?14:14
*** sigmavirus24_awa is now known as sigmavirus2414:15
pabelangersdague: ubuntu-xenial still not online yet. We need to land https://review.openstack.org/#/c/299948/ first14:15
sdaguepabelanger: ok, gotcha14:15
pabelangersdague: once we do, I don't see a ready why not.14:15
pabelangerexcept for official support from -infra won't happen until xenial is official released :D14:16
*** cbader has joined #openstack-infra14:16
dmsimardclarkb: oi, more fix_disk_layout woes. I wrongly assumed that function ran by default on devstack-* nodes before being handed out to jobs. Should I just call it explicitely for Packstack jobs or is there another way ?14:16
pabelangerDoing my best to keep it online in the beta stage14:16
sdaguesure, but an experimental devstack job actually means we can address issues early14:16
* mordred hands pabelanger a box of kittens14:16
mordredsdague: ++14:16
*** Guest67601 has quit IRC14:16
*** cbader has quit IRC14:17
pabelangersdague: Ya, devstack-gate actually ran on my ubuntu-xenial dib. Which is good news14:17
*** cbader_ has quit IRC14:17
*** cbader has joined #openstack-infra14:17
*** ajmiller has joined #openstack-infra14:17
*** kushal has joined #openstack-infra14:18
*** esker_ has quit IRC14:18
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add centos-7 to bindep-fallback  https://review.openstack.org/29998814:18
*** burgerk has joined #openstack-infra14:18
*** cdent has joined #openstack-infra14:19
crinklepabelanger: fungi oh this was happening on infra-cloud too, but we solved it by just not removing the default sources list :(14:19
openstackgerritRaildo Mascena proposed openstack-infra/project-config: Changing gate on devstack identity v3 only voting  https://review.openstack.org/29289414:19
*** cdent has quit IRC14:20
crinklepabelanger: fungi we could have an exec { 'apt-get update': } with a before relationship to Package<||> and a require on the apt source file14:20
mordredcrinkle: that sounds sane14:21
crinklei don't really understand stages so i'm hesitant to recommend it, in my mind it's a workaround to something that in theory puppet should handle well14:21
*** hogepodge has joined #openstack-infra14:21
*** eharney has joined #openstack-infra14:21
fungicrinkle: yeah, leaving the old sources behind is probably not terrible since the goal was to pick up newer packages from official ubuntu locations when rackspace is lagging behind (for example, they commit the unpardonable sin of mirroring the ubuntu security repo and then setting that as teh only source for security updates in their images)14:22
pabelangerI'm fine with crinkles suggestion. I've just done stages in the past, and don't want to push it if people don't think we need them.  Also, switching to stages will be more work14:23
fungicrinkle: i was also looking at the Package<||> trick, but from what i was reading it seemed like it was going to exec an apt-get update every time we apply regardless of whether we're installing packages... i guess that's the only way for it to know there are new packages anyway though?14:23
fungioh, though we don't rely on puppet to upgrade our packages, that's what we have unattended upgrades set up for14:24
*** cdent has joined #openstack-infra14:24
crinklefungi: it wouldn't need to apt-get update every time14:24
csabaclarkb: ping14:24
crinkleit could refreshonly on updating the apt source file14:24
fungiaha, right you are14:24
fungii'll see if i can get that passing tests. i had it halfway written up before i passed out last night14:25
crinklecool14:25
fungithanks!14:25
crinklenp :)14:25
csabaAJaeger: thanks.14:26
*** rfolco has joined #openstack-infra14:26
csabaclarkb: I have some permission issues with uploading log files, AJaeger suggested to talk to you. (Please see http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-03-31.log.html#t2016-03-31T10:44:04 and http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-03-31.log.html#t2016-03-31T11:53:36 for details)14:27
*** zaro has quit IRC14:28
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci: Run tempest tests with periodic jobs  https://review.openstack.org/29703814:28
*** dingyichen has quit IRC14:28
*** links has joined #openstack-infra14:29
*** e0ne has joined #openstack-infra14:29
*** e0ne has quit IRC14:29
*** mohankumar has joined #openstack-infra14:31
*** zaro has joined #openstack-infra14:31
*** esikachev has quit IRC14:33
*** mizar has joined #openstack-infra14:34
*** mizar is now known as flongo14:34
*** Jeffrey4l_ has quit IRC14:35
*** Jeffrey4l_ has joined #openstack-infra14:35
openstackgerritCarlos Camacho proposed openstack-infra/tripleo-ci: Added validation to tripleo.sh --overcloud-deploy option.  https://review.openstack.org/29636214:35
*** oanson has quit IRC14:36
Zaraanteaya: btw, I'm going to end up approaching this updating-from-gerrit stuff with my usual thinking-out-loud, so currently doing that in #storyboard, can in here or someplace else, but concerned it'll be a bit spammy.14:37
Zarathey're used to me waffling away to myself in there14:37
anteayaI'lll join you in storyboard14:37
anteayafeel free to waffling away14:38
*** ajmiller has quit IRC14:39
*** ajmiller has joined #openstack-infra14:41
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Always apt-get update after replacing sources  https://review.openstack.org/30000514:43
fungicrinkle: that ^ what you were suggesting?14:43
fungipabelanger: ^14:43
*** esker has joined #openstack-infra14:43
sdaguewe're getting some really weird and slow vexxhost nodes - http://logs.openstack.org/85/262585/28/check/gate-nova-python27-db/080afb3/console.html14:45
*** e0ne has joined #openstack-infra14:45
sdaguethat took an hour to not start running unit tests14:45
fungisdague: yeah, i'm trying to catch and hold one. i have several log examples14:45
sdaguefungi: it looks like the whole thing is on molasses14:46
fungisdague: i have a feeling it's (lack of) write caching on some of their hosts, which makes apt/dpkg painfully slow14:46
sdaguethe timing for package installs14:46
*** jlanoux_ has quit IRC14:46
sdagueright, io constraints probably14:46
fungiyeah, debian package operations are meticulous about syncing between writes to avoid broken systems, and without write caching that turns into pain14:47
crinklefungi: yep that looks right14:47
*** andrewbogott has quit IRC14:47
*** andrewbogott has joined #openstack-infra14:47
*** esker has quit IRC14:48
fungisdague: anyway, i want to catch and hold an impacted instance, but second best thing would be to just get a list of some affected hostnames so i can turn them into nova instance uuids to pass along to mnaser14:48
mrmartinhi14:48
*** links has quit IRC14:48
mnaseri'll have a look in a few14:48
AJaegersdague, fungi, anteaya : Could either of you review https://review.openstack.org/299892 to fix release notes translations, please? pabelanger already gave a +2.14:49
*** SumitNaiksatam has quit IRC14:49
mrmartinyolanda, the groups-dev is ok on the trusty instance, may I ask you to switch the dns and drop the old one? (it is the dev!)14:49
*** SumitNaiksatam has joined #openstack-infra14:49
mnaserwe're making some adjustments to the scheduler, the whole "compute node with most free memory = best host to place instances on" does not prove to be correct14:49
*** links has joined #openstack-infra14:49
*** lyarwood has quit IRC14:49
fungialso, the ticket we opened with rackspace about network issues in ord is going nowhere. we're down a couple hundred quota until we can turn rax-ord back on but i'm not comfortable doing that until we have more details on whether the network issues will come right back when we do14:50
pabelangerfungi: 2 comments14:50
mnaserbut impacted instance uuids can help confirm the theory of this14:50
*** zaro has quit IRC14:50
fungimnaser: yep, will be working on coming up with some, or better still holding an instance that's impacted so we can experiment further14:50
*** SumitNaiksatam has quit IRC14:51
mnaserfungi: if the behaviour is common, it could be the block storage seeing some problems therefore it's really slow to respond too14:51
*** esikachev has joined #openstack-infra14:51
mnaserwe found out that the journal drives we were using were less-than-ideal so they're all being replaced, but the price of this = recovery of cluster which can cause io slowdowns14:51
*** jaosorior_ has joined #openstack-infra14:52
fungimnaser: yep, that would also account for it. seems to be disk i/o that's especially slow14:52
*** zaro has joined #openstack-infra14:52
mnaserso the mirror instance might be stuggling (that might be easy to check)14:52
fungiparticularly write operations though i think14:52
mnaserdisk i/o to block storage (for the mirror instance?) or local?14:52
mnaseryes, writes have been bad mainly because the ceph journal drive got punished with writes, so we're replacing them with much higher durability drives14:52
fungimnaser: oh, the mirror doesn't seem to be affected but i'll look closer14:52
*** jaosorior has quit IRC14:53
fungii was thinking it looked like slow i/o on the rootfs of teh worker instances, but it was hard to tell14:53
mnasercould also be. we have a customer that came has be pounding our CPUs so we've been playing with the scheduler14:53
mnasermore specifically the utilization based scheduler14:53
fungiif it were i/o struggling on the mirror block device i'd have expected high iowait but that server is basically idle14:54
*** xnox has quit IRC14:54
*** SumitNaiksatam has joined #openstack-infra14:54
mnaserso that means that it's the local instance store, thanks fungi14:54
yolandahi mrmartin, going to do it14:55
fungii ocattionally see iowait pop up to 0.1% of a processor on the mirror for a single poll and then drop back to 0.0% again14:55
fungier, occasionally14:56
mrmartinyolanda, thank you, and can you make me a similar instance for groups.o.o also?14:56
mnaseryeah that shouldn't be the reason behind it14:56
yolandayep14:56
*** sdake has joined #openstack-infra14:57
mmedvedehi, can any puppet/infra/nodepool core take a look at https://review.openstack.org/#/c/276964/ ? It needs +A14:57
*** FallenPegasus has joined #openstack-infra14:57
fungiso anyway, for rackspace, i think we're going to need to add `traceroute6 git.openstack.org` in our jobs and then inch ord back up a little bit in nodepool to see if we can catch one with network issues14:58
mmedvedepabelanger: hi, were you able get a tempest run on xenial VM? any problems with volume snapshots?14:58
*** jsavak has quit IRC14:59
*** sshnaidm has quit IRC14:59
*** xnox has joined #openstack-infra14:59
*** jsavak has joined #openstack-infra15:00
pabelangermmedvede: {1} tempest.api.volume.admin.test_snapshots_actions.SnapshotsActionsV1Test.test_snapshot_force_delete_when_snapshot_is_creating [273.344596s] ... FAILED15:00
pabelangerbut devstack-gate timed out before the rest finished15:00
*** kushal has quit IRC15:00
*** mtanino has joined #openstack-infra15:00
mmedvedepabelanger: ok, that might be confirming the same problem I have seen15:01
mmedvedepabelanger: lvm2 snapshots are slow15:01
*** e0ne has quit IRC15:01
*** maishsk has quit IRC15:01
pabelangermmedvede: We should have ubuntu-xenial uploaded today, then we can setup an experimental job for people to debug with15:02
mmedvedepabelanger: for me most of snapshots tests where unstable/timing out. I found the workaround, described in https://bugs.launchpad.net/ubuntu/+source/lvm2/+bug/156122815:03
openstackLaunchpad bug 1561228 in lvm2 (Ubuntu) "dmevent can not open shared object file libdevmapper-event-lvm2snapshot.so leads to lvcreate/lvremove being slow" [Undecided,New]15:03
*** IlyaG has joined #openstack-infra15:04
yolandamrmartin, dns record updated, going to launch the replacement for groups15:04
pabelangermmedvede: Ya, here was the error: Details: (SnapshotsActionsV1Test:test_snapshot_force_delete_when_snapshot_is_creating) Failed to delete volume-snapshot 2cdcf5dc-27c0-44ef-903c-1049790e4d1f within the required time (196 s)15:04
yolandait will take a time for dns to be updated, once it's done i'll remove the old groups-dev15:05
*** zaro has quit IRC15:05
mmedvedepabelanger: that is exactly it. There is a script in the bug to reproduce the problem15:05
mmedvedepabelanger: if you have access to syslog, it also would have errors in it15:06
*** electrichead is now known as redrobot15:06
*** stevemar has joined #openstack-infra15:06
*** zaro has joined #openstack-infra15:06
*** fhubik has quit IRC15:07
*** jistr has quit IRC15:07
*** esker has joined #openstack-infra15:07
*** mtanino has quit IRC15:09
*** jerryz has joined #openstack-infra15:09
*** jistr has joined #openstack-infra15:09
*** mrmartin has quit IRC15:09
*** mtanino has joined #openstack-infra15:11
*** salv-orlando has quit IRC15:11
*** esker has quit IRC15:12
*** nmagnezi has quit IRC15:12
*** andymaier has joined #openstack-infra15:12
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Traceroute to git.o.o in net-info macro  https://review.openstack.org/30002115:13
fungiinfra-root: that's ^ intended to help us help rackspace once we turn ord back up a bit, but it's going to affect a ton of job configs so will need some manual jjb updates once merged15:13
anteayaAJaeger: done15:14
*** piet has joined #openstack-infra15:14
smarcetfungi: morning , seems that https://review.openstack.org/#/c/299732/ timeouted ?15:14
pabelangerfungi: okay, I am offline in 45mins for the next few hours, but should be able to help monitor once back online15:15
fungismarcet: indeed, looks also affected by the slowness we've been seeing in vexxhost. i'll look up the uuids for those workers and let them know (and i'll reapprove the change in the meantime)15:16
*** vgridnev has quit IRC15:16
*** mohankumar has quit IRC15:16
*** esker has joined #openstack-infra15:16
*** zz_dimtruck is now known as dimtruck15:16
smarcetfungi: tnx a lot :)15:17
*** jlanoux has joined #openstack-infra15:17
*** kdas_ has joined #openstack-infra15:18
*** links has quit IRC15:19
*** armax_ has joined #openstack-infra15:22
openstackgerritMerged openstack-infra/project-config: Fix releasenotes translation  https://review.openstack.org/29989215:22
fungimnaser: here's three affected uuids to start you out: 273e30d6-19d4-4b56-9a91-43146fada083 8a44c948-d30f-40c3-a024-a48fd4b15fea f3feed0e-a6ba-4503-bc86-28c35672f96a15:22
fungione's the job sdague mentioned earlier, the other two we just hit testing another change15:23
*** rkukura has joined #openstack-infra15:23
*** kdas_ has quit IRC15:23
*** jaosorior_ has quit IRC15:24
*** bpokorny has joined #openstack-infra15:24
*** ayoung has quit IRC15:24
*** Qiming has quit IRC15:24
*** armax has quit IRC15:25
*** armax_ is now known as armax15:25
*** ayoung has joined #openstack-infra15:25
*** arxcruz has quit IRC15:25
*** esker has quit IRC15:26
openstackgerritMerged openstack-infra/devstack-gate: Use journald when it is present  https://review.openstack.org/29968215:26
openstackgerritMerged openstack-infra/puppet-openstackid: Fix Vhost (Apache2.4)  https://review.openstack.org/29973215:26
fungismarcet: ^ much better. passed and merged this time15:26
smarcetcool thx15:27
*** vgridnev has joined #openstack-infra15:27
*** bpokorny has quit IRC15:28
*** jistr has quit IRC15:28
*** zaro has quit IRC15:28
*** zhurong_ has quit IRC15:28
*** jistr has joined #openstack-infra15:28
*** tiswanso has quit IRC15:29
*** bpokorny has joined #openstack-infra15:30
*** zaro has joined #openstack-infra15:30
fungismarcet: i'm keeping tabs on when the next puppet apply pulse hits the server, then we can see if there's anything else missing15:31
fungishould hopefully be within the next 15 minutes15:31
smarcetfungi: cool15:31
smarceti think that would be all, hopefully15:31
*** ams_ has joined #openstack-infra15:31
*** jcoufal_ has joined #openstack-infra15:33
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Always apt-get update after replacing sources  https://review.openstack.org/30000515:33
*** mohankumar has joined #openstack-infra15:34
*** dimtruck is now known as zz_dimtruck15:34
*** jcoufal has quit IRC15:35
*** FallenPegasus has quit IRC15:35
yolandamordred, i'm having issues on launch node15:36
yolandabootstrap server fails , although it creates the server, stops after that15:36
*** SumitNaiksatam has quit IRC15:37
*** markvoelker has joined #openstack-infra15:37
*** kdas_ has joined #openstack-infra15:37
yolandamm, actually is puppet so it may be a problem with the manifest15:37
*** e0ne has joined #openstack-infra15:37
*** tosky has quit IRC15:38
fungiyolanda: i believe you're running into the same thing i was, which prompted 30000515:38
*** sshnaidm has joined #openstack-infra15:38
*** kdas_ has quit IRC15:38
*** sigmavirus24 is now known as sigmavirus24_awa15:38
fungiyolanda: add the --keep option when you run launch-node.py so that it won't delete the instance, then ssh into it and look in /var/log/syslog for puppet apply failures15:39
*** jerryz has quit IRC15:39
fungiyolanda: if it's failing on an attempt to install some package(s) then that's what we're working on trying to fix15:39
yolandayep, i have it with keep, going to check15:39
*** e0ne has quit IRC15:40
mnaserfungi: thank you, this is noted15:40
*** markvoelker has quit IRC15:40
*** sigmavirus24_awa is now known as sigmavirus2415:40
yolandafungi i see a drush site-install failed error15:40
*** torandu has quit IRC15:40
*** markvoelker_ has joined #openstack-infra15:41
*** sigmavirus24 is now known as sigmavirus24_awa15:41
fungiyolanda: another possibility then is it's hitting a timeout? i know a fresh groups deployment takes... a while15:41
*** sigmavirus24_awa is now known as sigmavirus2415:41
yolandagoing to run it with a limit15:42
fungiand also the output from it last i checked was so verbose syslog stopped accepting entries15:42
fungiso the only way to diagnose was to try to run the same exec myself manually15:42
*** kushal has joined #openstack-infra15:42
*** torandu has joined #openstack-infra15:42
fungiand redirect the output to a file for easier inspection (and to be able to hand off to mrmartin as well)15:43
yolandafungi http://paste.openstack.org/show/492609/15:43
*** e0ne has joined #openstack-infra15:44
yolandalooks as a problem with the manifest, so i'll check with mrmartin15:44
yolandai actually thing i saw the same error on groups-dev15:44
*** Swami has joined #openstack-infra15:44
smarcetfungi: seems to be up https://openstackid-dev.openstack.org !!! :)15:46
*** dtantsur is now known as dtantsur|afk15:46
*** thiagop is now known as thiagop-afk15:46
fungiyolanda: looks like it was the `drush dsd-init @groups` command that failed (returned nonzero)15:46
*** e0ne has quit IRC15:46
fungismarcet: yep, i just noticed it updated ~6 minutes ago15:46
smarcetfungi: cool tnx for all your help on this :)15:47
*** e0ne has joined #openstack-infra15:47
fungismarcet: so anyway, next steps are to confirm your load tests there are returning the desired results, and then we can plan for a replacement build of the production server and try for a fast cut-over15:47
smarcetfungi: i think so15:48
*** bpokorny has quit IRC15:48
fungithat'll require another system-config patch to set the production redis version and also adjust the disablement of the production server to be by uuid instead of by name so we can successfully build the replacement with the same name15:48
*** bpokorny has joined #openstack-infra15:48
fungialso i'll need to see if there's more cleanup we can do in that tenant so i'll have enough quota to temporarily have the new 15gb server at the same time we're still using the old one15:49
smarcetfungi: roger that15:50
*** markvoelker_ has quit IRC15:50
*** markvoelker has joined #openstack-infra15:51
openstackgerritMerged openstack-dev/hacking: Update the assertRaises example to be more modern  https://review.openstack.org/28409715:52
*** zhurong has joined #openstack-infra15:53
*** kushal has quit IRC15:53
*** eharney has quit IRC15:53
*** zz_dimtruck is now known as dimtruck15:54
*** Guest87578 is now known as jlvillal15:55
*** hashar has quit IRC15:55
smarcetfungi: i am testing i would need to propose a patch to tweak a little connection settings15:55
*** markvoelker has quit IRC15:55
*** jaypipes has joined #openstack-infra15:55
openstackgerritMerged openstack-infra/project-config: Traceroute to git.o.o in net-info macro  https://review.openstack.org/30002115:56
*** markvoelker has joined #openstack-infra15:56
fungismarcet: that's fine--i expected the values may still need some adjusting based on your subsequent testing. those are easy to review15:56
smarcetfungi: ok cool tnx15:56
*** baoli_ has joined #openstack-infra15:56
*** markvoelker_ has joined #openstack-infra15:57
*** baoli has quit IRC15:57
*** e0ne has quit IRC15:58
robcresswellHello. How do I go about getting a project to sync with global-reqs automagically?15:58
clarkbrobcresswell: add your project name to the projects list in the requirements repo and add the requirements sync jenkins jobs to your project15:58
robcresswellclarkb: Straightforward. Whats the order, assuming we need a Depends-On?15:59
*** esker has joined #openstack-infra15:59
fungirequirements check jobs (the requirements sync is a job on the requirements repo controlled via the projects.txt file there)15:59
clarkbyup that15:59
*** kzaitsev_mb has quit IRC16:00
*** tphummel has joined #openstack-infra16:00
*** jlanoux_ has joined #openstack-infra16:00
*** eharney has joined #openstack-infra16:00
clarkbcsaba: the log file explains the issue http://logs.openstack.org/22/299322/2/check/gate-manila-tempest-dsvm-glusterfs/e7a146b/console.html#_2016-03-31_06_04_55_76116:00
*** esker has quit IRC16:00
clarkbcsaba: you will need to correct that in your jobs16:00
*** esker has joined #openstack-infra16:00
*** angdraug has joined #openstack-infra16:00
*** jlanoux has quit IRC16:00
*** markvoelker has quit IRC16:01
zxiiroAny Zuul core-reviewers around who can merge my 2 documentation patches? They've been sitting around for awhile now.... https://review.openstack.org/271868 https://review.openstack.org/27187216:01
robcresswellOh I see, so its just a single patch?16:01
yolandafungi about https://review.openstack.org/#/c/300005/2/modules/openstack_project/manifests/template.pp ... we are normally not using the arrow syntax for dependencies, i actually remember having -1 on some of my patches for that16:01
*** sdake has quit IRC16:01
clarkbrobcresswell: no its two separate changes. But the jobs go in first then the project list update controls the jobs16:01
yolandais something we want to continue enforcing or are we ok with arrow deps?16:01
fungiyolanda: it's magic symbols to me. suggest a more standard syntax and i'm happy to update the change16:02
crinkleyolanda: you can't not use the arrow syntax when using collectors (the <||>)16:02
robcresswellclarkb: Gotcha. Thanks.16:02
*** bmjen has quit IRC16:03
*** Hunner has quit IRC16:03
fungiright, and in this case we need a collector to be able to force the exec to run before any package resource is satisfied, aiui16:03
crinkleright16:03
*** Hunner has joined #openstack-infra16:03
*** bmjen has joined #openstack-infra16:03
*** Hunner has quit IRC16:03
*** Hunner has joined #openstack-infra16:03
fungii assume -> acts similar to a before parameter in this case?16:03
yolandaok so then there is no way to do with requires?16:04
crinklefungi: yes16:04
clarkbfungi: crinkle yolanda that said did we confirm we need an explicit exec or is it sufficient to write that file before installing any packages?16:04
*** Daisy has quit IRC16:04
*** kzaitsev_mb has joined #openstack-infra16:04
*** abregman has joined #openstack-infra16:04
fungiyolanda: i think to make it work with a requires statement, we'd have to add requires to every single package resource we declare16:04
*** jsavak has quit IRC16:04
*** Daisy has joined #openstack-infra16:04
*** esker has quit IRC16:04
*** Jeffrey4l_ has quit IRC16:04
*** Jeffrey4l_ has joined #openstack-infra16:05
yolandawe don't want that :) i was thinking there was some alternative with requires or before syntax, as we normally use16:05
fungiclarkb: i looked at the puppet stdout order and the sequence was 1. updates sources list, 2. tries to install packages and some are not found, 3. after that failure, i am able to manually install the same packages16:05
fungiso something somewhere was doing an apt-get update, but not echoed to the log and not early enough to solve the package install race apparently16:05
*** achanda has joined #openstack-infra16:05
*** SumitNaiksatam has joined #openstack-infra16:06
clarkbI see, in that case do we want to make that a refreshonly or just unconditionally update?16:06
fungiclarkb: i only saw it on bootstrapping, it was eventually consistent, so i think only running it if we modify the sources list should be sufficient16:06
*** [HeOS] is now known as HeOS16:07
openstackgerritGraham Hayes proposed openstack-infra/project-config: Add designate stable/mitaka to gerritbot  https://review.openstack.org/30005816:07
crinkleclarkb: it's only really a problem once in a deployment, between when /etc/apt/sources.list is removed and apt-get update is run16:07
*** asettle has joined #openstack-infra16:07
crinkleso i think refreshonly is sufficient16:07
*** vgridnev has quit IRC16:07
clarkbI see16:07
*** esker has joined #openstack-infra16:07
clarkbits the remove add that we are sorting out not just the add, makes sense16:07
*** zhurong has quit IRC16:07
*** vgridnev has joined #openstack-infra16:08
*** salv-orlando has joined #openstack-infra16:08
clarkbdo we need to explicitly run the purge before the apt-get update as ewll?16:08
*** vgridnev has quit IRC16:08
*** thorst is now known as thorst_afk16:08
*** pcaruana has quit IRC16:08
*** mohankumar has quit IRC16:09
*** Daisy has quit IRC16:09
crinklei think the plabs apt module figures that out for us, let me check16:09
*** esker has quit IRC16:09
openstackgerritRob Cresswell proposed openstack-infra/project-config: Add requirements job to horizon-cisco-ui  https://review.openstack.org/30006116:10
*** sridhar_ram has joined #openstack-infra16:10
*** maishsk has joined #openstack-infra16:10
*** esker has joined #openstack-infra16:10
openstackgerritRob Cresswell proposed openstack/requirements: Add horizon-cisco-ui to projects.txt  https://review.openstack.org/30006216:11
*** e0ne has joined #openstack-infra16:11
*** achanda has quit IRC16:11
openstackgerritShamail Tahir proposed openstack-infra/irc-meetings: Adding AsiaPac Regional PWG Meeting  https://review.openstack.org/29977616:11
*** asettle has quit IRC16:12
*** Jeffrey4l_ has quit IRC16:12
*** rfolco has quit IRC16:12
openstackgerritRob Cresswell proposed openstack-infra/project-config: Add translation-jobs to horizon-cisco-ui  https://review.openstack.org/30006516:14
crinkleclarkb: the purge will happen before the apt-get update https://github.com/puppetlabs/puppetlabs-apt/blob/master/manifests/init.pp#L10816:14
crinkleand we could maybe reuse that apt::update class instead of the exec if we wanted16:14
*** kushal has joined #openstack-infra16:14
clarkbthat seems reasonable16:15
clarkbdelete the Exec and change out the resource on the left side of the ->16:15
clarkband make sure the file is before that16:15
*** jtomasek has quit IRC16:16
*** nmagnezi has joined #openstack-infra16:16
*** fawadkhaliq has joined #openstack-infra16:19
*** shardy has quit IRC16:19
*** kzaitsev_mb has quit IRC16:20
*** kzaitsev_mb has joined #openstack-infra16:21
*** shardy has joined #openstack-infra16:21
clarkbpabelanger: looks like xenial builds are still failing, also we will be cutting it close on disk space on nodepool I think16:22
*** cdent has quit IRC16:22
clarkbso will want to keep an eye on that16:22
*** cdent has joined #openstack-infra16:23
clarkblooks like it is still mad about openjdk 716:23
*** sdake has joined #openstack-infra16:24
*** sdake has quit IRC16:24
*** mikelk has quit IRC16:24
clarkbI cleaned out one extra copy of an image in /opt I made for mounting and editing purposes16:24
*** sdake has joined #openstack-infra16:24
clarkbI don't think there is anymore throw away data on that volume now. Its all dib cache or dib images16:25
*** sdake has quit IRC16:28
*** sdake has joined #openstack-infra16:29
*** apoorvad has joined #openstack-infra16:29
*** yamahata has quit IRC16:30
*** sdake has quit IRC16:30
*** jsavak has joined #openstack-infra16:30
*** davideagnello has joined #openstack-infra16:30
*** LaT0rtue has joined #openstack-infra16:31
*** esker has quit IRC16:31
dmsimardclarkb: did you come across my question earlier about running fix_disk_layout ?16:31
*** e0ne has quit IRC16:32
clarkbdmsimard: I did not, is it in scrollback?16:32
*** esker has joined #openstack-infra16:32
*** nadya has quit IRC16:32
clarkbah yup its part of the job itself16:32
dmsimardclarkb: yeah, but basically it seems like I wrongly assumed fix_disk_layout ran on essentially everything16:32
clarkbwe avoid the overhead where it isn't necessary (like unittests)16:32
clarkbprobably the simplest thing is to call the function if you need it16:33
dmsimardso should I clone devstack-gate and just call fix_disk_layout before starting my testsor something like that ?16:33
dmsimardis devstack gate implicitly cloned somewhere already ?16:33
clarkbwe could also put it in the jenkins scripts instead of devstack-gate so that you don't need to hvae devstack gate cloned16:33
clarkbnothing is implicitly cloned. The slaves do have a repo cache on them and if you use zuul-cloner it will take advantage of that. The devstack-gate jobs do this for the devstack-gate setup16:34
openstackgerritJohn L. Villalovos proposed openstack-infra/project-config: Ironic grenade jobs do not work until after Mitaka  https://review.openstack.org/30008216:34
clarkbI think it even has a macro you can reuse16:34
*** jerryz has joined #openstack-infra16:34
*** rossella_s has quit IRC16:34
*** mylu has quit IRC16:34
dmsimardclarkb: I guess https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/macros.yaml#L57316:35
clarkbdmsimard: yup16:35
*** rossella_s has joined #openstack-infra16:35
*** esker has quit IRC16:36
*** fawadkhaliq has quit IRC16:36
openstackgerritJohn Trowbridge proposed openstack-infra/project-config: Add tripleo-quickstart project to tripleo team  https://review.openstack.org/29993216:37
*** fawadkhaliq has joined #openstack-infra16:37
dmsimardclarkb: okay I'll try something along these lines. Thanks for the help.16:39
*** sfinucan has quit IRC16:39
*** cdent has quit IRC16:41
*** fawadkhaliq has quit IRC16:41
*** trown has joined #openstack-infra16:41
*** fawadkhaliq has joined #openstack-infra16:41
*** pvaneck has joined #openstack-infra16:42
*** matrohon has quit IRC16:42
trowncan I ignore the -1 Jenkins on https://review.openstack.org/#/c/300084/ or is there something I need to do for that?16:43
*** cdent has joined #openstack-infra16:43
clarkbtrown: "This change depends on a change that failed to merge"16:44
clarkbyou will likely need to address that if you wish to get it merged16:44
jrolltrown: recheck it, you'll get that if you update the dependent patch while CI is running (which you did)16:44
AJaegertrown: add a "recheck" once the other change has passed testing16:44
trownah perfect, thanks!16:45
*** e0ne has joined #openstack-infra16:47
*** piet has quit IRC16:48
*** sdake has joined #openstack-infra16:48
*** maishsk has quit IRC16:48
*** yaume has quit IRC16:49
*** e0ne has quit IRC16:50
*** dizquierdo has quit IRC16:51
*** resker has joined #openstack-infra16:51
*** maishsk has joined #openstack-infra16:52
clarkbpabelanger: ianw: do we have a rough plan for getting off of devstack-trusty yet? I think I have seen some experimental jobs for centos-7 floating around16:52
openstackgerritRob Cresswell proposed openstack-infra/project-config: Add translation-jobs to horizon-cisco-ui  https://review.openstack.org/30006516:53
*** nwkarsten has quit IRC16:54
*** esker_ has joined #openstack-infra16:54
*** nwkarsten has joined #openstack-infra16:54
*** piet has joined #openstack-infra16:55
*** flongo has quit IRC16:55
*** resker has quit IRC16:56
clarkbfungi: crinkle so what do you think? should we update that change to use the update resource that already exists? or should I go ahead and approve it as is?16:57
*** thorst_afk is now known as thorst16:57
clarkbI think my preference is to avoid the exec if we can but its not strong enough to not merge as is16:57
*** rpodolyaka is now known as rpodolyaka_afk16:57
crinklei have a slight preference for using the existing apt::update class16:57
crinklefungi: ^16:58
fungiclarkb: i'm open to either solution. avoided using the apt module for managing the sources list since it didn't really have the features we needed there, but for calling update it seems perfectly well suited16:58
fungii'll update the change shortly16:58
clarkbsounds good thanks16:58
*** esker_ has quit IRC16:58
*** nwkarsten has quit IRC16:58
*** mylu has joined #openstack-infra16:59
openstackgerritsebastian marcet proposed openstack-infra/puppet-openstackid: Tweaked server connection settings  https://review.openstack.org/30009716:59
*** nadya has joined #openstack-infra16:59
*** nwkarsten has joined #openstack-infra16:59
*** jistr has quit IRC17:00
*** nadya has quit IRC17:01
*** rbrndt has joined #openstack-infra17:03
openstackgerritMerged openstack-infra/release-tools: script to propose final releases for a series  https://review.openstack.org/29955117:03
*** yamamoto_ has quit IRC17:03
*** yamamoto_ has joined #openstack-infra17:04
*** jlanoux_ has quit IRC17:04
*** Guest11037 has quit IRC17:05
*** tosky has joined #openstack-infra17:05
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Fix image caching logic  https://review.openstack.org/29953717:05
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Pre install packages on the instack image  https://review.openstack.org/29953817:05
AJaegerclarkb, pleia2 : Could you review rossella_s's reviewday/neutron change again, please? https://review.openstack.org/#/c/29877917:05
*** yamahata has joined #openstack-infra17:05
*** ZZelle has quit IRC17:05
*** ZZelle has joined #openstack-infra17:06
*** rcernin has quit IRC17:06
*** tqtran has joined #openstack-infra17:06
*** derekh has quit IRC17:06
rossella_sAJaeger, thanks a lot for pushing this forward :)17:07
*** esikachev has quit IRC17:08
*** ihrachys has quit IRC17:08
*** tqtran has quit IRC17:10
*** angdraug has quit IRC17:10
mordredinfra-root, Shrews: tagging shade 1.7.0 unless there are outcries of derision17:13
*** lezbar has quit IRC17:13
fungimordred: outcries only of elation from me17:13
clarkbwe'll just throw groceries if it breaks17:13
fungiand wet goats17:13
*** woodster_ has joined #openstack-infra17:14
*** achanda has joined #openstack-infra17:15
*** rajinir has quit IRC17:15
*** IlyaG has quit IRC17:16
*** kzaitsev_mb has quit IRC17:16
*** sigmavirus24 is now known as sigmavirus24_awa17:17
anteayacan I use them to sop up water from my basement before you toss them?17:20
pabelangerclarkb: ianw: Yup, I am moving from devstack-centos7 to centos-7 now.17:20
pabelangerclarkb: will create a topic17:21
pabelangerclarkb: also, landed a fix for xenial today, checking the status now17:21
clarkbrossella_s: commented, one thing about pathing that I don't expect to work quite right17:21
clarkbpabelanger: I think xenial is waiting for the daily builds to finish then it will start again17:22
pabelangerclarkb: Yup, it should work this time17:22
clarkbok17:22
*** sputnik13_ has joined #openstack-infra17:22
rossella_sclarkb, thanks!17:23
AJaegerpabelanger: are there any changes you need reviews for?17:23
pabelangerclarkb: AJaeger: https://review.openstack.org/#/q/topic:centos-7 for migration from devstack-centos717:23
pabelangergoing to add reset of ansible jobs today too17:23
mordredfungi, clarkb: done17:23
Shrewsmordred: you don't want the server list change in?17:24
*** sputnik13net has joined #openstack-infra17:24
Shrewsmordred: looks like it just needs a recheck17:24
mordredBLASTIT17:24
Shrewsyay 1.8!!!17:25
fungimmmmmmore tags!17:25
mordredsorry. my query to check that everything had been landed was broke17:25
*** daemontool has quit IRC17:25
mordredShrews: it can be a 1.7.1 - it's just fixing a thundering herd17:25
AJaegerclarkb: do you want to review some of pabelanger's changes? Those three have a +2 from me already...17:25
Shrewsyay 1.7.1!17:25
mordredall hail 1.7.117:26
openstackgerritKhai Do proposed openstack-infra/system-config: Replace the Gerrit icon for Workflow -1 vote.  https://review.openstack.org/16522817:26
clarkbAJaeger: yup I will queue those up17:26
mordredclarkb, fungi: I just sent one of my regular emails to the mailing list reminding people to not think that floating ips are the same thing as public ips17:26
clarkbmordred: yup and +1 to changing devstack's defaults even if the "public" range in devstack is unroutable in most devstacks17:26
mordredyup17:27
fungiflublic ips17:27
*** sputnik13 has quit IRC17:27
mordredclarkb: yah - people need to start grokjking th edual model, since it is the defacto standard way opentsack works17:27
*** fawadkhaliq has quit IRC17:28
openstackgerritsebastian marcet proposed openstack-infra/puppet-openstackid: Tweaked server connection settings  https://review.openstack.org/30009717:29
*** rbrndt has quit IRC17:30
*** fawadkhaliq has joined #openstack-infra17:30
*** kzaitsev_mb has joined #openstack-infra17:30
timrcclarkb: In TX we have "BBQ resorts" (or will, soon)... http://austin.eater.com/2016/3/30/11332762/texas-chainsaw-massacre-barbecue-resort-bastrop17:30
mordred++17:31
clarkbpabelanger: what is windmill17:31
rossella_sclarkb, sorry I don't get your comment...what do you mean exactly regarding relying on path resolution? I don't know exactly how all this is installed on the infra server..can you maybe explain me please, thanks!17:31
openstackgerritPaul Belanger proposed openstack-infra/project-config: Switch ansible-role-functional-jobs-centos7 to centos-7  https://review.openstack.org/30010417:31
clarkbrossella_s: src/gerrit-dash-creator is where you have cloned that repo to17:31
zarocrinkle, yolanda : would you be interested in getting puppet code highlighting in gerrit? https://review.openstack.org/#/c/260324/17:31
clarkbrossella_s: but if you don't clone the repo and just pip install it that is not where it will end up17:31
clarkbrossella_s: you can confirm this by running the pip install for it in a virtuaelnv and see how it gets laid on in the virtualenv17:31
pabelangerclarkb: ansible playbook to test some roles I have upstream.  I used to deploy / test my CI changes17:32
*** yonglihe has quit IRC17:32
clarkbpabelanger: ok just making sure we eren't changing things under someone that would be surprised17:32
mnaserfungi: made some changes to the scheduler, new instances should perform better as the importance of compute nodes with less CPU contention was increased17:32
pabelangerclarkb: Ya, I've tested it yesterday and already using bindep for it17:32
mnaserexisting instances might still struggle but new ones should (hopefully) perform much better17:32
*** tiswanso has joined #openstack-infra17:33
fungimnaser: we turn them over pretty rapidly so should see an improvement shortly if so--thanks!17:33
fungimnaser: did that seem to be the case for the uuids i gave you?17:33
AJaegerclarkb: looking at puppet-reviewday, do we pip install at all? I know too little about puppet but wonder how this gets installed...17:34
*** akaszuba has quit IRC17:34
clarkbAJaeger: I would assume so, but let me double check17:34
mnaserfungi: yeah, they were on compute nodes that had a lot of free memory but heavy cpu contention (those are some older compute nodes which was being selected)17:34
anteayaso storyboard folks need to create a gerrit account for their bot to interact with gerrit and storyboard17:35
*** esker has joined #openstack-infra17:35
crinklezaro: ooh neat17:35
anteayato naming options are Codethink StoryBoard Bot or just StoryBoard Bot17:35
anteayaI'm thinking StoryBoard Bot would work17:35
clarkbAJaeger: you are correct it apperas we do not17:35
anteayaany objections?17:35
clarkbAJaeger: but maybe we should17:35
*** jsavak has quit IRC17:35
AJaegerfungi, pabelanger I tried to add an other-requirements file to swift following the nova change - looking at cschwede's comment, I'm a bit confused. Could either of you check this, please? https://review.openstack.org/#/c/298313/517:36
*** jsavak has joined #openstack-infra17:36
rossella_sclarkb, I ran the pip install on the host, not in a virtual env, anyway according to the doc of pip it will be placed in <venv path>/src/17:36
fungianteaya: Zara: no objection from me on "StoryBoard Bot"17:36
anteayafungi: thank you17:36
AJaegerclarkb: I wonder how the dependency like gerrit-dash-creator will get installed if we don't run "pip install"17:36
*** sdague has quit IRC17:36
clarkbAJaeger: it won't :) so we will need ot address that17:37
*** sridhar_ram is now known as sridhar_ram_afk17:37
anteayaZara: give it another few minutes to let anyone else share an opinion if they have one17:37
clarkbrossella_s: ya it should be all relative to the base python install17:37
clarkbrossella_s: it is possible that pkg_resources will give you that rooted path now that I think about it17:38
clarkbrossella_s: since pkg_resources knows about the things that are installed17:38
openstackgerritMerged openstack-infra/project-config: Add centos-7 to bindep-fallback  https://review.openstack.org/29998817:38
*** dimtruck is now known as zz_dimtruck17:38
*** sdague has joined #openstack-infra17:38
mugsieI am seeing some weird errors in gate right now - its looking for traceroute & traceroute6 but they are not installed17:39
mugsieis this a known issue?17:39
*** esker has quit IRC17:39
mugsiehttp://logs.openstack.org/84/296784/1/gate/gate-designate-dsvm-powerdns/c4ca379/console.html17:39
rossella_sclarkb, you mean querying  pkg_resources from the python code to get the path?17:39
pabelangerAJaeger: So, until we add fedora / centos-7 gate checks to other-requirements.txt I see adding support for them as a best effort.17:39
*** tqtran has joined #openstack-infra17:39
openstackgerritMerged openstack-infra/project-config: Add experimental centos-7 puppet-apply  https://review.openstack.org/29340917:39
clarkbrossella_s: yup17:39
pabelangerAJaeger: I'd rather land ubuntu support, and figure out add a bindep job, and follow up patch for fedora support17:39
*** dims_ has quit IRC17:39
AJaegerpabelanger: agreed. The other issue is the double "tox -e bindep" invocation.17:40
fungimugsie: uhoh, we need to revert that. i didn't even think about the possibility17:40
clarkbmugsie: I direct yout to fungi :)17:40
mugsie:)17:40
pabelangerAJaeger: which double tox?17:40
pabelangerthe comments are 2 different OSes, showing different dependencies17:40
*** dims has joined #openstack-infra17:40
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Revert "Traceroute to git.o.o in net-info macro"  https://review.openstack.org/30011117:41
*** zz_dimtruck is now known as dimtruck17:41
fungii'm directly submitting that now17:41
openstackgerritMerged openstack-infra/project-config: Move windmill to centos-7 dib  https://review.openstack.org/29959717:41
AJaegerpabelanger: "So, the error on the first bindep run keeps me from adding a +2 on this yet..."17:41
mugsiefungi: great, thanks17:41
fungiand then i'll manually jjb everything17:41
AJaegerpabelanger: "Interestingly, this was only when running "tox -e bindep" the first time, ie when creating the venv. The second run returned the following outputs"17:41
pabelangerAJaeger: well, that is tox that is failing to compile python libs right?17:41
AJaegerpabelanger: yes17:42
pabelangerSo, tox -edocs would fail too17:42
AJaegerpabelanger: indeed.17:42
pabelangerbasically, gcc is a dependency of our tox17:42
openstackgerritMerged openstack-infra/project-config: Revert "Traceroute to git.o.o in net-info macro"  https://review.openstack.org/30011117:42
fungijjb'ing commencing now17:42
*** mylu has quit IRC17:42
pabelangerAJaeger: So, for that, you need to bootstrap your Os to run tox and tox dependency. For me it is usually, python-pip, gcc17:43
mnaserfungi: is there a way to see average times for a specific commonly ran job?17:43
pabelangerotherwise, tox doesn't work17:43
AJaegerpabelanger: So, is adding "usedevelop = False" a good recommendation? I doubt it.17:43
mnaseri'd like to see if the average run time for a commonly run job is running faster overall17:43
AJaegerpabelanger: could you comment on the review, please? Thanks for thinking this through, yes, this is valid.17:43
pabelangerAJaeger: Ya, not sure about that. I would have somebody else confirm that17:43
pabelangerAJaeger: sure17:43
*** nwkarsten has quit IRC17:44
*** skolekonov is now known as skolekonov|away17:44
Shrewsmordred: oh, wow. the server list change is failing all over the place now17:44
fungiShrews: that may have been me17:44
*** nwkarsten has joined #openstack-infra17:44
Shrewstraceroute: not found17:44
Shrewsfungi: you broke tracing the routes???17:44
fungiyep, i'm undoing it as we speak17:44
fungii did indeed17:45
*** shardy has quit IRC17:45
openstackgerritChris Dent proposed openstack/requirements: Add microversion-parse to requirements  https://review.openstack.org/30011317:46
*** fawadkhaliq has quit IRC17:46
AJaegerfungi, so, first adding the packages to build images, then building new images, then merging again?17:46
*** fawadkhaliq has joined #openstack-infra17:46
dmsimardclarkb: I've been trying different ways of running the fix disk layout but I feel your idea to just take that out of functions.sh and into a jenkins slave script by itself is the right one17:47
fungi#status notice Job failures for missing traceroute packages are in the process of being fixed now, ETA 30 minutes to effectiveness for new jobs17:47
openstackstatusfungi: sending notice17:47
*** fawadkhaliq has quit IRC17:47
*** jsavak has quit IRC17:47
*** IlyaG has joined #openstack-infra17:47
clarkbdmsimard: I wouldn't delete it from devstack-gate to start, first step I would copy it then migrate devstack-gate away from its old thing17:47
*** jsavak has joined #openstack-infra17:47
clarkbor possibly leave it in devstack-gate too I dunno17:47
*** fawadkhaliq has joined #openstack-infra17:47
dmsimardclarkb: I planned on doing a depends-on, but I can do that too17:47
fungiAJaeger: yeah, i think we probably want to add whatever provides traceroute and traceroute6 to our infra-package-needs element17:47
*** fawadkhaliq has quit IRC17:47
pabelangerAh, what happen with traceroute patch?17:48
fungior perhaps in puppet17:48
pabelangererr, oh17:48
*** degorenko is now known as _degorenko|afk17:48
*** fawadkhaliq has joined #openstack-infra17:48
-openstackstatus- NOTICE: Job failures for missing traceroute packages are in the process of being fixed now, ETA 30 minutes to effectiveness for new jobs17:48
fungiminimal images being _minimal_ again17:48
pabelangerAJaeger: comment left17:48
*** fawadkhaliq has quit IRC17:48
AJaegerthanks, pabelanger17:48
fungii'm kicking myself for not confirming those were installed17:49
*** nwkarsten has quit IRC17:49
pabelangerDoh17:49
*** fawadkhaliq has joined #openstack-infra17:49
*** jaosorior has joined #openstack-infra17:49
openstackstatusfungi: finished sending notice17:49
*** fawadkhaliq has quit IRC17:50
*** mriedem has quit IRC17:50
*** shashank_hegde has joined #openstack-infra17:50
*** vgridnev has joined #openstack-infra17:50
*** fawadkhaliq has joined #openstack-infra17:50
openstackgerritPaul Belanger proposed openstack-infra/project-config: Migrate chef jobs to centos-7  https://review.openstack.org/30011417:50
*** mriedem has joined #openstack-infra17:51
AJaegerfungi, see http://packages.ubuntu.com/trusty/amd64/traceroute/filelist17:51
*** mrmartin has joined #openstack-infra17:51
AJaegerI just wonder why it's /usr/sbin/tcptraceroute.db17:51
AJaegerI mean /usr/bin/traceroute.db - still why .db?17:53
pleia2it's just how it is, always been that way as far as I can recall17:53
*** netsin has joined #openstack-infra17:53
AJaegerpleia2: so, traceroute.db instead of traceroute?17:53
*** gyee has joined #openstack-infra17:53
pleia2AJaeger: that's *tcp*traceroute17:54
pleia2regular traceroute is still in /usr/bin17:54
pleia2no .db17:54
openstackgerritPaul Belanger proposed openstack-infra/project-config: Move gate-solum-devstack-dsvm-centos7 to centos-7  https://review.openstack.org/30011717:54
AJaegerpleia2: looking at the filelist, it's not17:54
clarkbmaybe an install script makes a symlink? I can check my local install17:55
*** sridhar_ram_afk is now known as sridhar_ram17:55
fungiwhich executable provides the traceroute command is handled by the alternatives system (at least on debian)17:55
clarkblrwxrwxrwx 1 root root 33 Aug 13  2015 /usr/sbin/traceroute -> /etc/alternatives/traceroute.sbin17:55
clarkband lrwxrwxrwx 1 root root 22 Aug 13  2015 /etc/alternatives/traceroute.sbin -> /usr/bin/traceroute.db17:55
clarkbso ya17:55
AJaegerfungi, clarkb: Ah, thanks!17:56
openstackgerritPaul Belanger proposed openstack-infra/project-config: Move gate-tempest-dsvm-docker-centos7 to centos-7  https://review.openstack.org/30011917:56
fungiyou can't have a /usr/bin/traceroute binary _and_ a symlink at the same path, so the binary has to be renamed17:56
fungii think the .db was for "debian" but i don't really know17:56
pleia2ah yes, thanks fungi17:56
AJaegerfungi: yeah, missed the alternatives17:57
openstackgerritDavid Moreau Simard proposed openstack-infra/project-config: Add fix_disk_layout script and builder  https://review.openstack.org/30012217:57
openstackgerritDavid Moreau Simard proposed openstack-infra/project-config: Ensure VMs used in Packstack jobs have consistent swap configuration  https://review.openstack.org/30012317:57
dmsimardclarkb: ^17:57
*** esikachev has joined #openstack-infra17:58
openstackgerritDoug Hellmann proposed openstack-infra/project-config: upload both sdist and wheel on pre-release  https://review.openstack.org/30012417:58
openstackgerritDavid Moreau Simard proposed openstack-infra/project-config: Add fix_disk_layout script and builder  https://review.openstack.org/30012217:59
dmsimardI nitpicked myself17:59
clarkbtraceroute is an odd thing to worry about alternatives for17:59
openstackgerritPaul Belanger proposed openstack-infra/project-config: Move periodic-package-afs-centos7 to centos-7  https://review.openstack.org/30012617:59
clarkbhow many traceroutes do you need?17:59
fungiwell, there's bsd traceroute, tcptraceroute, i think hping2 might also provide a traceroute tool?18:00
fungiquite a few floating around, probably more i'm forgetting18:00
Clintiputils18:00
fungiyep, that18:00
dmsimardmtr > traceroute :)18:00
pleia2heh18:00
*** jaosorior has quit IRC18:01
pleia2I just went to the movies with the author of tcptraceroute, I should ask him18:02
fungijenkins03 updated in a mere 19 minutes. that was fast18:03
*** zara_the_lemur__ has joined #openstack-infra18:04
mnasercompletetly unrelated but18:04
mnaserjenkins css is a bit messed up :)18:05
fungiyeah, we know18:05
mnaserthe background that says Jenkins CI is under the actaul logo, maybe a missing display: none;18:05
*** kzaitsev_mb has quit IRC18:05
fungii think they tweaked something about their css/dom in the version we just upgraded to18:05
*** kzaitsev_mb has joined #openstack-infra18:06
*** fawadk has joined #openstack-infra18:06
*** jsavak has quit IRC18:06
*** fawadkhaliq has quit IRC18:07
zara_the_lemur__hi, so it's late in the day for me, but we've been talking (again!) about updating storyboard task statuses from gerrit. so some questions18:07
*** jsavak has joined #openstack-infra18:07
fungilemur is such a fun word18:07
zara_the_lemur__fungi: :D I'm named after my long tail of underscores18:08
clarkbfungi: for some reason that made me think of velure which put zapp brannigans voice in my head18:08
fungithere's an awesome lemur habitat and research facility i used to go to a lot back when i lived closer18:08
zara_the_lemur__(the questions are 1) if we make a user acct for a storyboard bot in launchpad, what's the best email to use for it? 2) are we okay to call it 'StoryBoard Bot'? )18:09
pleia2fungi: aw, that's awesome18:09
*** fawadk has quit IRC18:09
fungipleia2: if you're ever around durham, nc you should go18:09
pleia2fungi: noted :)18:09
clarkbzara_the_lemur__: we already have an account for that in launchpad18:09
clarkbzara_the_lemur__: gerrig18:09
*** gyee has quit IRC18:10
clarkbor openstack-hudson18:10
mnaserclarkb: we've made updates/changes to the ceph cluster so these glance image upload issues should be gone (so if you see any failures agian, please do let me know!)18:10
clarkbmnaser: will do thank you18:10
*** SumitNaiksatam has quit IRC18:10
clarkbmnaser: also not sure if you saw that nova bug about the config drive and cinder conflict over the device name, nova is asking if you cna provide the version of libvirt in use there18:10
pabelangerclarkb: ubuntu-xenial build started18:10
fungiclarkb: zara_the_lemur__: it's https://launchpad.net/~hudson-openstack18:10
mnaserclarkb: oh i did not, not sure im subscribed to that bug, would you have a link?18:11
clarkbmnaser: https://bugs.launchpad.net/bugs/155880718:11
openstackLaunchpad bug 1558807 in OpenStack Compute (nova) "Volume attached at different path than reported by nova/cinder when using config drive" [Undecided,Incomplete]18:11
trownAJaeger: I got both present and future PTL sign off on https://review.openstack.org/#/c/299932 :)18:11
*** kzaitsev_mb has quit IRC18:11
fungitrown: the future is now18:11
*** Sukhdev has joined #openstack-infra18:12
trownand also the future18:12
fungithis is true18:12
AJaegertrown: then I have to +2 ;)18:12
trown\o/ thanks18:13
*** devananda has quit IRC18:14
mnaserclarkb: replied, thanks18:14
*** ajmiller has quit IRC18:14
*** gyee has joined #openstack-infra18:14
smarcetfungi: when u get some time, please review https://review.openstack.org/#/c/300097/ :)18:14
*** tqtran has quit IRC18:14
*** ajmiller has joined #openstack-infra18:14
zara_the_lemur__clarkb: fungi: I'm confused; is that capable of updating storyboard task statuses somehow? I think I thought that was for updating launchpad based on a bug's status in another task tracker? or maybe I'm wrong again?18:15
AJaegermordred, fungi, clarkb: Could you review https://review.openstack.org/#/c/300124/ and explain why for pre-releases we only upload wheels but no tarballs to pypi? This seems to be the case since two years now.18:15
*** Sukhdev has quit IRC18:15
fungismarcet: excellent--i was waiting for test results to come back on that second patchset and got sidetracked. approved now so we should see that on the server shortly18:15
smarcetok cool tnx18:15
AJaegerdhellmann: https://review.openstack.org/#/c/56760/ has you as committer and mordred as initial author ;)18:15
mordredAJaeger: it has to do with supportin old pip18:15
mnaserAre the respnose times on this page https://jenkins01.openstack.org/computer/ (and other jenkins) graphed somewhere?  I'd like to see if we can trend the response times for the nodes deployed on our side to see if the scheduler changes we did have improved things18:15
clarkbzara_the_lemur__: it should be able to authenticate to an openid consumer and perform actions18:16
*** dizquierdo has joined #openstack-infra18:16
zara_the_lemur__clarkb: ohh, I see, sorry.18:16
AJaegermordred: is it safe now to change it - or should we -2 300124 ?18:16
fungiAJaeger: it's because versions of pip old enough to know that prereleases are not to be installed without explicitly requesting them are also old enough to know how to find and install wheels by default18:16
mordredAJaeger: old pip that doesn't grok pre-release versions also doesn't grok wheels18:16
mordredI think it's safe now personally18:16
mordredbecuase we just flat do not support versions of pip that old anymore18:16
clarkbmnaser: the openstack health dashboard keeps track of job runtimes18:16
clarkbI think18:17
clarkblet me see if I can find that18:17
*** baoli_ has quit IRC18:17
fungiyeah, it's mainly an issue for people wanting to use default distro-provided pip in ubuntu precise, centos 6, et cetera18:17
mordredwe have job runtimes in graphite18:17
* mordred looks at his private graphs real quick18:17
clarkbyes grafana.openstack.org/dashboard/db/nodepool-vexxhost shows two of the jobs but not a real aggregate18:17
zara_the_lemur__thanks for the help, all :)18:17
dhellmannmordred , AJaeger : if we don't want pre-release sdists we should probably add a comment in the layout file. We hit this with trove-dashboard, and the release team needs to sort out whether UI plugins are "libraries" or something else, and if they should even be doing release candidates like this, but it wasn't obvious why we had separate job definitions there18:17
mordreddhellmann: I think we should just release all the things now18:18
mordreddhellmann: we waited long enough18:18
dhellmannmordred : wfm. AJaeger, I should update the comment in my new patch, let me do that.18:18
*** FallenPegasus has joined #openstack-infra18:18
AJaegerdhellmann: and remove the colon18:18
*** jcoufal_ has quit IRC18:19
dhellmannAJaeger : well spotted, yes18:19
*** sigmavirus24_awa is now known as sigmavirus2418:20
AJaegerclarkb, fungi, mordred: Nakato has updated the change to use zuul-cloner in post pipeline. It's quite a change in usage IMHO so once you have some spare time, please review whether the changed semantics are what we need - and the code as well: https://review.openstack.org/29319418:20
mnaseryeah, the nodepool grafana just shows those 2 jobs which don't seem to run all that often on our side18:20
*** zul has quit IRC18:20
clarkbmnaser: ya so you could take those graphs and add additional ones for all the jobs whihc I assume is what mordred has hidden away18:20
pabelangermordred: still planning on nodepool-shade upgrade today?18:21
clarkbmnaser: http://status.openstack.org/openstack-health/#/g/node_provider/ymq-1?groupKey=node_provider&resolutionKey=hour&duration=P7D&end=2016-03-31T18:17:10.558Z doesn't appear to have the runtimes there18:21
clarkbso doing like what mordred has may be best (and maybe we add that to grafana)18:22
*** zul has joined #openstack-infra18:22
openstackgerritDoug Hellmann proposed openstack-infra/project-config: upload both sdist and wheel on pre-release  https://review.openstack.org/30012418:22
dhellmannAJaeger, mordred : ^^18:22
*** sigmavirus24 is now known as sigmavirus24_awa18:22
fungiAJaeger: on 298313 for swift, it might make sense to wait for https://review.openstack.org/200672 to merge to nova first. would like to get lifeless to review that too and see if he has suggestions for how we should be better leveraging bindep (maybe with a "test" profile for some packages or something?). that was a minimal "here's what i think it looks like" patch18:23
*** sigmavirus24_awa is now known as sigmavirus2418:24
fungisufficient to pass tests, maybe still needs polish before we make it a recommendation18:24
*** bpokorny has quit IRC18:24
*** dims_ has joined #openstack-infra18:24
*** yolanda has quit IRC18:24
*** dims has quit IRC18:24
AJaegerfungi, ok. let's point both reviews out to lifeless then ;)18:25
AJaegerthanks, fungi18:25
openstackgerritMerged openstack-infra/puppet-openstackid: Tweaked server connection settings  https://review.openstack.org/30009718:25
*** rfolco has joined #openstack-infra18:25
fungii think the concerns raised by cschwede on 298313 may also apply to 20067218:25
*** salv-orlando has quit IRC18:26
fungias it looks like i missed that we need to add usedevelop = False there too so as to prevent trying to do a develop install of nova18:26
*** FallenPegasus is now known as MarkAtwood18:26
AJaegerfungi: yep18:26
fungiwhen i add that, it's way faster18:26
clarkbpabelanger: I think xenial failed, trying to figure out where things are in this log18:27
fungiwaaaaay faster18:27
*** devananda has joined #openstack-infra18:27
fungiupdating now18:27
*** jiggalanator has quit IRC18:27
fungicschwede: thanks for spotting that!18:27
*** mrmartin has quit IRC18:27
*** yolanda has joined #openstack-infra18:27
pabelangerclarkb: still using openjdk-718:28
clarkbpabelanger: or maybe I am misreading logs? so much information to process18:28
pabelangerhmm18:28
clarkbah yup there it is18:28
pabelangerchecking logs why18:28
*** nwkarsten has joined #openstack-infra18:29
*** nwkarsten has quit IRC18:29
clarkbfungi: wait develop should be faster18:29
*** fawadkhaliq has joined #openstack-infra18:29
fungiclarkb: why?18:29
clarkbodd that it slows stuff down18:29
mordredI agree with clarkb18:29
mordreddevelop avoids the need to make a tarball18:29
clarkbfungi: because it doens't make an sdist it just symlinks some stuff around18:29
fungiclarkb: usedevelop = True causes it to pip install the repo18:29
mordredand just makes an egg-link in site-packages18:29
*** nwkarsten has joined #openstack-infra18:29
clarkbfungi: tox should always pip install the repo18:30
mordred++18:30
fungioh, hrm. double-checking18:30
clarkbnot using develop is fine, its just odd that that would make stuff faster18:30
mordredmight want to make sure the skipsdist = True is in there too18:30
AJaegerpabelanger: question on your comment at https://review.openstack.org/#/c/300126/1/jenkins/jobs/infra.yaml18:30
mordredotherwise you'd buidling that sdist AND then doing a develop install18:30
mordredfungi: ^^18:31
AJaegerpabelanger: I doubt there's a other-requirments file that we could add the content to - or can we?18:31
fungiclarkb: mordred: it most decidedly does _not_ install nova18:31
fungiif i add usedevelop = False18:31
fungiaccording to .tox/bindep/bin/pip list18:31
mordredfungi: what thing are we looking at so that I can look at?18:31
fungimordred: just a sec and let me push up this new patchset18:32
*** zul has quit IRC18:32
AJaegermordred:  https://review.openstack.org/298313 and https://review.openstack.org/20067218:32
AJaegermordred: fungi is updated 20067218:32
clarkbmnaser: I might be blind but I don't see any comment on https://bugs.launchpad.net/nova/+bug/155880718:32
openstackLaunchpad bug 1558807 in OpenStack Compute (nova) "Volume attached at different path than reported by nova/cinder when using config drive" [Undecided,Incomplete]18:32
*** zul has joined #openstack-infra18:32
pabelangerclarkb: odd: https://review.openstack.org/#/c/299948/ didn't merge18:33
fungiAJaeger: mordred: clarkb: https://review.openstack.org/200672 is now commented and updated accordingly18:33
AJaegerpabelanger: what is ubuntu-centos7? ;)18:34
pabelangerAJaeger: .trolo a fail by me :)18:34
fungisounds both scary and awesome18:34
mordredan awesome new thing18:34
mordredsort of like Ubuntu Windows 1018:34
pabelangerzing!18:34
pleia2hah18:34
*** kushal has quit IRC18:35
pabelangerAJaeger: which review, I can fix18:35
clarkbmordred: wow18:36
AJaegerpabelanger: https://review.openstack.org/30011718:37
openstackgerritPaul Belanger proposed openstack-infra/project-config: Move periodic-package-afs-centos7 to centos-7  https://review.openstack.org/30012618:37
mordredfungi: OH18:38
mordredfungi: I get it now18:38
openstackgerritPaul Belanger proposed openstack-infra/project-config: Move gate-solum-devstack-dsvm-centos7 to centos-7  https://review.openstack.org/30011718:38
*** esikachev has quit IRC18:38
mordredfungi: with skipsdist AND usedevelop=False then nova does not get installed, which is what you are, in fact, looking for here18:38
*** cdent has quit IRC18:38
pabelangerAJaeger: both fixed18:39
*** piet has quit IRC18:39
fungimordred: should i explicitly skipdist in that env just to be extra safe future-proofing and for the benefit of people cargo-culting examples?18:39
mordredskipsdist is a global not an env setting18:39
fungioh, okay18:39
mordreda comment might be useful though18:39
fungidefinitely18:39
*** piet has joined #openstack-infra18:39
AJaegermordred: do you want to +2 dhellmann's pre-release change ? https://review.openstack.org/#/c/30012418:40
*** bpokorny has joined #openstack-infra18:40
*** annegentle has quit IRC18:40
lifelessfungi: +1 on a test profile18:40
*** bpokorny has quit IRC18:40
fungi"note that this needs tox.skipdist = True to be effective"18:40
*** zara_the_lemur__ has quit IRC18:40
lifelessfungi: I shall look in more detail later today - I have meetingish things end to end this morning though18:40
*** annegentle has joined #openstack-infra18:40
fungilifeless: more hoping you have a concrete example of what that would look like, and what we should be doing to make sure we include the default profile for the platform in addition to the test profile in our scripts18:41
*** bpokorny has joined #openstack-infra18:41
fungilifeless: so any time you have to play with it is much appreciated!18:41
fungilifeless: extra context can be found in the script we use to run bindep: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/install-distro-packages.sh18:41
fungilikely we'd need to improve that for specific additional profiles18:42
fungii mostly am wanting to make sure we're following your vision for how the tool should work and evolve as it was your idea to begin with18:43
anteayachannel ops meeting-alt and meeting-3 need a topic refresh18:43
fungianteaya: awesome, on it18:43
anteayathanks18:43
fungidone!18:43
lifelessfungi: ack thanks; have tabs open for later18:44
fungithanks!18:45
jrollAJaeger: https://review.openstack.org/300146 for ironic upper-constraints use18:45
jrollif you'd like a look18:45
anteayafungi: thank you18:46
fungiyou're welcome18:46
pabelangerfungi: what was that package again to improve apt-get performance, but at the cost of corruption?18:46
fungipabelanger: there are several options, but eatmydata was one18:47
pabelangerYes, that was it18:47
pabelangervexxhost is pretty slow ATM: https://jenkins03.openstack.org/job/gate-puppet-jenkins-puppet-syntax/29/console18:47
clarkbits likely we won't see much speedups with eatmydata18:47
fungii've been pondering whether we should play with it in some of our jobs, though there's the possibility it could have other side effects18:47
clarkbI have tested that in the past and I think most/all our clouds already do unsafe disk io for the VMs18:47
fungipabelanger: slow still? mnaser though he fixed it18:47
clarkbpabelanger: fungi how old is the VM that is running on?18:48
openstackgerritMerged openstack-infra/project-config: Add tripleo-quickstart project to tripleo team  https://review.openstack.org/29993218:48
pabelangerfungi: Oh, missed that18:48
fungiclarkb: that's what i'm checking18:48
pabelangerclarkb: let me check18:48
clarkbI suppose its possible vexxhost isn't doing unsafe io and that is why it is slower18:48
fungiand getting a uuid for it18:48
pabelanger1a050c99-7d4c-4e45-bbbc-da917382ad8618:49
pabelangeronly 11:12 mins old18:49
fungithanks pabelanger!18:49
fungimnaser: ^18:49
*** kushal has joined #openstack-infra18:49
AJaegerjroll: LGTM.18:50
jrollthanks18:50
mnaserthanks for the uuid pabelanger18:51
mnaserthis is strange, ~2% iowait on the host node, ~70% idle cpu on the host node18:51
pabelangermnaser: hack the gibson18:51
mnaserpabelanger: how possible is it to not let jenkins kill that machine and get access to it?18:52
pabelangermnaser: fungi can put a hold18:52
clarkbpabelanger: you can too18:52
mnaserthe IO is fine on the actual compute node.18:53
pabelangerclarkb: not sure syntax :(18:53
mnaser%Cpu(s): 27.3 us,  3.7 sy,  0.0 ni, 66.7 id,  2.1 wa,  0.0 hi,  0.2 si,  0.0 st18:53
*** cody-somerville has joined #openstack-infra18:53
clarkbpabelanger: sudo -H -u nodepool nodepool hold 925815618:53
pabelangerclarkb: done, thanks18:53
mnasernow the 2nd request which i think is a bit harder, would it be possible to get access to the instance?18:53
clarkbmnaser: yup18:53
mnaseri want to do some basic io check (im curious to know if it could be a network issue?)18:53
pabelangerjust need a public SSH key18:54
mnasersure, 1 secon18:54
mnaserpabelanger: http://paste.openstack.org/show/492650/18:54
pabelangermnaser: ssh jenkins@162.253.54.21018:55
mnasermerci18:55
*** tqtran has joined #openstack-infra18:57
clarkbpabelanger: then when mnaser is done with the host you can s/hold/delete/ (assuming jenkins is also done with it)18:57
pabelangerclarkb: sure18:57
mnaseroh got iostat looks awful on that node18:58
AJaegerfungi: to make it perfect ;) shouldn't it be "further relies on tox.skip..."?18:58
fungi_on_ yes. *sigh* thanks!18:59
pleia2mnaser: btw created re: jenkins css https://storyboard.openstack.org/#!/story/200053418:59
pleia2^^ and everyone else, I added low-hanging-fruit tag, because tags are a thing now!18:59
AJaegerpabelanger's centos-7 changes look fine to me - anybody else wants to review some more? https://review.openstack.org/#/q/topic:centos-718:59
mnaseriostat on compute node has w_await of around 3ms, inside the virtual machine around ~150ms18:59
clarkbsee I am not getting email about that, not surethat the project group subscription is sufficient, will have to play with it more18:59
clarkbalso tags19:00
*** jordanP has quit IRC19:00
clarkbpleia2: wow thats an old plugin19:01
pleia2clarkb: yep19:01
pleia2we likely need someone to find us a new one to use or something19:02
pleia2it's a very good newcomer task19:02
fungimnaser: that definitely sounds suboptimal by a couple orders of magnitude19:02
mnaseryeah.. but I have no idea why. :(19:02
mnaserso im investigating to see what's going on19:02
*** MarkAtwood has quit IRC19:02
fungimnaser: feel free to do anything you like to/in/with that instance. we don't need it for anything, just let us know when you're done so we can delete it19:02
mnaserthere's no cpu contention on the hypervisor (~75% idle cpu), almost no iowait on the hypervisor, but it's struggling inside the VM only19:03
clarkbmnaser: we had io issues with infracloud and SpamapS thought it was related to the kvm disk caching19:03
*** openstackgerrit has quit IRC19:03
mnaserwe use file=none so no caching19:03
*** openstackgerrit has joined #openstack-infra19:04
*** maishsk has quit IRC19:04
mnaserSSDs are fast enough, + the raid controllers we use usally have ~2GB of DDR3 memory write-back cache19:04
SpamapSmnaser: no hypervisor<->VFS layer caching, but do you have BBWC?19:04
mnaserso overall we've noticed faster performance by taking this path19:04
SpamapSthat's why19:04
mnaseryes19:04
SpamapSand SSD, oh yeah19:04
*** annegentle has quit IRC19:04
mnaserSpamapS: compute node has almost 0 iowait and ~2-3ms w_await19:05
openstackgerritKhai Do proposed openstack-infra/puppet-gerrit: Allow puppet to configure Gerrit's gitHttpUrl option  https://review.openstack.org/30016019:05
SpamapSno reason at all to mess with VFS's idiosyncracies if you have a few GB of BBWC and SSD19:05
mnaserthe VMs inside are sitting at 150ms w_await ..19:05
*** annegentle has joined #openstack-infra19:05
mnaser40 core not, ~70% idle cpu (across all 40 cores)19:05
mnaseroh this is interesting19:06
mnaseravgqu-sz is almost stuck at 119:06
mnaseras if it's literally limited at 119:06
*** matt-borland has joined #openstack-infra19:07
*** austin81 has joined #openstack-infra19:07
*** maishsk has joined #openstack-infra19:07
*** cdent has joined #openstack-infra19:10
*** Sukhdev has joined #openstack-infra19:10
*** yamamoto_ has quit IRC19:10
mugsiefungi: how long should it take for jenkins to catch up and remove the traceroutes?19:13
pleia2fungi: before brain gets stale, transferring my zuul restart notes into proper docs in the system-config zuul.rst, unless there's a reason I shouldn't19:13
clarkbpleia2: go for it, you might want to add a section about clearing out the used nodes in nodepool too, or someone can add that on top of your change19:13
pleia2clarkb: right! I will make a quick comment about it, and perhaps someone else can flesh it out more19:14
anteayapleia2: awesome19:14
*** annegentle has quit IRC19:14
*** ociuhandu has quit IRC19:15
openstackgerritAustin Clark proposed openstack-infra/project-config: Create stackviz element and script in nodepool (WIP)  https://review.openstack.org/27931719:15
clarkbemail so much email19:16
fungimugsie: that should have taken effect at least 30 minutes ago. are you still seeing failures?19:17
*** [1]Thelo has joined #openstack-infra19:17
fungipleia2: sounds great19:17
mugsieyeah - https://jenkins06.openstack.org/job/gate-designate-dsvm-bind9/622/console19:17
mugsiestarted 15 mins ago19:17
*** Thelo has quit IRC19:19
*** [1]Thelo is now known as Thelo19:19
mugsiefungi: https://jenkins06.openstack.org/job/gate-shade-dsvm-functional-neutron/262/console started a min ago, it is hitting it as well.19:19
*** rbrndt has joined #openstack-infra19:20
*** LaT0rtue has quit IRC19:21
clarkbwe turned off rax-ord right?19:21
clarkband that was what the traceroute change was going to help debug?19:22
AJaegerclarkb: yes19:22
fungiclarkb: yes and yes19:22
pabelangerYa, ord is still disabled I believe19:22
fungimugsie: that is disconcerting if me pulling project-config and manually rerunning jenkins-jobs update on our jenkins masters didn't update the job configs19:22
clarkbfungi: is rax unabe to do their own tracerouting between networks they own?19:23
clarkbor are we just trying to get as much data as possible so that we can point back ath the lcoud?19:23
fungiclarkb: apparently. a list of affected uuids means nothing when it comes to them troubleshooting their network19:23
clarkbwow ok19:23
sdaguehttp://logs.openstack.org/35/299535/7/gate/gate-grenade-dsvm/ade68d4/console.html19:23
fungithey asked us if we could collect some traceroutes if we end up deciding to use ord again19:23
openstackgerritMerged openstack-infra/puppet-jenkins: Ubuntu xenial supports openjdk-8-jdk  https://review.openstack.org/29994819:23
sdaguealso, apparently traceroute... not in the osic image19:23
bnemectripleo is still hitting the traceroute problem as of a few minutes ago too: http://logs.openstack.org/42/300142/1/check-tripleo/gate-tripleo-ci-f22-nonha/382ada6/console.html19:23
*** annegentle has joined #openstack-infra19:24
fungisdague: it's not in our dib images period (we upload the same image everywhere)19:24
mugsienearly make me miss hpcloud :)19:24
sdaguefungi: um, ok, well it's very early used in devstack-gate19:24
sdagueso... fun hard fail?19:24
sdagueI just had 3 jobs insta fail on that here - https://review.openstack.org/#/c/299535/719:25
sdaguehttp://logs.openstack.org/35/299535/7/gate/gate-grenade-dsvm-multinode/5ea58a3/console.html19:25
sdaguehttp://logs.openstack.org/35/299535/7/gate/gate-tempest-dsvm-full/a95f209/console.html19:25
sdaguehttp://logs.openstack.org/35/299535/7/gate/gate-grenade-dsvm/ade68d4/console.html19:25
pabelangerfungi: do you have a check up to add traceroute into dib elements?19:25
fungisdague: yes, we reverted it a while ago and i personally applied the reverted configuration on all our jenkins masters. i'm trying to figure out why jjb updates aren't apparently effective19:25
fungipabelanger: no, i haven't decided yet if that should be dib or puppet. i'd like to get the original fallout behind us before i even think about trying again19:26
pabelangerfungi: ack19:26
fungiwhat gives? it's definitely still in that job config on jenkins0619:26
*** jamielennox|away is now known as jamielennox19:26
fungii a good minute after it merged, i did `cd /etc/project-config ; sudo git pull --ff-only`19:27
fungiand then `time sudo -H jenkins-jobs update --delete-old /etc/jenkins_jobs/config`19:27
clarkbfungi: jjb uses /etc/jenkins_jobs19:27
fungiand it completed, with no errors19:27
clarkbso that pull is not sufficient19:27
fungiugh19:27
clarkbneed puppet to run first19:28
fungiwhat am i missing that normally moves the files into /etc/jenkins_jobs?19:28
*** dtardivel has quit IRC19:28
clarkbpuppet19:28
fungii guess it's puppeted by now. i'll double-check that19:28
fungiseems to be19:30
*** sileht has quit IRC19:30
fungiupdating everywhere _again_19:30
pabelangerclarkb: okay, made it passed know error, but apparently openjdk is hard coded in puppet-jenkins, fixing19:30
*** apoorvad has quit IRC19:30
clarkbpabelanger: meaning https://review.openstack.org/#/c/299948/2 was insufficient?19:31
pabelangerclarkb: right, I didn't see the openjdk-7 stuff below19:31
pabelangerand gate didn't catch it because of missing ubuntu-xenial image19:31
*** piet has quit IRC19:32
*** vgridnev has quit IRC19:32
*** nwkarsten has quit IRC19:32
*** piet has joined #openstack-infra19:33
*** ociuhandu has joined #openstack-infra19:37
*** maishsk has quit IRC19:39
*** krtaylor has quit IRC19:40
*** binbincong has quit IRC19:41
openstackgerritPaul Belanger proposed openstack-infra/puppet-jenkins: Don't update-java-alternatives on ubuntu-xenial  https://review.openstack.org/30018619:41
pabelangerclarkb: okay, going to test that now ^19:41
*** binbincong has joined #openstack-infra19:41
smarcetfungi: i ran several load tests against dev openstackid and now handles 2k concurrent connections :)19:42
fungismarcet: that's excellent! well double of what you were shooting for19:42
pabelangerclarkb: that could can likely be removed in the future, as it looks to only be added because of bare-* nodes with had openjdk 6 installed19:42
smarcetfungi: however , jpmaxman and jimmy wants to wait till apple approves the mobile app19:42
smarcetfungi: and then move to production19:43
*** ociuhandu has quit IRC19:43
clarkbpabelanger: yup I think so19:43
fungismarcet: just let me know and i'm happy to work on upgrading the production server whenever you're all redy19:43
*** sdake_ has joined #openstack-infra19:43
fungiready19:43
clarkbpabelanger: once bare-precise is gone?19:43
pabelangerclarkb: Yup, keep forgetting about that little guy19:44
matt-borlanddoug-fish, I wanted to solve the OPENSTACK_IMAGE_BACKEND issue you pointed out in Image Edit, but had a problem.19:44
*** sdake has quit IRC19:44
smarcetfungi: sure, perhaps we could do more improvements on the meantime to puppet script, i would like to put a watch dog for redis like ps-watcher or something19:44
pabelangerokay, new build going19:44
*** ociuhandu has joined #openstack-infra19:44
pabelangerwill test with local nodepool too19:44
* matt-borland was in the wrong room :(19:44
smarcetfungi: and try to tweak a little redis config too19:44
openstackgerritKhai Do proposed openstack-infra/system-config: Change download command to point to git.o.o  https://review.openstack.org/30018719:45
fungismarcet: okay, though definitely want to be able to reenable puppet on production soon, which at this point we'll only be able to do when we can replace it with a 14.04 server19:45
*** mrmartin has joined #openstack-infra19:45
smarcetfungi: i think that next week we would have some news from apple19:46
smarcetand then we could move forward on that19:46
*** dizquierdo has quit IRC19:46
*** Swanson has joined #openstack-infra19:46
fungismarcet: okay, sounds good19:46
fungithanks!19:46
*** mrmartin has quit IRC19:47
*** dizquierdo has joined #openstack-infra19:48
fungimugsie: sdague: the couple of jenkins masters which aren't done updating with the revert have been set to not start any new jobs until they're fixed, so should be safe to recheck things now19:50
*** SumitNaiksatam has joined #openstack-infra19:50
pabelangermnaser: let me know once you are done with the slave we held19:50
mnaserpabelanger: sorry, will do, i'm at loss at what's going on19:51
*** tonytan4ever has quit IRC19:52
fungiokay, jenkins01 and jenkins02 updates have completed so i've taken them back out of prepare for shutdown19:53
fungi#status notice Any jobs which erroneously failed on missing traceroute packages should be safe to recheck now19:53
openstackstatusfungi: sending notice19:53
pabelangermnaser: no rush, just a reminder19:54
*** nwkarsten has joined #openstack-infra19:54
*** nwkarsten has quit IRC19:54
-openstackstatus- NOTICE: Any jobs which erroneously failed on missing traceroute packages should be safe to recheck now19:54
*** ihrachys has joined #openstack-infra19:54
mugsiefungi: thanks!19:55
*** nwkarsten has joined #openstack-infra19:55
fungimugsie: no thanks necessary. just cleaning up the mess i made for 3k developers19:55
mugsieit happens :)19:56
openstackstatusfungi: finished sending notice19:56
*** sdake_ has quit IRC19:56
*** dizquierdo has quit IRC19:57
*** briancline has quit IRC19:58
*** mylu has joined #openstack-infra19:58
*** briancline has joined #openstack-infra19:58
*** Cormite has joined #openstack-infra20:00
*** tphummel has quit IRC20:00
*** esikachev has joined #openstack-infra20:01
pabelangerclarkb: actually, I have no idea how to get diskimage-builder to pick up that puppet-jenkins commit.  We don't have code exposed to change puppet modules20:01
pabelangerunless I quickly hack modules.env to use that sha120:02
clarkbpabelanger: yes I think you have to hack modules.env20:02
*** mrmartin has joined #openstack-infra20:02
clarkband there is code in our dib elements to use a specific system config ref20:02
*** MarkAtwood has joined #openstack-infra20:02
pabelangeryup20:03
openstackgerritAndrew Woodward proposed openstack-infra/project-config: Move puppet-ceph under puppet-openstack ACL  https://review.openstack.org/30019220:04
openstackgerritPaul Belanger proposed openstack-infra/system-config: DNM - testing ubuntu-xenial  https://review.openstack.org/30019320:04
*** esikachev has quit IRC20:06
*** amrith is now known as _amrith_20:06
*** salv-orlando has joined #openstack-infra20:08
*** Sukhdev has quit IRC20:09
*** briancline has quit IRC20:10
*** yamamoto_ has joined #openstack-infra20:11
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Always apt-get update after replacing sources  https://review.openstack.org/30000520:12
*** sridhar_ram1 has joined #openstack-infra20:14
*** SumitNaiksatam has quit IRC20:15
*** sridhar_ram has quit IRC20:15
*** baoli has joined #openstack-infra20:15
*** sileht has joined #openstack-infra20:15
*** baoli_ has joined #openstack-infra20:15
*** dprince has quit IRC20:16
openstackgerritDoug Wiegley proposed openstack-infra/devstack-gate: Fail the test if the pre test hook fails  https://review.openstack.org/29217520:17
*** briancline has joined #openstack-infra20:17
*** SumitNaiksatam has joined #openstack-infra20:17
mnaserpabelanger can i shutdown and start the instance or will that likely destroy it?20:18
*** kgiusti has left #openstack-infra20:18
clarkbmnaser: that should be fine20:19
mnaseralright, thanks!20:19
*** baoli has quit IRC20:19
*** yamamoto_ has quit IRC20:20
*** abregman is now known as abregman|afk20:20
fungiyeah, reboots work fine in this situation20:22
fungiwe've basically told nodepool hands-off with that instance until we say otherwise20:22
*** piet has quit IRC20:23
*** tphummel has joined #openstack-infra20:23
mnaserfungi: the instance can be killed, we've found an oddity which is worked around.20:23
*** piet has joined #openstack-infra20:23
fungimnaser: oh? great!20:23
mnaserif it happens again (it could, on another compute node, but not as likely), we'll check20:23
fungii'll let pabelanger handle switching it to delete at his convenience20:23
mnaserbut if we get another slowdown that spawns on another compute node, then we know that it was the reason for the problem20:24
fungimnaser: thanks for investigating--well keep an eye out20:24
mnaserwhats the best way to keep an eye on this?20:24
fungimnaser: unfortunately we mostly find out about it when someone notices a job run too long and/or hit a configured timeout. which can sometimes be for legitimate reasons too. we don't directly track it20:24
clarkbyou can query logstash for timed out jobs in vexxhost20:25
clarkblet me get a query together for that20:25
mnaserthat would be useful keep up20:25
fungiyep, that would yield _likely_ (but not guaranteed) recurrences20:25
pabelangermnaser: fungi okay, deleted20:25
fungiwould still need manual vetting to see if they're the general everything-is-slow symptom, or more like an indefinite hang on some operation20:26
*** briancline has quit IRC20:26
fungithe former is what we'd been witnessing sometimes in vexxhost recently20:26
fungiteh latter can be due to a variety of external causes20:26
*** fawadkhaliq has quit IRC20:27
mnaseryep, that makes sense20:27
clarkbmnaser: message:"Build timed out" AND node_provider:"ymq-1" AND filename:"console.html"20:27
*** fawadkhaliq has joined #openstack-infra20:27
clarkbmnaser: you can query that on logstash.openstack.org using kibana or hit the elasticsearch api directly at http://logstash.openstack.org/elasticsearch as the api root20:27
mnaserbuild_node in there is super useful20:28
*** krtaylor has joined #openstack-infra20:28
*** ihrachys has quit IRC20:28
clarkbya that will give you a specific instance which we can track back via nodepool/nova20:28
*** apoorvad has joined #openstack-infra20:28
mnaserso we can just query the db for the uuid (but build_node_uuid would be nice too :D)20:28
*** sridhar_ram1 has quit IRC20:29
clarkbmnaser: unfortunately the instance uuids are not exposed to anything but nodepool so hard to pass that along automatically20:29
clarkbat least I think it isn't passed along anywhere20:29
fungiright, the nova instance uuid unfortunately isn't something passed through at the job layer20:29
mnaserfungi gr20:29
mnaseractually more a grrr towards us20:30
fungii end up tracking it back in nodepool's debug log20:30
mnaserthe build timed out in logstash for the last 7 days20:30
mnaseri think it will be quite easy to see when its fied20:30
clarkbmnaser: and in theory ouy could us that info to see if there were other affected hypervisors20:30
*** piet has quit IRC20:31
mnaseryeah the name is fine, we can get the uuid easily after20:31
fungiyeah, at least those build_node names are the corresponding nova instance names\20:31
fungiso it's something to key from20:31
mnaserunless you randomize the instance name before deleting it #nextgensecurity20:31
fungiheh20:31
*** bpokorny has quit IRC20:32
*** stevemar has quit IRC20:33
clarkbgrun timeout of 0 apepars to have made it into our images ok20:33
clarkb*grub20:33
*** stevemar has joined #openstack-infra20:33
clarkbso everything should be 10 seconds faster20:34
clarkbin aggregate thats pretty awesome20:34
*** bpokorny has joined #openstack-infra20:34
*** rossella_s has quit IRC20:34
*** rossella_s has joined #openstack-infra20:35
*** nwkarsten has quit IRC20:35
*** kzaitsev_mb has joined #openstack-infra20:36
*** moravec has quit IRC20:38
pabelangerokay, how do we checkout a review.o.o ref with modules.env?20:38
clarkbpabelanger: it should take any name it can fetch/checkout. I now realize the hard part is the fetch20:39
pabelangerYa, not sure how to do what with the existing code20:40
clarkbI think the "easy" solution is put it on not gerrit, then make a branch or tag for it20:40
clarkband point modules.env at that20:40
pabelangerya, will have to do that20:40
clarkbcan be local to your build host or on github or whatever20:41
*** asettle has joined #openstack-infra20:42
*** fawadkhaliq has quit IRC20:42
openstackgerritPaul Belanger proposed openstack-infra/system-config: DNM - testing ubuntu-xenial  https://review.openstack.org/30019320:43
*** ayoung has quit IRC20:43
*** briancline has joined #openstack-infra20:43
pabelangerclarkb: okay, testing again.20:44
pabelangerwould be nice to have zuul-clone support for our elements :)20:44
clarkbpabelanger: one of the reasons its designed this way is to minimize bootstrap deps20:45
clarkbpabelanger: you need bash and git20:45
*** fawadkhaliq has joined #openstack-infra20:45
pabelangertrue20:46
pabelangerhowever, I think zuul is installed before we run puppet20:46
clarkbpabelanger: it shouldn't be20:46
clarkband its not installed on any of our real servers20:46
clarkbexcept for zuul and zuul mergers20:46
ianwclarkb: i'm pretty convinced that centos-7 runs in the same amount of time as the image-based builds, so we can swap out the devstack jobs.  this is just based on min() of runs, because variability is quite high20:47
clarkbpart of that was when I attempted to use librarian and broke everything due to its crazy idea of deps20:47
pabelangerclarkb: Ah, it was pip I seen20:47
clarkbianw: ya20:47
fungiclarkb: pabelanger: well, with dib we install it into /usr/zuul-env so that we have access to zuul-cloner on our images20:47
clarkbfungi: ya but our puppet runs without dib20:48
pabelangerfungi: Ya. I think it is done after puppet, but ya20:48
clarkbbasically its doable20:48
fungianyway, i've broken enough things for one day, so am going to buzz off and find early dinner20:48
ianwwe know something about everything failing with "traceroute not found" right?20:48
clarkbwe just need to be careful that it works for more than slave builds20:48
clarkbianw: yes20:48
fungiianw: it shouldn't be any longer20:48
fungiis it still?20:48
*** smarcet has quit IRC20:48
*** |-paul-| has joined #openstack-infra20:48
pabelangerianw: clarkb: here are the current reviews open to move to centos-7: https://review.openstack.org/#/q/status:open+project:openstack-infra/project-config+branch:master+topic:centos-720:48
pabelangerleaving puppet until the last20:48
pabelangerlikely do kolla in the next day20:49
ianwfungi: probably not, doing rechecks.  sorry, just coming online here20:49
fungijobs should have been completely updated with the revert by about 19:50 utc20:49
fungiso for about an hour20:49
fungiback later20:49
*** rcarrillocruz has left #openstack-infra20:50
*** rcarrillocruz has joined #openstack-infra20:50
*** jsavak has quit IRC20:53
mgagnecould someone review those changes? They are changes to Gerrit UI https://review.openstack.org/#/q/status:open+project:openstack-infra/system-config+branch:master+topic:gerrit-ui20:53
mgagneI've been using them with tampermonkey for a couple of months without problem20:54
*** IlyaG has quit IRC20:54
ianwmgagne: maybe drop a screenshot on imgur, might make it easier for reviewers to visualise?20:55
mgagnethere are some already20:55
*** rfolco has quit IRC20:56
mgagneexcept for max-width, can add this one20:56
*** stevemar_ has joined #openstack-infra20:56
*** knikolla has quit IRC20:56
*** stevemar has quit IRC20:56
ianwheh, that was the one i looked at :)21:00
*** edmondsw has quit IRC21:00
mgagnedone21:00
*** mylu has quit IRC21:00
*** asettle has quit IRC21:01
*** asettle has joined #openstack-infra21:01
*** sputnik13_ has quit IRC21:01
clarkbmnaser: http://logs.openstack.org/80/299980/1/gate/gate-puppet-openstack-integration-3-scenario003-tempest-ubuntu-trusty/15db1b1/console.html is possibly another slow node that happened ~12 minutes ago21:01
*** aysyd has quit IRC21:02
*** sputnik13 has joined #openstack-infra21:03
openstackgerritDavid Moreau Simard proposed openstack-infra/project-config: Ensure VMs used in Packstack jobs have consistent swap configuration  https://review.openstack.org/30012321:04
*** Goneri has quit IRC21:04
*** Cormite has quit IRC21:07
*** thorst has quit IRC21:07
*** trown is now known as trown|outtypewww21:07
*** thorst has joined #openstack-infra21:08
*** cdent has quit IRC21:08
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/shade: Follow name_or_id pattern on domain operations  https://review.openstack.org/28871521:10
*** mylu has joined #openstack-infra21:12
*** abregman|afk has quit IRC21:12
*** thorst has quit IRC21:12
*** thorst has joined #openstack-infra21:14
*** piet has joined #openstack-infra21:14
*** ldnunes has quit IRC21:15
rcarrillocruzmordred, fungi, clarkb, jeblair_: i'm off a couple days, be back on Sunday. Let me know what you find on the ansible launch role. I'll continue adding some enhancements I have in mind on Monday and potentially prepare another blogpost with an example of a deploy with similar config to what you folks do in the various providers21:15
anteayarcarrillocruz: have fun21:16
rcarrillocruzthx anteaya :-)21:16
anteayawelcome :)21:16
*** aeng has joined #openstack-infra21:16
*** baoli_ has quit IRC21:17
*** Sukhdev has joined #openstack-infra21:18
*** mrmartin has quit IRC21:18
*** thiagop-afk has quit IRC21:18
*** piet has quit IRC21:18
*** eharney has quit IRC21:18
*** thorst has quit IRC21:19
*** achanda has quit IRC21:22
*** mylu has quit IRC21:23
*** baoli has joined #openstack-infra21:24
openstackgerritMerged openstack-infra/project-config: Switch ansible-role-functional-jobs-centos7 to centos-7  https://review.openstack.org/30010421:26
openstackgerritIan Wienand proposed openstack-infra/project-config: Switch devstack job to centos-minimal  https://review.openstack.org/30022021:26
*** matt-borland has quit IRC21:27
ianwpabelanger: ^ maybe we should do that before switching other dsvm jobs21:27
*** mylu has joined #openstack-infra21:27
ianwpabelanger: also, did you skip the anvil jobs, or just not got to yet?21:27
SwansonSo. traceroute stuff still broken?21:28
*** kzaitsev_mb has quit IRC21:28
ianwharlowja: ^21:28
*** knikolla has joined #openstack-infra21:28
ianwSwanson: should not be21:28
*** amitgandhinz has quit IRC21:29
harlowjawhats up21:29
harlowjau guys can switch over the anvil jobs21:29
harlowjai need to work on that stuff sometime soon anyway21:29
*** sdake has joined #openstack-infra21:30
SwansonI just rechecked a patch and got traceroute6 and traceroute not found.21:31
*** imcsk8 is now known as imcsk8|afk21:31
ianwSwanson: yeah, me too.  i wonder if jjb is a bit stuck updating things21:31
Swansonianw, so long as people know it is still broken (and I saw the message saying it wasn't an hour or two ago so maybe they don't) I'm happy.21:32
ianwharlowja: cool, yeah i don't think it will affect them.  you can quote me on that :)21:32
harlowjaianw all good :)21:32
*** asselin_ has quit IRC21:32
*** EricGonc_ has joined #openstack-infra21:32
openstackgerritMerged openstack-infra/project-config: Migrate chef jobs to centos-7  https://review.openstack.org/30011421:33
openstackgerritElizabeth K. Joseph proposed openstack-infra/system-config: Add Zuul restart documentation  https://review.openstack.org/30022321:34
*** esker has joined #openstack-infra21:34
pabelangerianw: WFM21:35
pabelangerianw: Ya, haven't done it yet21:35
*** mriedem has quit IRC21:35
*** nwkarsten has joined #openstack-infra21:36
*** esker has quit IRC21:36
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/shade: Follow name_or_id pattern on domain operations  https://review.openstack.org/28871521:36
*** nwkarsten has quit IRC21:36
*** EricGonczer_ has quit IRC21:36
*** esker has joined #openstack-infra21:37
*** dasm has joined #openstack-infra21:38
*** fawadkhaliq has quit IRC21:38
*** imcsk8|afk is now known as imcsk821:39
*** fawadkhaliq has joined #openstack-infra21:39
*** _ari_ has quit IRC21:40
*** fawadkhaliq has quit IRC21:40
*** fawadkhaliq has joined #openstack-infra21:41
*** _ari_ has joined #openstack-infra21:42
openstackgerritPaul Belanger proposed openstack-infra/system-config: DNM - testing ubuntu-xenial  https://review.openstack.org/30019321:42
*** fawadkhaliq has quit IRC21:44
*** fawadkhaliq has joined #openstack-infra21:44
*** _ari_ has quit IRC21:44
*** asselin_ has joined #openstack-infra21:45
*** ybathia has joined #openstack-infra21:45
*** gyee has quit IRC21:46
*** tojuvone_ has joined #openstack-infra21:47
*** angdraug has joined #openstack-infra21:48
*** gyee has joined #openstack-infra21:48
*** |-paul-| has quit IRC21:49
*** burgerk has quit IRC21:49
*** tojuvone has quit IRC21:51
openstackgerritPaul Belanger proposed openstack-infra/puppet-jenkins: Don't update-java-alternatives on ubuntu-xenial  https://review.openstack.org/30018621:53
*** dasm has left #openstack-infra21:53
pabelangerclarkb: crinkle: I think that might be the better approach ^21:54
*** denisra has joined #openstack-infra21:55
*** mylu has quit IRC21:56
*** sigmavirus24 is now known as sigmavirus24_awa21:56
*** krtaylor has quit IRC21:57
crinklepabelanger: i liked the first approach better, i don't get the new one21:57
*** yolanda has quit IRC21:59
clarkbpabelanger: ya the new one will still try to update alternative on xenial21:59
*** xyang1 has quit IRC22:01
*** amotoki has joined #openstack-infra22:01
*** nmagnezi has quit IRC22:02
rcarrillocruzhmm22:06
rcarrillocruzfolks22:06
rcarrillocruzyou seeing tracerout6 missing package errors on tests22:06
rcarrillocruz?22:06
rcarrillocruzhttps://jenkins07.openstack.org/job/gate-shade-dsvm-functional-nova/276/console22:06
clarkbrcarrillocruz: yes was already removed22:07
*** yolanda has joined #openstack-infra22:07
clarkbbasically rax asked us for traceroute data so we added it to the jobs but minimal images don't have traceroute22:07
clarkbthings went boom so we removed it from the jobs22:07
ianwi wonder if it's jenkins07 ... that's where i saw a recent failure22:07
ianwhttps://jenkins07.openstack.org/job/gate-tempest-dsvm-neutron-full/26238/console22:07
rcarrillocruzso recheck ?22:08
clarkbrcarrillocruz: yes22:08
rcarrillocruzcool thx22:08
clarkbianw: I think fungi had to redo the cleanup22:09
SwansonI did a recheck after it was claimed to be fixed and it failed again.22:09
*** fbo_ has quit IRC22:09
clarkbSwanson: after it was claimed to be fixed the first time or the second time?22:09
* clarkb logs in to see the config of that job22:10
Swansonclarkb, 2016-03-31 21:24:44.059 | /tmp/hudson6254995475921326830.sh: 23: /tmp/hudson6254995475921326830.sh: traceroute6: not found22:10
*** Goneri has joined #openstack-infra22:10
SwansonIf that helps22:10
*** icey has quit IRC22:10
*** fbo has joined #openstack-infra22:10
clarkbhuh its still there I guess fungi didnt manage to get it reconfigured on 0722:11
clarkbI will run that now22:11
*** resker has joined #openstack-infra22:11
ianwyeah, everything i'm seeing fail in devstack queue is jenkins0722:12
*** resker has quit IRC22:12
clarkbjjb is running there now22:12
clarkbI will also put it in shutdown mode22:12
*** esker has quit IRC22:13
*** icey has joined #openstack-infra22:13
clarkbit will stop running new jobs now22:13
Swansonrecheck then?22:13
*** esker has joined #openstack-infra22:13
*** nwkarsten has joined #openstack-infra22:13
*** nwkarsten has quit IRC22:14
mnaserdoes anyone know if ubuntu vms will prefer ipv6 over ipv4 (or at least the build machines?)22:14
clarkbyes now a recheck should be fine22:14
clarkbmnaser: yes they should22:14
mnaserhmm22:14
mnaserim trying to find other builds of the same type22:14
*** nwkarsten has joined #openstack-infra22:14
mnaserand how long the git clone process has taken22:14
clarkbmnaser: the software running in them at least22:14
clarkbyes git definitely will22:14
pabelangerclarkb: crinkle: The issue is, I think we need a conditional anchor, not to be present when we do xenial... otherwise, the update-alt will be called22:14
mnaseri think there has been issues with ipv6 at rackspace?22:15
clarkbpabelanger: won't it get mad the anchor doesn't exist?22:15
mnaserso if it tries to contact git.openstack.org via ipv622:15
craigeo/22:15
pabelangerclarkb: that is what I am testing, but I can wrap it with and if statement again too22:15
*** thorst has joined #openstack-infra22:16
openstackgerritPaul Belanger proposed openstack-infra/puppet-jenkins: Don't update-java-alternatives on ubuntu-xenial  https://review.openstack.org/30018622:17
*** tpsilva has quit IRC22:17
*** erlon has quit IRC22:17
clarkb07 claims no jobs need updating so its got corrupt cache, we can fix that22:18
*** annegentle has quit IRC22:18
SwansonSee you in hell, 07!  Thanks for looking into this!22:19
*** annegentle has joined #openstack-infra22:20
rcarrillocruz++22:20
crinklepabelanger: i don't really think the anchor matters, the update-java-alternatives exec gets called unconditionally, it just happens after the anchor22:21
crinkleit's requiring the anchor, not subscribing to it22:21
ianwno jenkins07, i expect you to die!22:25
*** andymaier has quit IRC22:25
*** pasquier-s has quit IRC22:26
*** sridhar_ram has joined #openstack-infra22:27
*** bpokorny_ has joined #openstack-infra22:28
*** bpokorny_ has quit IRC22:30
*** bpokorny_ has joined #openstack-infra22:30
*** dimtruck is now known as zz_dimtruck22:31
*** bpokorny has quit IRC22:32
openstackgerritMerged openstack-infra/irc-meetings: Adding AsiaPac Regional PWG Meeting  https://review.openstack.org/29977622:33
*** regXboi has quit IRC22:33
pabelangercrinkle: Ya, still running a test but for what ever reason, it was still calling the update-java exec, regardless of my conditional logic. Will report back in 10mins22:35
*** tphummel has quit IRC22:36
*** esker has quit IRC22:36
openstackgerritPaul Belanger proposed openstack-infra/puppet-jenkins: Don't update-java-alternatives on ubuntu-xenial  https://review.openstack.org/30018622:37
clarkbwe are up to gate-hacking now22:37
clarkb(it is alphabetical22:37
*** thorst has quit IRC22:37
*** tphummel has joined #openstack-infra22:38
*** baoli has quit IRC22:38
pabelangerYa, this openjdk is frustrating.... must be on 12th DIB build22:38
*** rossella_s has quit IRC22:38
*** ybathia has quit IRC22:38
clarkbjust thinking out loud, the if else around the anchor shouldn't be necessary22:40
clarkbits just that the anchor is only used on the exec side if on older ubuntu22:41
*** annegentle has quit IRC22:41
*** annegentle has joined #openstack-infra22:42
pabelangerclarkb: okay, testing again22:44
*** bpokorny_ has quit IRC22:44
ianwpabelanger: now you see why f23 took so long :)22:44
pabelangerianw: no doubt22:44
*** bpokorny has joined #openstack-infra22:44
openstackgerritPaul Belanger proposed openstack-infra/puppet-jenkins: Don't update-java-alternatives on ubuntu-xenial  https://review.openstack.org/30018622:44
pabelangerI think the issue is the before => Anchor syntax22:45
*** ayoung has joined #openstack-infra22:45
pabelangerhow can that be true, if it is conditional22:45
clarkban anchor is just a noop reasource to affect ordering22:46
clarkbso before anchor should be fine22:46
*** nwkarste_ has joined #openstack-infra22:47
openstackgerritThomas Goirand proposed openstack-infra/project-config: Add a job to build openstack-pkg-tools  https://review.openstack.org/29402222:47
*** nwkarste_ has quit IRC22:47
zigomordred: Around? I still didn't see you reviewing this patch, and it's been opened for 2 weeks.22:48
zigomordred: I wonder if you really want to do something with this PTL hat that you've asked for...22:49
*** nwkarsten has quit IRC22:49
*** sdague has quit IRC22:50
*** akshai has quit IRC22:51
*** tsg has joined #openstack-infra22:51
*** annegentle has quit IRC22:53
*** sdake has quit IRC22:57
crinkleclarkb: http://logs.openstack.org/31/300131/2/check/gate-tempest-dsvm-full/379c52e/console.html failed on the traceroute thing just a few minutes ago, not sure if this was jenkins07 or a different issue?22:57
clarkbif it started before I put jenkins07 in shutdown mode it could be that one22:58
clarkbbut if it failed a few minutes ago then its like a different server :(22:58
clarkbI will look for it22:58
crinkle2016-03-31 21:43:34.302 | Started by user anonymous22:58
openstackgerritPaul Belanger proposed openstack-infra/system-config: DNM - testing ubuntu-xenial  https://review.openstack.org/30019322:59
clarkbya you have to look at each of hte masters because we don't record the master in the job but maybe we can/should22:59
clarkbcrinkle: https://jenkins07.openstack.org/job/gate-tempest-dsvm-full/31842/console it was on 0723:00
crinkleclarkb: okay, so i'll recheck23:01
clarkbyup should be fine, I haven't seen any recent fails for that specific job on the other masters23:01
*** akshai has joined #openstack-infra23:02
*** austin81 has left #openstack-infra23:03
*** akshai_ has joined #openstack-infra23:03
*** ams_ has quit IRC23:03
*** sputnik13 has quit IRC23:05
*** akshai has quit IRC23:06
pabelangerfinally23:08
pabelangerokay, https://review.openstack.org/#/c/300186/ appears to work if I manually run it23:09
clarkbpabelanger: got it to work?23:09
pabelangerjust waiting for DIB to finish23:09
*** sdake has joined #openstack-infra23:09
pabelangermy issue was a typo in my modules.env file and becuase install_modules.sh doesn't have set -e it was not bombing on error23:09
clarkbit has my +223:09
clarkb07 update is into the periodic jobs23:09
pabelangerlet me confirm with this nodepool dib first23:09
pabelangergetting a drink first23:10
clarkbpabelanger: ok, we should get that merged if the rest of the build is happy though23:10
clarkbso that we stop looping around and failing23:10
*** knikolla has quit IRC23:10
*** annegentle has joined #openstack-infra23:10
pabelangerclarkb: Yup, in about 5mins here23:11
*** salv-orlando has quit IRC23:11
*** rbrndt has quit IRC23:11
openstackgerritJay Pipes proposed openstack/gertty: py3k: Prevent stacktrace opening links in browser  https://review.openstack.org/30024723:12
*** sdake_ has joined #openstack-infra23:12
pabelangerianw: my only concern about 300126, if we remove install-distro-packages, it will almost be an empty slave.23:13
*** sdake has quit IRC23:13
*** amotoki has quit IRC23:13
pabelangerianw: but I can remove it23:13
pabelangerclarkb: okay, lets merge it. This dib is slow23:14
ianwpabelanger: i don't think it matters much, but i think the dependencies are all in the job23:14
clarkbpabelanger: feel free to self approve you have my +223:15
pabelangerclarkb: ok23:15
pabelangerianw: okay, removingt23:15
clarkbI am going to approve ianw's centos-7 dsvm job switch as soon as 07 is done updating its jobs23:15
openstackgerritPaul Belanger proposed openstack-infra/project-config: Move periodic-package-afs-centos7 to centos-7  https://review.openstack.org/30012623:15
pabelangerclarkb: ack23:15
pabelangerwe should be able to do puppet jobs over the weekend23:16
pabelangersince it will be a big JJB update23:16
pabelangereverything else we can finish off tomorrow23:16
clarkb07 is done udpating so I have put it back into the rotation23:19
clarkbI have also approved the centos-7 dsvm change23:19
*** ybathia has joined #openstack-infra23:19
*** nwkarsten has joined #openstack-infra23:19
pabelangerwe can likely do https://review.openstack.org/#/c/300119/ and https://review.openstack.org/#/c/300117/ too for centos-723:22
* clarkb looks23:23
clarkbI mostly want to be around to field problmes if we break solum or nova-docker23:24
* pabelanger nods23:24
*** nwkarsten has quit IRC23:24
clarkbI figure ianw will be around to debug his change23:24
clarkbetc23:24
*** ayoung has quit IRC23:26
*** annegentle has quit IRC23:26
*** pradk has quit IRC23:26
clarkbpabelanger: you going to approve your puppet change for xenial?23:27
pabelangerclarkb: ya, I can do that now23:27
*** aeng has quit IRC23:27
*** ybathia has quit IRC23:27
ianwyeah, we should do dsvm stuff in regular hours23:27
openstackgerritMerged openstack-infra/project-config: Switch devstack job to centos-minimal  https://review.openstack.org/30022023:28
*** niska has quit IRC23:30
lifelessfungi: ugh, current release of bindep has clamped pbr dep :(23:31
lifelessCollecting pbr!=0.7,<1.0,>=0.6 (from bindep)23:31
mgagneI should have started using shade a long time ago :O23:31
*** gordc has quit IRC23:32
*** tosky has quit IRC23:32
*** sridhar_ram has quit IRC23:34
*** tphummel has quit IRC23:34
*** tphummel_ has joined #openstack-infra23:35
*** niska has joined #openstack-infra23:35
openstackgerritMerged openstack-infra/puppet-jenkins: Don't update-java-alternatives on ubuntu-xenial  https://review.openstack.org/30018623:35
*** mriedem has joined #openstack-infra23:36
*** thorst has joined #openstack-infra23:38
*** bpokorny_ has joined #openstack-infra23:41
*** bpokorny has quit IRC23:41
*** bpokorny_ has quit IRC23:41
*** bpokorny has joined #openstack-infra23:42
*** sdake_ has quit IRC23:43
lifelessfungi: AJaeger: feedback on those changes given23:43
*** aeng has joined #openstack-infra23:43
*** thorst has quit IRC23:46
pabelanger2016-03-31 23:50:39,232 INFO nodepool.image.build.ubuntu-xenial: Notice: Finished catalog run in 161.85 seconds23:50
pabelanger\o/23:51
pleia2pabelanger: exciting!23:51
pabelangerpleia2: Indeed! Also, I would like to file my complaint with ubuntu for removing openkjdk-7 after I did all my work to make it work with our manifests23:52
pleia2complaint filed ;)23:53
pabelanger:D23:53
*** knikolla has joined #openstack-infra23:53
openstackgerritMerged openstack/diskimage-builder: ironic-python-agent should use console output  https://review.openstack.org/29550123:54
pleia2it's in main, so I suspect they didn't want to drag support for it along in another LTS (I don't blame them)23:54
* pabelanger nods23:55
pabelangerjust surprising they waited this long to remove it23:55
pleia2indeed23:55
*** SumitNaiksatam has quit IRC23:55
clarkbright after beta 2 or something23:55
*** tsg has quit IRC23:55
pleia2they also swapped out the gui software installer quite late for something that barely works, so23:56
*** moravec has joined #openstack-infra23:56
clarkbooh fun23:56
clarkbI have already decided that I won't be upgrding and will be tumbleweeding at some point soon23:56
pleia2yeah, switched to gnome-software from the rickety old ubuntu software center written in python223:56
clarkbat least on the laptop which doens't have a giant nvidia card that needs to run23:57
pabelangerAh, I heard that23:57
*** ajmiller has quit IRC23:58
*** Qiming has joined #openstack-infra23:58
anteayaI have no problem tumbleweeding myself, as long as I am able23:59
clarkbmostly I like the idea of a continuously updated and gated distro23:59
clarkbsince thats sort of what we do here for openstack23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!