Friday, 2017-08-18

*** slaweq_ has quit IRC00:00
*** rhallisey has quit IRC00:02
*** Swami has quit IRC00:07
*** SumitNaiksatam has quit IRC00:08
*** darvon has joined #openstack-infra00:10
*** thorst has joined #openstack-infra00:11
*** masuberu has joined #openstack-infra00:17
*** masber has quit IRC00:20
*** tjones has left #openstack-infra00:21
*** EricGonczer_ has quit IRC00:28
*** EricGonczer_ has joined #openstack-infra00:30
*** dave-mccowan has joined #openstack-infra00:30
*** claudiub|3 has quit IRC00:30
*** felipemonteiro has joined #openstack-infra00:31
*** xarses_ has quit IRC00:37
*** baoli has joined #openstack-infra00:49
*** marst has quit IRC00:50
*** esberglu has quit IRC00:53
*** baoli has quit IRC00:53
*** slaweq_ has joined #openstack-infra00:56
*** Julien-zte has quit IRC00:57
*** bobh has quit IRC00:58
*** Julien-zte has joined #openstack-infra01:01
*** slaweq_ has quit IRC01:01
*** xarses_ has joined #openstack-infra01:07
*** bobh has joined #openstack-infra01:08
*** andreww has joined #openstack-infra01:10
*** xarses_ has quit IRC01:11
*** zhurong has joined #openstack-infra01:11
*** gildub has quit IRC01:12
*** vhosakot has joined #openstack-infra01:13
*** andreww has quit IRC01:14
*** liujiong has joined #openstack-infra01:14
*** esberglu has joined #openstack-infra01:21
*** phenixrizen has quit IRC01:23
*** niska has quit IRC01:23
*** phenixrizen has joined #openstack-infra01:23
*** zzzeek has quit IRC01:23
*** zzzeek has joined #openstack-infra01:24
*** r-daneel has quit IRC01:24
*** niska has joined #openstack-infra01:24
*** tonyb has quit IRC01:24
*** zzzeek has quit IRC01:25
*** sflanigan has quit IRC01:25
*** tonyb has joined #openstack-infra01:25
*** zzzeek has joined #openstack-infra01:25
*** esberglu has quit IRC01:26
pabelangermordred: clarkb: config-drive is true in feature/zuulv3 now01:26
*** vhosakot has quit IRC01:26
*** masber has joined #openstack-infra01:34
*** masber has quit IRC01:34
*** masuberu has quit IRC01:34
*** thorst has quit IRC01:36
flwangcan anybody give me an idea why there is no doc job for our repo https://review.openstack.org/#/c/493407/ ? Thanks a lot01:36
*** numans has joined #openstack-infra01:37
pabelangerflwang: check zuul/layouts.yaml in openstack-infra/project-config. That will list the jobs you are setup to use01:39
*** markvoelker has joined #openstack-infra01:41
*** sflanigan has joined #openstack-infra01:42
*** sflanigan has joined #openstack-infra01:42
*** gildub has joined #openstack-infra01:49
*** puiterwijk|cld has quit IRC01:49
*** puiterwijk|cld has joined #openstack-infra01:49
*** felipemonteiro has quit IRC01:51
*** dave-mccowan has quit IRC01:56
*** cuongnv has joined #openstack-infra01:59
*** Apoorva has quit IRC02:02
*** Apoorva has joined #openstack-infra02:02
*** EricGonczer_ has quit IRC02:05
*** EricGonczer_ has joined #openstack-infra02:05
*** Apoorva has quit IRC02:07
*** Apoorva has joined #openstack-infra02:08
*** hongbin has joined #openstack-infra02:10
*** Apoorva has quit IRC02:12
ianwi don't belive the vexxhost mirror had AAAA records before02:13
ianwnot sure if intentional ,but does now, so will monitor for a bit02:13
*** markvoelker has quit IRC02:15
*** Hal has joined #openstack-infra02:20
*** Hal is now known as Guest7218902:21
*** iyamahat has quit IRC02:21
*** rlandy has quit IRC02:26
*** gouthamr has quit IRC02:27
*** Guest72189 has quit IRC02:28
*** SumitNaiksatam has joined #openstack-infra02:40
*** bobh has quit IRC02:54
*** slaweq_ has joined #openstack-infra02:59
*** slaweq_ has quit IRC03:03
openstackgerritKien Nguyen proposed openstack-infra/project-config master: Switch capability_scope to global  https://review.openstack.org/49477403:04
*** andreww has joined #openstack-infra03:04
*** EricGonczer_ has quit IRC03:07
*** xarses_ has joined #openstack-infra03:07
*** xarses_ has quit IRC03:08
*** andreww has quit IRC03:08
*** xarses_ has joined #openstack-infra03:10
*** markvoelker has joined #openstack-infra03:12
pabelangerianw: clarkb: https://review.openstack.org/#/c/433738/ interesting bug in devstack about 502 proxy errors. Might be related to infracloud03:14
pabelangerbug 168186603:15
openstackbug 1681866 in devstack "Bad response code while validating token: 502" [Undecided,New] https://launchpad.net/bugs/168186603:15
*** felipemonteiro has joined #openstack-infra03:22
*** felipemonteiro has quit IRC03:24
*** nicolasbock_ has joined #openstack-infra03:25
ianwpabelanger: we sort of did that didn't we03:28
ianwhttps://review.openstack.org/#/c/491339/03:28
*** bobh has joined #openstack-infra03:28
pabelangerianw: ya, does seem to be doing anything now on xenial03:29
pabelangerthere are treads on google about keepalives too03:30
pabelangermight bump logging in one infracloud region tomorrow to trace, and see if I can see anything03:30
pabelangeralso seen a 502 in sto2 to buildlogs too03:31
ianwyeah, it was unclear ... the best reference i found was the apache doc03:34
*** thorst has joined #openstack-infra03:36
*** sree has joined #openstack-infra03:36
ianwthe other things is we could probably go back to event_mpm, which i tihnk is default03:37
ianwget back closer to standard03:38
*** thorst has quit IRC03:40
*** sflanigan has quit IRC03:41
*** udesale has joined #openstack-infra03:42
*** markvoelker has quit IRC03:46
*** gildub has quit IRC03:46
clarkbin devstack the problem was client side03:47
*** ihrachys has quit IRC03:47
clarkbrequests was trying to send a request down a connection the server was in thr act of closing03:48
clarkband ratgerthan send a new request on new connection it just failed03:48
clarkbits possible apache has a similar problem in mod proxy when it makes request as  client to buildlogs03:48
*** masber has joined #openstack-infra03:49
*** hongbin has quit IRC03:50
*** bobh has quit IRC03:52
*** masber has quit IRC03:53
*** xarses_ has quit IRC03:55
*** sflanigan has joined #openstack-infra03:57
*** sflanigan has joined #openstack-infra03:57
*** gongysh has joined #openstack-infra04:11
*** ykarel has joined #openstack-infra04:13
*** links has joined #openstack-infra04:23
*** eventingmonkey has quit IRC04:29
*** iyamahat has joined #openstack-infra04:36
*** makowals has quit IRC04:37
openstackgerritOpenStack Proposal Bot proposed openstack-dev/hacking master: Updated from global requirements  https://review.openstack.org/44002904:38
openstackgerritOpenStack Proposal Bot proposed openstack-dev/pbr master: Updated from global requirements  https://review.openstack.org/49478804:38
openstackgerritOpenStack Proposal Bot proposed openstack-infra/shade master: Updated from global requirements  https://review.openstack.org/49478904:38
openstackgerritOpenStack Proposal Bot proposed openstack/diskimage-builder master: Updated from global requirements  https://review.openstack.org/49480004:39
*** gongysh has quit IRC04:43
*** markvoelker has joined #openstack-infra04:43
*** gongysh has joined #openstack-infra04:43
*** makowals has joined #openstack-infra04:43
*** gongysh has quit IRC04:44
*** pbourke has quit IRC04:47
*** dhajare has joined #openstack-infra04:49
openstackgerritOpenStack Proposal Bot proposed openstack/os-client-config master: Updated from global requirements  https://review.openstack.org/49486504:49
*** pbourke has joined #openstack-infra04:49
flwangpabelanger: thanks a lot04:50
*** gildub has joined #openstack-infra04:52
*** eventingmonkey has joined #openstack-infra04:54
*** pgadiya has joined #openstack-infra04:55
*** adisky__ has joined #openstack-infra04:55
*** slaweq_ has joined #openstack-infra05:01
*** cuongnv has quit IRC05:01
openstackgerritKien Nguyen proposed openstack-infra/project-config master: Kuryr: Setup multinode gate  https://review.openstack.org/49494605:03
openstackgerritMerged openstack-infra/tripleo-ci master: Set undercloud nameserver to localhost 127.0.0.1  https://review.openstack.org/49454505:05
*** slaweq_ has quit IRC05:05
*** masber has joined #openstack-infra05:09
*** markvoelker has quit IRC05:16
*** isaacb has joined #openstack-infra05:27
openstackgerritMerged openstack-infra/system-config master: Remove internap-nyj01  https://review.openstack.org/49230105:30
*** kjackal has quit IRC05:35
*** zhurong has quit IRC05:38
openstackgerritIan Wienand proposed openstack-infra/system-config master: Rename new infracloud, citycloud and vexxhost mirrors  https://review.openstack.org/49472805:40
openstackgerritIan Wienand proposed openstack-infra/system-config master: Move RAX mirrors to numbered hosts  https://review.openstack.org/49437005:40
*** zhurong has joined #openstack-infra05:41
*** esberglu has joined #openstack-infra05:42
*** jbadiapa has joined #openstack-infra05:44
*** ekcs has joined #openstack-infra05:45
*** esberglu has quit IRC05:47
*** ekcs has left #openstack-infra05:50
*** xarses_ has joined #openstack-infra05:50
*** xarses has quit IRC05:51
*** aeng has quit IRC05:54
*** slaweq_ has joined #openstack-infra06:02
*** SumitNaiksatam has quit IRC06:02
*** SumitNaiksatam has joined #openstack-infra06:03
*** markvoelker has joined #openstack-infra06:13
*** isaacb has quit IRC06:15
*** gongysh has joined #openstack-infra06:16
*** gongysh has quit IRC06:19
*** LindaWang has joined #openstack-infra06:19
openstackgerritIan Wienand proposed openstack/diskimage-builder master: [DNM] testing selinux  https://review.openstack.org/49496006:20
*** ykarel has quit IRC06:26
*** LindaWang has quit IRC06:26
*** ykarel has joined #openstack-infra06:32
*** slaweq_ has quit IRC06:35
openstackgerritSlawek Kaplonski proposed openstack-infra/shade master: DNM Test force cinder volume delete  https://review.openstack.org/49470506:36
*** rcernin has joined #openstack-infra06:41
*** amoralej|off is now known as amoralej06:45
*** markvoelker has quit IRC06:46
*** jpich has joined #openstack-infra06:52
*** gongysh has joined #openstack-infra07:00
*** iyamahat has quit IRC07:00
*** LindaWang has joined #openstack-infra07:02
*** kjackal has joined #openstack-infra07:06
*** markus_z has joined #openstack-infra07:07
*** baoli has joined #openstack-infra07:10
*** eroux has quit IRC07:12
*** tesseract has joined #openstack-infra07:13
*** sflanigan has quit IRC07:13
*** baoli has quit IRC07:14
*** claudiub|3 has joined #openstack-infra07:14
*** alexchadin has joined #openstack-infra07:15
*** MasterofJOKers_ has joined #openstack-infra07:15
*** pgadiya has quit IRC07:16
*** MasterofJOKers has quit IRC07:17
*** MasterofJOKers_ is now known as MasterofJOKers07:17
*** ccamacho has joined #openstack-infra07:17
*** namnh has joined #openstack-infra07:18
*** namnh has quit IRC07:18
*** namnh has joined #openstack-infra07:19
*** namnh has quit IRC07:20
*** namnh has joined #openstack-infra07:20
*** jpena|off is now known as jpena07:24
*** pgadiya has joined #openstack-infra07:29
*** esberglu has joined #openstack-infra07:30
*** pcaruana has quit IRC07:32
*** slaweq_ has joined #openstack-infra07:32
*** florianf has joined #openstack-infra07:33
*** esberglu has quit IRC07:34
openstackgerritMarkos Chandras (hwoarang) proposed openstack-infra/project-config master: zuul: layout: OSA: Make openSUSE Leap 42.3 an official gate job  https://review.openstack.org/49498507:36
*** egonzalez has joined #openstack-infra07:40
*** jbadiapa_ has joined #openstack-infra07:43
*** markvoelker has joined #openstack-infra07:43
openstackgerritMerged openstack-infra/tripleo-ci master: Revert "Installs openssh packages that try to workaround sshd start problem"  https://review.openstack.org/49403907:43
*** ykarel has quit IRC07:45
*** ykarel has joined #openstack-infra07:45
*** jbadiapa has quit IRC07:45
*** e0ne has joined #openstack-infra07:46
*** jbadiapa_ has quit IRC07:51
*** liujiong has quit IRC07:51
*** jbadiapa_ has joined #openstack-infra07:51
*** liujiong has joined #openstack-infra07:53
*** e0ne has quit IRC07:53
*** pcaruana has joined #openstack-infra07:54
*** e0ne has joined #openstack-infra07:55
*** jtomasek has joined #openstack-infra07:57
*** liusheng has quit IRC07:58
*** liusheng has joined #openstack-infra08:04
*** slaweq_ has quit IRC08:04
*** e0ne has quit IRC08:05
*** e0ne has joined #openstack-infra08:06
*** liujiong has quit IRC08:08
*** dmellado has quit IRC08:10
*** dmellado has joined #openstack-infra08:10
*** derekh has joined #openstack-infra08:12
*** nicolasbock_ has quit IRC08:14
*** markvoelker has quit IRC08:16
*** ralonsoh has joined #openstack-infra08:21
*** iyamahat has joined #openstack-infra08:25
*** masber has quit IRC08:26
*** lucas-afk is now known as lucasagomes08:27
openstackgerritColleen Murphy proposed openstack-infra/puppet-accessbot master: Depend on helper gem for spec_helper_acceptance  https://review.openstack.org/49511908:28
*** gongysh has quit IRC08:30
*** camunoz has joined #openstack-infra08:35
*** bnemec has quit IRC08:39
*** bnemec has joined #openstack-infra08:40
*** mugsie has quit IRC08:41
*** Adri2000 has quit IRC08:42
*** mugsie has joined #openstack-infra08:43
*** mugsie has quit IRC08:43
*** mugsie has joined #openstack-infra08:43
*** baoli has joined #openstack-infra08:43
gibihi! It seems we have stuck jobs in the CI preventing the check queue to move forward.08:44
gibiif you look at the top of the queue the stuck job links does not point to a log folder but point to a telnet ip+port address08:45
*** gongysh has joined #openstack-infra08:46
*** baoli has quit IRC08:48
*** mugsie has quit IRC08:52
*** iyamahat has quit IRC08:53
*** slaweq_ has joined #openstack-infra09:03
fricklergibi: the CI is heavily loaded, what you describe is normal behaviour when the job is still running, you can use the link to see the live output of the job's console log09:04
*** adreznec has quit IRC09:08
*** adreznec has joined #openstack-infra09:11
*** markvoelker has joined #openstack-infra09:14
gibifrickler: ohh OK, I didn't know that09:16
gibifrickler: thanks09:16
*** jascott1 has quit IRC09:17
*** jascott1 has joined #openstack-infra09:18
*** esberglu has joined #openstack-infra09:18
*** ykarel is now known as ykarel|lunch09:18
*** jascott1 has quit IRC09:22
*** esberglu has quit IRC09:22
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config master: Normalize projects.yaml  https://review.openstack.org/49513409:23
*** tosky has joined #openstack-infra09:23
openstackgerritRosario Di Somma proposed openstack-infra/shade master: WIP: Use direct calls to get_<resource>_by_id  https://review.openstack.org/49453509:24
openstackgerritMerged openstack-infra/tripleo-ci master: Set the undercloud_docker_registry_mirror for upstream jobs  https://review.openstack.org/49464409:25
openstackgerritMerged openstack-infra/tripleo-ci master: Create a whitelist for /etc configs  https://review.openstack.org/49397309:26
*** Adri2000 has joined #openstack-infra09:30
*** Adri2000 has joined #openstack-infra09:30
*** sambetts|afk is now known as sambetts09:31
*** rtjure has joined #openstack-infra09:32
*** ykarel|lunch has quit IRC09:32
*** ykarel|lunch has joined #openstack-infra09:33
*** pgadiya has quit IRC09:35
*** slaweq_ has quit IRC09:35
*** shardy has joined #openstack-infra09:39
*** jbadiapa_ has quit IRC09:41
openstackgerritFlavio Percoco proposed openstack-infra/project-config master: Split COE scenarios: 1 for each COE  https://review.openstack.org/49427209:44
*** markvoelker has quit IRC09:47
*** tumbarka has quit IRC09:47
*** pgadiya has joined #openstack-infra09:47
*** yamamoto has joined #openstack-infra09:51
*** yamamoto has quit IRC09:53
*** tushar has joined #openstack-infra09:55
*** jbadiapa_ has joined #openstack-infra09:57
*** ykarel|lunch has quit IRC09:57
*** ykarel|lunch has joined #openstack-infra09:57
*** yamamoto has joined #openstack-infra09:57
*** LindaWang has quit IRC09:59
*** pcaruana has quit IRC10:02
*** ykarel|lunch has quit IRC10:03
*** yamamoto has quit IRC10:03
*** udesale has quit IRC10:04
*** ykarel has joined #openstack-infra10:05
*** yamamoto has joined #openstack-infra10:08
*** yamamoto has quit IRC10:10
openstackgerritAttila Darazs proposed openstack-infra/tripleo-ci master: Use local mirrors for multinode during DLRN build  https://review.openstack.org/49453710:11
*** yamamoto has joined #openstack-infra10:13
*** wolverineav has joined #openstack-infra10:13
*** LindaWang has joined #openstack-infra10:14
*** yamamoto has quit IRC10:15
*** gongysh has quit IRC10:16
*** kjackal has quit IRC10:21
*** pcaruana has joined #openstack-infra10:22
*** LindaWang has quit IRC10:26
*** namnh has quit IRC10:27
openstackgerritMerged openstack-infra/shade master: Fix determining if IPv6 is supported when it's disabled  https://review.openstack.org/49432110:27
*** yamamoto has joined #openstack-infra10:29
*** mat128 has joined #openstack-infra10:30
*** slaweq_ has joined #openstack-infra10:32
*** dchen has joined #openstack-infra10:34
*** dingyichen has quit IRC10:36
*** dchen has quit IRC10:41
*** dingyichen has joined #openstack-infra10:42
*** yamamoto has quit IRC10:43
*** yamamoto has joined #openstack-infra10:43
*** markvoelker has joined #openstack-infra10:44
*** jkilpatr has quit IRC10:46
*** yamamoto has quit IRC10:47
openstackgerritSlawek Kaplonski proposed openstack-infra/shade master: DNM Test force cinder volume delete  https://review.openstack.org/49470510:48
*** yamamoto has joined #openstack-infra10:49
*** pgadiya has quit IRC10:51
*** ralonsoh has quit IRC10:52
*** yamamoto has quit IRC10:55
*** priteau has joined #openstack-infra10:57
*** shardy is now known as shardy_lunch10:58
*** zhurong has quit IRC11:01
*** dizquierdo has joined #openstack-infra11:01
*** rtjure has quit IRC11:03
*** pgadiya has joined #openstack-infra11:04
*** slaweq_ has quit IRC11:06
*** esberglu has joined #openstack-infra11:06
*** jkilpatr has joined #openstack-infra11:07
*** esberglu has quit IRC11:11
*** lucasagomes is now known as lucas-hungry11:13
*** markvoelker has quit IRC11:17
aspiersis there any equivalent to http://specs.openstack.org/openstack/ for unofficial projects' spec repos?11:17
aspiersI'd love to be able to point people at HTML renderings of https://review.openstack.org/#/q/project:openstack/openstack-resource-agents-specs11:17
*** kjackal has joined #openstack-infra11:18
aspierscurrently the only renderings I'm aware of are the CI build outputs, but these get garbage collected after a while11:18
*** jascott1 has joined #openstack-infra11:19
aspiersoh, well there's github I suppose11:21
*** jascott1 has quit IRC11:24
*** _jbadiapa has joined #openstack-infra11:24
*** jbadiapa_ has quit IRC11:26
openstackgerritOpenStack Proposal Bot proposed openstack-dev/hacking master: Updated from global requirements  https://review.openstack.org/44002911:27
openstackgerritOpenStack Proposal Bot proposed openstack-infra/shade master: Updated from global requirements  https://review.openstack.org/49478911:27
*** LindaWang has joined #openstack-infra11:28
openstackgerritOpenStack Proposal Bot proposed openstack/diskimage-builder master: Updated from global requirements  https://review.openstack.org/49480011:29
*** yamamoto has joined #openstack-infra11:29
*** yamamoto has quit IRC11:31
*** adisky__ has quit IRC11:31
*** yamamoto has joined #openstack-infra11:32
*** yamamoto has quit IRC11:37
*** yamamoto has joined #openstack-infra11:39
openstackgerritOpenStack Proposal Bot proposed openstack/os-client-config master: Updated from global requirements  https://review.openstack.org/49486511:39
*** pgadiya has quit IRC11:43
*** dizquierdo has quit IRC11:47
*** ldnunes has joined #openstack-infra11:49
*** mattmceuen has joined #openstack-infra11:51
*** rhallisey has joined #openstack-infra11:56
*** rhallisey has quit IRC11:58
*** rhallisey has joined #openstack-infra11:58
*** ykarel has quit IRC11:58
*** thorst has joined #openstack-infra11:59
*** ykarel has joined #openstack-infra12:00
*** pgadiya has joined #openstack-infra12:01
*** slaweq_ has joined #openstack-infra12:03
*** derekh has quit IRC12:06
*** shardy_lunch is now known as shardy12:07
*** pgadiya has quit IRC12:09
*** gcb has quit IRC12:10
*** trown|outtypewww is now known as trown12:18
*** gildub has quit IRC12:20
*** jcoufal has joined #openstack-infra12:20
*** rcernin has quit IRC12:21
*** rlandy has joined #openstack-infra12:22
rodrigodsanyone to take a look at https://review.openstack.org/#/c/492223/12:22
rodrigodsit is an important piece from our Outreachy project12:22
*** bogdando has joined #openstack-infra12:25
bogdandoPTAL e-r queries https://review.openstack.org/#/c/493535/ https://review.openstack.org/#/c/493525/  https://review.openstack.org/#/c/493520/ to reduce unknown/uncategorised numbers12:25
*** markvoelker has joined #openstack-infra12:25
openstackgerritAlexander Chadin proposed openstack-infra/project-config master: Add splitted multinode job to watcher-tempest-plugin  https://review.openstack.org/49528912:27
*** makowals has quit IRC12:28
*** makowals has joined #openstack-infra12:28
*** gildub has joined #openstack-infra12:30
*** dave-mccowan has joined #openstack-infra12:31
openstackgerritMerged openstack-infra/shade master: Don't determine local IPv6 support if force_ip4=True  https://review.openstack.org/49431912:33
*** lucas-hungry is now known as lucasagomes12:33
*** bobh has joined #openstack-infra12:34
*** slaweq_ has quit IRC12:35
*** jpena is now known as jpena|lunch12:36
*** bobh has quit IRC12:36
*** mattmceuen has quit IRC12:36
*** baoli has joined #openstack-infra12:37
*** heroux has joined #openstack-infra12:37
*** baoli has quit IRC12:38
*** rcernin has joined #openstack-infra12:38
*** baoli has joined #openstack-infra12:38
*** Dinesh_Bhor has quit IRC12:39
*** dprince has joined #openstack-infra12:42
*** alexchadin has quit IRC12:42
*** alexchadin has joined #openstack-infra12:42
efriedmordred Looking like sta still isn't publishing the latest version.12:44
*** alexchadin has quit IRC12:44
*** alexchadin has joined #openstack-infra12:45
efriedIt published a new one last night, but still with the old resource-cluster data.12:46
fungiaspiers: you could publish your specs to rtfd.org or somewhere like that which will do automated git pull and sphinx build for you. we even have a very simple job which will ping the refresh api endpoint there for any given update to a git repo in our system12:46
efriedmordred Hmph, cancel that, it looks fine, wtf was I seeing before?  Did a Shift+reload and everything.12:52
openstackgerritAlexander Chadin proposed openstack-infra/project-config master: Remove old functional job for Watcher  https://review.openstack.org/49529612:54
*** esberglu has joined #openstack-infra12:54
openstackgerritMerged openstack-dev/pbr master: Updated from global requirements  https://review.openstack.org/49478812:55
* fungi fades back into the aether for most of the next ~8 hours12:57
*** esberglu has quit IRC12:59
*** ldnunes has quit IRC12:59
*** florianf has quit IRC13:01
*** LindaWang has quit IRC13:01
*** elmiko has left #openstack-infra13:04
*** jaosorior has quit IRC13:05
*** sree has quit IRC13:08
*** sree has joined #openstack-infra13:08
*** gouthamr has joined #openstack-infra13:09
*** florianf has joined #openstack-infra13:09
*** tushar has quit IRC13:10
*** gildub has quit IRC13:11
*** ldnunes has joined #openstack-infra13:11
*** bobh has joined #openstack-infra13:11
*** tushar has joined #openstack-infra13:12
*** ramishra has quit IRC13:15
*** kjackal has quit IRC13:15
*** alexchadin has quit IRC13:20
*** ramishra has joined #openstack-infra13:21
mordredefried: thanks for the followup patch!13:25
efriedyessir13:25
efriedfrickler gets the credit, really.13:25
toskyuhm, how frequently does the OpenStack Proposal bot run? For each change in requirements.txt?13:26
openstackgerritOpenStack Proposal Bot proposed openstack-infra/shade master: Updated from global requirements  https://review.openstack.org/49478913:29
*** slaweq_ has joined #openstack-infra13:32
*** ihrachys has joined #openstack-infra13:32
*** david-lyle has joined #openstack-infra13:35
*** felipemonteiro_ has joined #openstack-infra13:35
mordredtosky: yup13:35
*** baoli has quit IRC13:36
*** dklyle has quit IRC13:36
toskymordred: would it make sense to consolidate the run periodically (like twice a day), or could it end up breaking more? Goal: save resource (especially these days)13:36
toskythere was a change 10 hours ago and another one now13:36
*** felipemonteiro__ has joined #openstack-infra13:37
*** esberglu has joined #openstack-infra13:37
*** amoralej is now known as amoralej|off13:40
*** felipemonteiro_ has quit IRC13:40
*** jpena|lunch is now known as jpena13:40
pabelangerYa, looks like we didn't finish all the periodic jobs this morning13:42
pabelangertosky: peroidic jobs already run once a day, starting at 02:00 UTC13:42
pabelangerhttps://review.openstack.org/490151/ was the tigger for this morning13:46
*** dingyichen has quit IRC13:46
pabelangergoing to pop into requirements and see what the plan it for them13:46
toskypabelanger: which maps to https://review.openstack.org/#/c/495272/, but then https://review.openstack.org/#/c/494921/ ?13:47
toskyI mean, 2 changes in than 10 hours when the gates are in full steam mode13:47
pabelangertosky: I don't believe it was intentional, removal of osic-cloud1 was quite a large hit to CI resources (1/2 our capacity). So, we likely need to just keep an eye out for things until pike has been released13:58
*** xyang1 has joined #openstack-infra13:58
*** sree has quit IRC13:59
toskypabelanger: but leaving aside the current availability of resources, wouldn't it make sense to consolidate the updates from changes to requirements, so that they are not sent for each change to that repository, but (say) twice a day?13:59
*** sree has joined #openstack-infra14:00
pabelangertosky: Right, there is some optimizations that likely can be done now, but we'd need to talk about that.  proposal bot is smart enough to rebase an open review for an update, but will still require CI resources.  For now, I think asking the requirements team to keep an eye on status.o.o/zuul is a good first step here.14:03
toskyack14:03
pabelangertosky: I suggest adding a topic to our next infra meeting, tuesday, and see if we could make those optimizations14:03
*** kjackal has joined #openstack-infra14:04
*** sree has quit IRC14:04
*** slaweq_ has quit IRC14:06
*** efried is now known as fried_rice14:06
pabelangermnaser: clarkb: we look to be averaging 2-4 timeouts per 3 hours in vexxhost currently.14:10
pabelangerinfracloud-vanilla is a little lower, chocolate appears to be about 114:11
*** rbrndt has joined #openstack-infra14:11
*** Julien-zte has quit IRC14:15
*** ramishra has quit IRC14:15
pabelangermnaser: clarkb: maybe drop vexxhost back down to 20 until r+1 or until we start profiling the failures14:16
*** david-lyle has quit IRC14:16
*** dklyle has joined #openstack-infra14:16
*** baoli has joined #openstack-infra14:17
*** ramishra has joined #openstack-infra14:17
DiabelkoI'm trying to migrate my CI system from Jenkins to Z-L model and I'm running into funky issue that I don't have an idea how to debug xD14:17
pabelangerianw: clarkb: new mirrors are holding up really well! Aside from 502 proxy errors we've randomly seen, yum client failures appear to be download significantly14:18
*** Julien-zte has joined #openstack-infra14:18
Diabelkobasically, only one gearman client (either Z-L or Jenkins) can connect to Gearman with proper CAN_DO job list14:18
clarkbpabelanger: ya and I think we need to start filtering out valid timeouts, I saw some tempest jobs with an hour timeojt for example. That is just not possible14:18
*** bogdando has quit IRC14:18
DiabelkoI noticed first one that connects to Gearman is usually the one with proper job list reporting14:19
Diabelkoany ideas on how to debug this?14:19
DiabelkoI have to add I've tested that with both Gearman in python and gearmand14:20
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Revert "Bump vexxhost max-servers to 40"  https://review.openstack.org/49531814:20
*** _jbadiapa has quit IRC14:21
pabelangermnaser: clarkb:^14:21
pabelangerDiabelko: what does the debug long on zuul-launcher show?14:21
pabelangerlog*14:21
Diabelkodoesn't say anything about CAN_DO with job list14:22
DiabelkoI am a little bit desperate to a point where I'm really thinking about using tcpdump to see the whole traffic14:22
pabelangerDiabelko: what does your nodepool.yaml look like?14:25
pabelangerDiabelko: specifically your targets: http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/nodepool.yaml#n116014:25
*** dansmith is now known as superdan14:26
*** kgiusti has joined #openstack-infra14:26
openstackgerritThierry Carrez proposed openstack/ptgbot master: Let room operators clean up themselves  https://review.openstack.org/49532014:28
*** wolverineav has quit IRC14:28
openstackgerritThierry Carrez proposed openstack/ptgbot master: Shut up now, ptgbot  https://review.openstack.org/49532114:28
*** marst has joined #openstack-infra14:28
Diabelkopabelanger: don't have nodepool, I just have static nodes defined for my launchers14:29
Diabelkoand right this moment, Z-L reports properly to Gearman14:31
Diabelkobut my two jenkins masters don't ;)14:31
pabelangerDiabelko: okay, we do that with zlstatic.o.o today. What does your 'node' section look like in zuul.conf?14:31
*** wolverineav has joined #openstack-infra14:32
Diabelkosure, hang on14:33
Diabelkohttp://paste.openstack.org/show/618775/14:35
Diabelkobasically this14:35
Diabelkodon't mind the windows part - it's same situation for linux :)14:36
pabelangerDiabelko: that looks right, we have node and host the same value, but dns shouldn't be the issue14:36
*** _jbadiapa has joined #openstack-infra14:37
*** sree has joined #openstack-infra14:38
*** srobert has joined #openstack-infra14:40
mordredfried_rice: https://review.openstack.org/49532614:42
fried_ricelooking14:42
*** ramishra has quit IRC14:46
*** dhajare has quit IRC14:49
*** markus_z has quit IRC14:51
*** ramishra has joined #openstack-infra14:53
*** slaweq has quit IRC14:53
fried_ricemordred Feel free to slap me if I'm waaay off base there.14:56
fried_ricemordred (I didn't mention that we'll have to dice the URL if we want to use the relative path; wouldn't expect that to be a show-stopper.)14:56
mordredfried_rice: so - actually - the api-reference links in sta are stale, as we're now publishing everyting to its service-type14:57
mordredfried_rice: so next step is i believe to remove api-reference links from teh sta yaml file and generate them from the api base + service-type when we make the json14:58
fried_ricewhoah14:58
*** xarses has joined #openstack-infra14:58
fried_riceAnd hope nobody is using the yaml directly.  But I guess that's in the contract?14:58
fried_riceIn any case, my issue stands.  Otherwise we're duplicating logic in ^ and in sta.14:59
fried_riceAnd arguably we should do that sta swizzle first.15:00
fried_ricemordred You want me to tackle that?15:01
*** bogdando has joined #openstack-infra15:02
*** slaweq has joined #openstack-infra15:03
*** dpawlik is now known as danpawlik_15:03
*** danpawlik_ is now known as dpawlik_15:03
*** david-lyle has joined #openstack-infra15:04
*** dklyle has quit IRC15:04
*** andreww has joined #openstack-infra15:08
*** andreww has quit IRC15:08
*** links has quit IRC15:10
*** xarses has quit IRC15:10
*** andreww has joined #openstack-infra15:11
clarkbpabelanger: I'm going to finish up the citycloud mirror upgrades this morning even though those regions are in use so that we are ready to go when we can turn them back on15:12
clarkb*are not in use'15:12
*** sambetts is now known as sambetts|afk15:13
pabelangerclarkb: k, I'll be working on zuulv3 and testing glean systemd fixes15:16
*** dizquierdo has joined #openstack-infra15:16
*** egonzalez has quit IRC15:16
*** camunoz has quit IRC15:18
pabelangerclarkb: are you okay with 495318? That reduces vexxhost by 50%15:18
clarkbI was going to pull up the logstash query for job timeouts and see if those timeouts are just due to impossible timeout values or are valid15:18
clarkbpabelanger: I had noticed manilla has tempest jobs with an hour timeout which just isn't possible on any cloud at this point I don't think15:18
pabelangerk15:20
*** sree has quit IRC15:21
*** armax has quit IRC15:21
*** udesale has joined #openstack-infra15:21
*** armax has joined #openstack-infra15:22
clarkbpabelanger: also cloud-rh1 is our biggest aggregate timeout source (not percentage of cloud quota or jobs run, just total number of timeouts that logstash knows about)15:23
clarkbwhich may also be skewed because vexxhost is relatively new15:23
clarkbpabelanger: looking at vexxhost slow setup workspaces we appear to set up each repo twice per setup_workspace. So a total of 4 times per grenade job15:29
pabelangerclarkb: have a log you can share?15:30
pabelangerclarkb: http://logs.openstack.org/39/494639/1/gate/gate-tripleo-ci-centos-7-nonha-multinode-oooq/4014b45/logs/undercloud/home/jenkins/install_packages.sh.log.txt.gz#_2017-08-18_15_08_37 another 502 bad gateway in infracloud-vanilla also15:30
clarkboh wait15:31
clarkbit is because it is a multinode job15:31
clarkbso I just need some caffeine15:31
clarkbpabelanger: mnaser http://logs.openstack.org/74/494974/1/check/gate-grenade-dsvm-neutron-multinode-ubuntu-xenial/91b7aef/logs/devstack-gate-setup-workspace-new.txt.gz#_2017-08-18_12_33_49_850 if I read that right it took almost a minute to stat that directory15:33
clarkbthat may account for some of the slowness there15:33
clarkbin which case ya I think we can scale back a bit until we understand that better15:33
mnaserclarkb ceph backed storage, maybe there was some slow requests in flight there (though not common)15:34
* mnaser is really looking forward for new release of ceph15:34
clarkbpabelanger: it would be interesting to see if the centos buildlogs server lgos an error there or if it thnnks everything is fine15:35
*** slaweq has quit IRC15:35
clarkbpabelanger: the behavior we saw in devstack was that the client wasn't handling keepalive'd connections that had been closed by the server due to the keepalive timing out15:35
*** iyamahat has joined #openstack-infra15:36
clarkbso from the server perpsective it was happy and doing what it was supposed to but the client wasn't. Now we may be hitting that on the yum side to our mirror/proxy too15:36
pabelangerclarkb: Ya, that might be harder to do, we're hitting the buildlogs.cdn at that point. I am not sure if centos people have access to that infra15:36
pabelangerI'll ask however15:36
clarkboh right its all cdn except for that one file15:36
pabelangerya15:37
pabelangerI do think we should bump logging level in apache there to trace, and see what we get back in debug logs15:37
pabelangermaybe see something obvious15:37
*** david-lyle has quit IRC15:37
*** dklyle has joined #openstack-infra15:37
clarkbis it possible to tell the client side of our proxy eg connection to cdn to not use keepalives? I know we can disable the server side keepalives15:37
clarkbya that can't hurt (just make sure we don't fill disk with that data)15:38
clarkbzuul status looks like day of feature freeze used to look15:39
clarkbmakes me wonder if feature freeze being early like that is doing its job15:39
clarkb(thats the requirements proposals generated load that tosky was talking about earlier?)15:40
*** armax_ has joined #openstack-infra15:40
*** armax has quit IRC15:40
*** armax_ is now known as armax15:40
clarkbpabelanger: I've +2'd the vexxhost change but need caffeine and breakfast so won't approve it with one +2.15:41
*** dklyle has quit IRC15:44
*** david-lyle has joined #openstack-infra15:46
pabelangerclarkb: ya, requirements chance added 2/3 new jobs into zuul. That didn't help :)15:47
*** Apoorva has joined #openstack-infra15:48
*** cody-somerville has joined #openstack-infra15:53
*** cody-somerville has quit IRC15:53
*** cody-somerville has joined #openstack-infra15:53
*** aarefiev is now known as aarefiev_afk15:55
*** andreww has quit IRC16:03
*** shardy has quit IRC16:03
*** xarses has joined #openstack-infra16:03
*** Apoorva_ has joined #openstack-infra16:04
*** gongysh has joined #openstack-infra16:07
*** Apoorva has quit IRC16:07
*** jpich has quit IRC16:09
*** wolverineav has quit IRC16:09
*** wolverineav has joined #openstack-infra16:09
*** csomerville has joined #openstack-infra16:09
*** jtomasek has quit IRC16:09
*** e0ne has quit IRC16:09
*** tbachman has joined #openstack-infra16:10
openstackgerritMerged openstack-infra/project-config master: Normalize projects.yaml  https://review.openstack.org/49513416:10
*** pcaruana has quit IRC16:12
*** tesseract has quit IRC16:12
*** rcernin has quit IRC16:12
*** cody-somerville has quit IRC16:12
*** tbachman has left #openstack-infra16:13
*** markvoelker has quit IRC16:16
*** bogdando has quit IRC16:17
*** Apoorva_ has quit IRC16:18
*** Apoorva has joined #openstack-infra16:18
clarkbpabelanger: thoughts on leaving the old mirrors in la1 and lon1 in place beacuse we don't have jobs verifying the new one work? I did manually hit the http server with my webbrowser and new ones seem fine so I am thinking I will just delete the old ones16:23
pabelangerclarkb: Ya, I usually make sure http works before I cut over DNS, so I think we should be okay there16:24
clarkbok16:24
*** slaweq has joined #openstack-infra16:32
*** jascott1 has joined #openstack-infra16:36
*** Guest75213 is now known as med_16:37
*** med_ has quit IRC16:37
*** med_ has joined #openstack-infra16:37
*** ramishra has quit IRC16:37
*** ykarel has quit IRC16:39
*** robled has quit IRC16:42
*** robled has joined #openstack-infra16:42
*** rcarrill1 is now known as rcarrillocruz16:42
*** wolverineav has quit IRC16:45
*** lucasagomes is now known as lucas-afk16:46
*** vhosakot has joined #openstack-infra16:46
*** esberglu has quit IRC16:48
*** dizquierdo has quit IRC16:51
*** wolverineav has joined #openstack-infra16:53
clarkbinterestingly apache on the la1 and lon1 mirrors is reporting GETs by IPs in rackspace16:56
clarkbso I'm gonna let those fall off before deleting the old servers just to be nice (also worried it may be something we are doing)16:57
*** ykarel has joined #openstack-infra16:58
*** jpena is now known as jpena|off17:03
*** hongbin has joined #openstack-infra17:04
*** slaweq has quit IRC17:05
*** SumitNaiksatam has quit IRC17:05
openstackgerritMonty Taylor proposed openstack-infra/bindep master: Refactor 'main.py'  https://review.openstack.org/45607117:06
openstackgerritMerged openstack-infra/system-config master: Move RAX mirrors to numbered hosts  https://review.openstack.org/49437017:09
*** ykarel has quit IRC17:11
openstackgerritMerged openstack-infra/system-config master: Rename new infracloud, citycloud and vexxhost mirrors  https://review.openstack.org/49472817:13
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Update base-test to use site_logs secret  https://review.openstack.org/49431417:17
*** bhavik1 has joined #openstack-infra17:19
*** armax has quit IRC17:19
*** armax has joined #openstack-infra17:21
*** tumbarka has joined #openstack-infra17:23
*** tushar has quit IRC17:23
*** bhavik1 has quit IRC17:25
*** iyamahat has quit IRC17:25
openstackgerritMerged openstack-infra/project-config master: Revert "Bump vexxhost max-servers to 40"  https://review.openstack.org/49531817:30
*** jbadiapa_ has joined #openstack-infra17:31
*** tosky has quit IRC17:32
*** _jbadiapa has quit IRC17:34
*** jascott1 has quit IRC17:34
*** signed8bit has joined #openstack-infra17:36
*** Hal has joined #openstack-infra17:38
*** Hal is now known as Guest2142517:38
*** kjackal has quit IRC17:38
*** Swami has joined #openstack-infra17:40
*** gomarivera has joined #openstack-infra17:41
*** iyamahat has joined #openstack-infra17:43
*** csomerville has quit IRC17:47
*** cody-somerville has joined #openstack-infra17:48
*** phenixrizen has quit IRC17:51
*** phenixrizen has joined #openstack-infra17:51
*** phenixrizen has quit IRC17:52
*** esberglu has joined #openstack-infra17:54
*** tosky has joined #openstack-infra17:57
openstackgerritMonty Taylor proposed openstack-infra/puppet-zuul master: Add python-bpo-27945-backport PPA to executor  https://review.openstack.org/49539917:58
*** Guest21425 has quit IRC17:59
mordredpabelanger, jeblair, clarkb: ^^18:00
AJaegerwow, 350 jobs in post pipeline waiting for free nodes ;( Good that weekend is near ;)18:00
pabelangermordred: +218:00
pabelangerAJaeger: ya, wait until you look at periodic-stable. Didn't clear that this morning18:01
clarkbmordred: has a bug with ubuntu been filed yet?18:02
*** slaweq has joined #openstack-infra18:02
*** slaweq has quit IRC18:05
mordredclarkb: yah - SpamapS has uploaded the SRU even18:05
*** phenixrizen has joined #openstack-infra18:05
clarkbcool I'm finishing up eggs and sausage breakfast then will review18:05
*** Douhet has quit IRC18:07
*** Douhet has joined #openstack-infra18:07
*** slaweq has joined #openstack-infra18:07
*** florianf has quit IRC18:07
AJaegerpabelanger: argh ;(18:12
*** slaweq has quit IRC18:13
*** jcoufal has quit IRC18:13
openstackgerritMonty Taylor proposed openstack-infra/project-config master: Emit and publish ara logs if available  https://review.openstack.org/49428118:14
mordredpabelanger, jeblair: ^^ that should be good to go now18:14
clarkbalright old citycloud mirrors appear idle now, deleting them18:15
*** Apoorva_ has joined #openstack-infra18:17
*** Apoorva has quit IRC18:19
openstackgerritClark Boylan proposed openstack-infra/system-config master: Move Lon1 and La1 mirrors to digitized names  https://review.openstack.org/49540818:20
clarkbinfra-root ^ that concludes the mirror cleanup in citycloud18:20
*** jcoufal has joined #openstack-infra18:22
*** Douhet has quit IRC18:22
*** Douhet has joined #openstack-infra18:23
*** trown is now known as trown|lunch18:23
jeblairmordred: that looks good -- do you want to roll the dice and merge that risking a force-merg if it breaks, or do you want to merge a parallel change to base-test first?18:23
openstackgerritDivyansh Acharya proposed openstack/os-testr master: Just changed the opening of the file  https://review.openstack.org/49469718:24
clarkbmordred: jeblair seems odd we'd expect ara to run globally? doesn't it run out of a venv today?18:28
clarkboh wait that is on localhost, nevermind18:29
openstackgerritMerged openstack-infra/project-config master: Switch capability_scope to global  https://review.openstack.org/49477418:29
openstackgerritEmilien Macchi proposed openstack-infra/project-config master: tripleo-quickstart: run tripleo upgrade jobs in check  https://review.openstack.org/49540918:30
EmilienMweshay: ^ fyi18:30
*** gomarivera has quit IRC18:31
*** gomarivera has joined #openstack-infra18:31
weshayEmilienM, you did it!!18:32
weshayvery nice.. very very nice18:32
EmilienMweshay: next step, make it voting once we have stable/pike18:32
*** dprince has quit IRC18:32
*** slaweq has joined #openstack-infra18:32
*** slaweq has quit IRC18:33
*** udesale has quit IRC18:34
openstackgerritMerged openstack-infra/shade master: Updated from global requirements  https://review.openstack.org/49478918:36
*** gomarivera has quit IRC18:36
*** rbrndt has quit IRC18:37
*** Hal has joined #openstack-infra18:37
pabelangerjeblair: mordred: base-test first?18:37
*** Hal is now known as Guest9265518:38
pabelangerjeblair: mordred: we could stack on  49431418:38
*** rbrndt has joined #openstack-infra18:38
*** gomarivera has joined #openstack-infra18:41
*** makowals has quit IRC18:41
openstackgerritJimmy McCrory proposed openstack-infra/project-config master: Add pike branch for OpenStack-Ansible gerritbot  https://review.openstack.org/49541018:45
*** gomarivera has quit IRC18:45
*** jascott1 has joined #openstack-infra18:45
*** csomerville has joined #openstack-infra18:47
*** cody-somerville has quit IRC18:47
*** dprince has joined #openstack-infra18:49
*** prad_ has joined #openstack-infra18:49
*** prad_ is now known as prad18:49
openstackgerritgreghaynes proposed openstack/diskimage-builder master: Add networkd-dhcp element  https://review.openstack.org/45941918:53
clarkbmy laptop can no longer upgrade its kernels because ubuntu default partitioning lacks lvm and only 256MB was given to /boot. So I'm gonna disappear after lunch and reinstall18:57
clarkb(that'll learn me to trust an installer)18:57
pabelangerclarkb: ack. I'm just building a new DIB with glean override.conf change18:58
jeblairclarkb: yeah it's terrible.  i run this: sudo apt-get remove --purge $(dpkg -l 'linux-image-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d') $(dpkg -l 'linux-headers-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d')19:01
*** jbadiapa_ has quit IRC19:01
jeblair(i found that on the Internet)19:01
clarkbimpressive19:02
pabelangerwow19:03
jeblairdeletes all but the current19:03
clarkb/boot/uefi is also massive and only contains ~2MB in it19:03
clarkbso I think there must've been confusion over where the big files go at some point19:04
jeblairclarkb: i shudder to think they're planning for more in the future :(19:04
*** markvoelker has joined #openstack-infra19:05
*** jbadiapa_ has joined #openstack-infra19:17
*** e0ne has joined #openstack-infra19:17
*** signed8bit is now known as signed8bit_Zzz19:19
*** gomarivera has joined #openstack-infra19:19
*** rkukura has joined #openstack-infra19:23
*** gomarivera has quit IRC19:26
*** vhosakot has quit IRC19:26
*** signed8bit_Zzz is now known as signed8bit19:26
*** dizquierdo has joined #openstack-infra19:29
*** slaweq has joined #openstack-infra19:29
*** yamamoto has quit IRC19:30
*** yamamoto has joined #openstack-infra19:33
*** greghayn1 has joined #openstack-infra19:34
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Remove publish-openstack-python-branch-tarball job  https://review.openstack.org/49541819:34
*** greghaynes has quit IRC19:34
openstackgerritPaul Belanger proposed openstack-infra/openstack-zuul-jobs master: Remove publish-openstack-python-branch-tarball job  https://review.openstack.org/49467019:35
*** jkilpatr has quit IRC19:35
*** vhosakot has joined #openstack-infra19:37
*** LindaWang has joined #openstack-infra19:44
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Add publish-openstack-python-branch-tarball job  https://review.openstack.org/49467219:44
*** trown|lunch is now known as trown19:44
*** cody-somerville has joined #openstack-infra19:47
*** rkukura has quit IRC19:47
*** xyang1 has quit IRC19:50
*** csomerville has quit IRC19:50
*** gomarivera has joined #openstack-infra19:50
*** jkilpatr has joined #openstack-infra19:50
*** e0ne has quit IRC19:51
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Remove publish-openstack-python-branch-tarball job  https://review.openstack.org/49541819:51
*** xyang1 has joined #openstack-infra19:51
openstackgerritJohn Trowbridge proposed openstack-infra/tripleo-ci master: Add containers periodic test  https://review.openstack.org/47574719:52
*** Guest92655 has quit IRC19:53
*** cody-somerville has quit IRC19:55
*** srobert has quit IRC19:56
*** cody-somerville has joined #openstack-infra19:56
*** cody-somerville has joined #openstack-infra19:56
*** makowals has joined #openstack-infra19:57
*** prad has quit IRC19:59
pabelangermordred: clarkb: do you mind a +3 on https://review.openstack.org/494314/ that adds our production static.o.o to base-test jobs for zuulv320:00
*** yamamoto has quit IRC20:01
*** slaweq has quit IRC20:02
*** slaweq has joined #openstack-infra20:04
openstackgerritColleen Murphy proposed openstack-infra/puppet-ansible master: Depend on helper gem for spec_helper_acceptance  https://review.openstack.org/49542020:05
*** priteau has quit IRC20:05
*** priteau has joined #openstack-infra20:05
*** gomarivera has quit IRC20:06
openstackgerritMonty Taylor proposed openstack-infra/project-config master: Promote ara logging and site_logs secret to base job  https://review.openstack.org/49428120:07
openstackgerritMonty Taylor proposed openstack-infra/project-config master: Add ara logging to base-test job  https://review.openstack.org/49542120:07
openstackgerritMonty Taylor proposed openstack-infra/zuul-jobs master: Add role for emitting ara logs on the executor  https://review.openstack.org/49542220:07
mordredjeblair, pabelanger: ^^ rebased the ara one on pabelanger's base-test job for logs, extracted it into a role and put it on base-test, then added a followup change with putting both things in base20:07
*** mat128 has quit IRC20:08
mordredpabelanger: done20:09
*** priteau has quit IRC20:09
jeblairmordred: +2s all around20:12
*** Apoorva_ has quit IRC20:13
*** Apoorva has joined #openstack-infra20:14
*** Apoorva has quit IRC20:18
openstackgerritMerged openstack-infra/project-config master: Update base-test to use site_logs secret  https://review.openstack.org/49431420:20
mordredjeblair: \o/20:20
mordredpabelanger, jeblair: do we have a job already using base-test we can use to validate that ^^ ? or should I make one real quick?20:22
jeblairmordred: not sure; i think in the past i just proposed a speculative change to reparent something to base-test.20:23
jeblairmordred: like maybe just proposed that change to unittest job on zuul-jobs20:23
mordredjeblair: k. I'll give that a shot20:23
openstackgerritMerged openstack-infra/puppet-zuul master: Add python-bpo-27945-backport PPA to executor  https://review.openstack.org/49539920:24
openstackgerritMonty Taylor proposed openstack-infra/zuul-jobs master: DNM Reparent unittests on base-test  https://review.openstack.org/49542420:26
openstackgerritPaul Belanger proposed openstack-infra/glean master: Create ARGS environmental varaible to allow for systemd override files  https://review.openstack.org/49542520:27
openstackgerritMonty Taylor proposed openstack-infra/project-config master: Mark publish-openstack-artifacts as post-review  https://review.openstack.org/49542620:27
mordredjeblair: ^^ there's the post-review flag20:28
jeblairmordred: oh we should probably add post-review to gate, post and release pipelines in same change20:28
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Create glean@.service.d/override.conf  https://review.openstack.org/49366520:29
openstackgerritMerged openstack-infra/zuul-jobs master: Add role for emitting ara logs on the executor  https://review.openstack.org/49542220:29
*** dizquierdo has quit IRC20:29
mordredjeblair: kk. can do20:29
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Allow multiple semaphore definitions within a project  https://review.openstack.org/49465020:29
pabelangerclarkb: ^ should simply our glean override.conf setting for systemd, but need another glean release20:29
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Reload configuration when branches are created or deleted  https://review.openstack.org/49465120:31
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Use cached branch list in dynamic config  https://review.openstack.org/49461820:31
*** dprince has quit IRC20:32
openstackgerritMonty Taylor proposed openstack-infra/project-config master: Mark publish-openstack-artifacts and pipelines as post-review  https://review.openstack.org/49542620:32
clarkbpabelanger: why do we need another glean release?20:34
pabelangerclarkb: I took a different approach: https://review.openstack.org/495425/20:35
clarkbdoes overriding the exec command not work?20:35
clarkboh I see means you don't need different overrides per distro20:35
jeblairclarkb, pabelanger, mordred: i rebooted ze01 recently and it got a new hostname.  ze01 is 'ze01'.  ze02 is 'ze02.openstack.org'.  any idea why?20:35
jeblair(i mean, /etc/hostname has those values on those hosts)20:36
pabelangerclarkb: ya, optimization. Think it might be the better approach20:36
clarkbare they in different clouds? we've seen different cloud-init and nova agent behavior with hostnames20:36
pabelangerjeblair: we see that issue on nb04 also, it was renamed from nb04.openstack.org to nb04, which confused nodepool-builder20:37
mordredI thought we set hostname and /etc/hosts in ansible/puppet - do we have something going around after we've done that and re-setting it at boot now?20:37
pabelangernot sure why that is, sorry20:37
pabelangeris set_hostname only called by launch-node.py?20:38
*** mat128 has joined #openstack-infra20:38
*** jcoufal has quit IRC20:38
pabelangeran never from puppetmaster.o.o20:38
clarkbpabelanger: where does %%gleansh_path%%  come from? curious why there are two different forms of variable expansion20:38
jeblairclarkb: all dfw20:39
pabelangerclarkb: glean/install.py IIRC20:39
*** heroux has quit IRC20:39
clarkbah its templated there got it20:39
pabelangermordred: ya, I think that is the issue. we don't have set_hostname in our ansible wheel20:39
pabelangerso we loose the values on reboot20:39
mordredpabelanger: do I even want to know why it's not persistent?20:40
*** csomerville has joined #openstack-infra20:40
pabelangercloud-init setting up /etc/hosts maybe?20:40
mordredoh - wow. these do have cloud-init20:41
pabelanger2017-08-17 22:16:35,520 - handlers.py[DEBUG]: finish: init-network/config-update_etc_hosts: SUCCESS: config-update_etc_hosts ran successfully20:41
mordredAND nova agent20:41
pabelangerI see that in /var/log/cloud-init.log20:41
mordredwow. that's just so f-d up20:41
jeblairnova-agent set the hostname correctly last year and hasn't mentioned it since:20:42
mordredisn't there some file you can drop somewhere that tells cloud-init to please stop trying to configure things?20:42
jeblair2017-06-06 18:15:38,053 [INFO] hostname: 'ze01.openstack.org' (from xenstore)20:42
jeblair-rw-r--r-- 1 root root 5 Aug 17 22:16 /etc/hostname20:42
jeblairbut the hostname *was* updated on last boot to the incorrect value20:42
pabelanger2017-08-17 22:16:35,516 - util.py[DEBUG]: Running command ['hostname', 'ze01'] with allowed return codes [0] (shell=False, capture=True)20:42
pabelangerthere it is20:42
mordredbut also ... 2017-08-17 22:16:35,520 - cc_update_etc_hosts.py[DEBUG]: Configuration option 'manage_etc_hosts' is not set, not managing /etc/hosts in module update_etc_hosts20:43
pabelangergoogle says touch /etc/cloud/cloud-init.disabled20:43
mordredoh- hostname, not hosts20:43
pabelangerto disable it20:43
*** Apoorva has joined #openstack-infra20:43
jeblair(note, cloud-init also set the hostname incorrectly last year, but nova-agent reset it correctly a few seconds later)20:43
*** cody-somerville has quit IRC20:44
*** Apoorva has quit IRC20:44
mordredI think we should touch that file in launch-node - and in puppet20:44
jeblairmordred: i like that idea20:44
pabelanger+120:44
*** mat128 has quit IRC20:44
mordredwe could also do:20:44
*** Apoorva has joined #openstack-infra20:44
mordred# This will cause the set+update hostname module to not operate (if true)20:44
mordredpreserve_hostname: false20:44
*** Apoorva has quit IRC20:45
*** cody-somerville has joined #openstack-infra20:45
*** cody-somerville has joined #openstack-infra20:45
mordrednah. wow there'sa ton of crap in that file20:45
*** csomerville has quit IRC20:46
mordredI say let's do disabled for now, and then if we want to spend any more time on the topic, we figure out managing our own base images20:46
pabelangerya, I'd love the spend the time one DIB controlplane images then hack up cloud-images :)20:48
*** trown is now known as trown|outtypewww20:48
mordred++20:49
openstackgerritMonty Taylor proposed openstack-infra/system-config master: Disable cloud-init in launch-node  https://review.openstack.org/49542720:49
*** gouthamr has quit IRC20:52
*** wolverineav has quit IRC20:53
*** Apoorva has joined #openstack-infra20:53
*** wolverineav has joined #openstack-infra20:56
*** csomerville has joined #openstack-infra20:56
*** makowals has quit IRC20:58
openstackgerritMerged openstack-infra/project-config master: Add ara logging to base-test job  https://review.openstack.org/49542120:59
openstackgerritMonty Taylor proposed openstack-infra/project-config master: Update yaml sad in base-test  https://review.openstack.org/49543121:00
*** cody-somerville has quit IRC21:00
*** dave-mccowan has quit IRC21:01
mordredpabelanger, jeblair: ^^ test of the base-test patch failed - I believe that will fix it21:01
*** slaweq has quit IRC21:02
pabelangerguess we need to run ansible-lint on project-config21:02
jeblairmordred: that has a merge conflict?21:02
*** slaweq has joined #openstack-infra21:02
openstackgerritMonty Taylor proposed openstack-infra/project-config master: Promote ara logging and site_logs secret to base job  https://review.openstack.org/49428121:03
openstackgerritMonty Taylor proposed openstack-infra/project-config master: Update yaml sad in base-test  https://review.openstack.org/49543121:04
mordredjeblair: fixed21:04
*** wolverineav has quit IRC21:04
*** rlandy is now known as rlandy|afk21:05
mordredjeblair, pabelanger: I'm just going to bump that in21:05
*** kgiusti has left #openstack-infra21:05
pabelangerwfm21:06
*** slaweq has quit IRC21:07
*** slaweq has joined #openstack-infra21:08
*** slaweq has quit IRC21:08
*** dave-mccowan has joined #openstack-infra21:12
*** dizquierdo has joined #openstack-infra21:13
*** gouthamr has joined #openstack-infra21:13
*** thorst has quit IRC21:13
*** dave-mccowan has quit IRC21:15
*** ldnunes has quit IRC21:17
*** dizquierdo has quit IRC21:18
*** superdan is now known as dansmith21:19
openstackgerritMerged openstack-infra/project-config master: Update yaml sad in base-test  https://review.openstack.org/49543121:23
openstackgerritMonty Taylor proposed openstack-infra/zuul-jobs master: DNM Reparent unittests on base-test job  https://review.openstack.org/49542421:24
*** vhosakot has quit IRC21:25
*** Apoorva_ has joined #openstack-infra21:28
*** thorst has joined #openstack-infra21:30
*** thorst has quit IRC21:31
*** Apoorva has quit IRC21:31
*** askb has joined #openstack-infra21:33
*** baoli has quit IRC21:35
clarkblooks like we are catching up to job demand finally21:38
clarkb(I'm fighting a netinstall image to actually install linux)21:38
openstackgerritMonty Taylor proposed openstack-infra/project-config master: Try base-test again, but this time with the right syntax  https://review.openstack.org/49543821:38
mordredpabelanger, jeblair: I think we can/should add an ansible-lint job to run on playbooks in project-config21:40
mordredpabelanger, jeblair: if we had, it would have told us:21:41
mordredrole dict {u'add-fileserver': {u'fileserver': u'{{ site_logs }}'}} does not contain a "role" or "name" key21:41
*** Apoorva_ has quit IRC21:41
mordredwhich would have led us to realize it shoul dhave been {'role': 'add-fileserver', 'fileserver': '{{ site_logs }}'}21:41
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP Add wrapper driver execution context  https://review.openstack.org/49543921:42
*** Apoorva has joined #openstack-infra21:42
jeblairmordred: ++21:43
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Document execution_wrapper setting.  https://review.openstack.org/49544021:44
*** rlandy|afk is now known as rlandy21:46
*** yamahata has joined #openstack-infra21:46
*** esberglu has quit IRC21:47
openstackgerritMerged openstack-infra/project-config master: Try base-test again, but this time with the right syntax  https://review.openstack.org/49543821:55
*** xyang1 has quit IRC21:56
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Add wrapper driver execution context  https://review.openstack.org/49543921:59
*** tjones has joined #openstack-infra21:59
*** lbragstad has quit IRC21:59
*** priteau has joined #openstack-infra22:01
*** felipemonteiro__ has quit IRC22:05
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Allow requesting secrets by a different name  https://review.openstack.org/49434322:13
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Document execution_wrapper setting.  https://review.openstack.org/49544022:13
*** makowals has joined #openstack-infra22:14
*** Swami has quit IRC22:14
*** bnemec has quit IRC22:15
*** priteau has quit IRC22:18
*** esberglu has joined #openstack-infra22:20
openstackgerritMonty Taylor proposed openstack-infra/zuul-jobs master: Document and update fileserver roles  https://review.openstack.org/49429122:22
*** esberglu has quit IRC22:25
*** bobh has quit IRC22:26
*** rlandy has quit IRC22:29
*** thorst has joined #openstack-infra22:31
*** marst has quit IRC22:31
*** thorst has quit IRC22:36
*** rkukura has joined #openstack-infra22:37
*** slaweq has joined #openstack-infra22:41
*** rbrndt has quit IRC22:42
*** rbrndt has joined #openstack-infra22:42
*** rbrndt has quit IRC22:42
*** slaweq has quit IRC22:46
*** signed8bit is now known as signed8bit_Zzz22:49
*** rkukura has quit IRC22:52
*** rkukura has joined #openstack-infra22:52
*** hongbin has quit IRC22:56
openstackgerritMonty Taylor proposed openstack-infra/project-config master: Update site-logs secret to have correct key content  https://review.openstack.org/49544822:57
mordredjeblair: ^^22:57
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Write secrets to tmpfs  https://review.openstack.org/49544922:57
pabelangermordred: ++22:58
mordredpabelanger: ohai22:58
mordredpabelanger: before I do that, was the other key on purpose and maybe we just need to add a new public key to logs.o.o ?22:58
pabelangermordred: that key should be our production key for static.o.o now22:59
pabelangeris there issues?22:59
mordredpabelanger: yah. it's not :)22:59
pabelangerOh, I could have copypasta the wrong secret23:00
mordredok. cool - just wanted to make sure I wasn't going to break something intentional23:00
pabelangerYa, it should be the same as  /var/lib/zuul/ssh/static_id_rsa23:01
pabelangerI can propose a new patch if you'd like23:01
openstackgerritMonty Taylor proposed openstack-infra/zuul-jobs master: Document and update fileserver roles  https://review.openstack.org/49429123:02
mordredpabelanger: nah - that's what I just grabbed - should be ok23:02
mordredpabelanger: https://review.openstack.org/#/c/495448 fwiw23:02
*** Swami has joined #openstack-infra23:03
pabelanger+323:03
mordred\o/23:03
mordredpabelanger: when it lands I'll recheck the zuul-jobs test job23:03
pabelangercool23:05
*** xarses has quit IRC23:05
openstackgerritMonty Taylor proposed openstack-infra/zuul-jobs master: Document and update fileserver roles  https://review.openstack.org/49429123:08
mordredok. I think that's the last of trying to get variable names to not confuse me23:09
pabelangermordred: in the head space to look at a quick glean systemd patch? https://review.openstack.org/49542523:11
*** Julien-zte has quit IRC23:11
pabelangerwill allow for https://review.openstack.org/493665/23:11
*** dhill_ has quit IRC23:11
*** Julien-zte has joined #openstack-infra23:13
mordredpabelanger: lgtm23:14
openstackgerritMerged openstack-infra/project-config master: Update site-logs secret to have correct key content  https://review.openstack.org/49544823:15
*** Apoorva_ has joined #openstack-infra23:18
*** Apoorva has quit IRC23:21
*** xarses has joined #openstack-infra23:21
openstackgerritMonty Taylor proposed openstack-infra/zuul-jobs master: Document and update fileserver roles  https://review.openstack.org/49429123:21
*** Apoorva_ has quit IRC23:22
*** Apoorva has joined #openstack-infra23:23
*** tosky has quit IRC23:29
*** makowals has quit IRC23:30
*** xarses has quit IRC23:31
*** vhosakot has joined #openstack-infra23:31
*** gildub has joined #openstack-infra23:33
openstackgerritMonty Taylor proposed openstack-infra/project-config master: Update base-test to remove non-existent group  https://review.openstack.org/49545623:35
mordredpabelanger: ^^23:35
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Add wrapper driver execution context  https://review.openstack.org/49543923:36
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Write secrets to tmpfs  https://review.openstack.org/49544923:36
openstackgerritMonty Taylor proposed openstack-infra/project-config master: Use delegate_to for the zuul_return  https://review.openstack.org/49545723:36
mordredpabelanger: also - how does that look ^^ ?23:36
*** thorst has joined #openstack-infra23:37
mordredpabelanger: AND - if you have a sec, https://review.openstack.org/#/c/494291/ ? (but no worries if you don't, it's getting later)23:37
*** xarses has joined #openstack-infra23:39
*** xarses has quit IRC23:40
pabelangerdone and done23:41
*** thorst has quit IRC23:41
mordred\o/23:43
mordredpabelanger: we're gonna nail this bad boy23:43
pabelangerYay23:44
*** gildub has quit IRC23:45
*** Apoorva_ has joined #openstack-infra23:48
*** Apoorva has quit IRC23:51
*** gouthamr has quit IRC23:51
openstackgerritMerged openstack-infra/project-config master: Update base-test to remove non-existent group  https://review.openstack.org/49545623:53
*** baoli has joined #openstack-infra23:54
*** baoli has quit IRC23:55
*** jascott1 has quit IRC23:55
*** jascott1 has joined #openstack-infra23:56
openstackgerritMerged openstack-infra/project-config master: Use delegate_to for the zuul_return  https://review.openstack.org/49545723:58
mordredpabelanger: ok. here goes nothing23:58
pabelangerwatching23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!