Saturday, 2016-06-18

*** yamahata has joined #openstack-infra00:00
*** ffrank has quit IRC00:03
openstackgerritDavid Moreau Simard proposed openstack-infra/zuul: WIP: Integrate ARA reporting inside Zuul Launcher  https://review.openstack.org/33087400:11
*** sarob has quit IRC00:13
*** Apoorva has quit IRC00:20
*** mriedem has joined #openstack-infra00:23
*** dimtruck is now known as zz_dimtruck00:23
*** simondodsley has quit IRC00:23
*** rfreire has joined #openstack-infra00:23
rfreireHello there #openstack-infra! First time comitter00:24
*** amitgandhinz has joined #openstack-infra00:24
rfreireGot a failed gerrit job (https://review.openstack.org/#/c/331375/). However, checking the logs (http://logs.openstack.org/75/331375/1/check/gate-tempest-dsvm-full-ceph-plugin-src-os-brick/a980de1/logs/devstacklog.txt.gz), seems that the build setup has failed, not the patch per se.00:24
*** IlyaG has quit IRC00:24
rfreireIs it possible to resubmit the job?00:24
openstackgerritDong Ma proposed openstack-infra/jenkins-job-builder: Use 'false' and 'true' in default docstrings  https://review.openstack.org/33122000:25
*** amitgandhinz has quit IRC00:29
*** IlyaG has joined #openstack-infra00:29
*** flepied has quit IRC00:30
mordredrfreire: yes - just leave a comment on the view of "recheck" (just that one word) and it'll resubmit it for testing00:31
mordredrfreire: and welcome to the fun00:31
rfreiremordred, hey man thanks \ o /00:32
rfreiremonester, btw: Where can I find the 'recheck'? :-}00:32
rfreire<grins>00:32
*** roxanaghe has quit IRC00:32
rfreiremordred, ^00:32
rfreireme--00:32
mordredheh00:32
rfreireor where should I put it00:33
rfreiremordred, is it clicking "reply" and add recheck over there00:35
rfreirereally clueless on that00:35
*** julim has quit IRC00:35
*** IlyaG has quit IRC00:36
*** mtanino has quit IRC00:38
*** tphummel has quit IRC00:39
mordredrfreire: ah - sorry - yes, just reply and have the text of what you say be "recheck"00:42
dmsimardrfreire: yes, you just reply to the review and leave a comment "recheck"00:42
rfreireyay mordred dmsimard _huge_ thank you guys!00:42
rfreiresecret.sauce++00:42
dmsimardhey we love first time committers, thankfully there'll be more :)00:43
rfreireAhahahaha00:43
rfreiredmsimard, I'm really fond of that, count on me, definitely!00:43
*** thorst has joined #openstack-infra00:45
*** hosanai has joined #openstack-infra00:45
*** thorst has quit IRC00:48
*** thorst has joined #openstack-infra00:49
*** hosanai has quit IRC00:50
*** IlyaG has joined #openstack-infra00:51
*** thorst has quit IRC00:58
*** banix has quit IRC01:06
*** zz_dimtruck is now known as dimtruck01:07
*** linuxmaniac_ has joined #openstack-infra01:07
*** linuxmaniac has quit IRC01:08
*** IlyaG has quit IRC01:09
*** tphummel has joined #openstack-infra01:13
openstackgerritSachi King proposed openstack-dev/pbr: Restore warnerrors behavior and support Sphinx 1.4  https://review.openstack.org/22995101:19
*** IlyaG has joined #openstack-infra01:21
*** pvaneck has quit IRC01:22
*** banix has joined #openstack-infra01:22
*** cody-somerville has joined #openstack-infra01:23
*** amitgandhinz has joined #openstack-infra01:25
*** IlyaG has quit IRC01:29
*** amitgandhinz has quit IRC01:30
*** rfreire has quit IRC01:31
*** gouthamr has joined #openstack-infra01:37
dmsimardOh. Vinz. https://github.com/openstack-infra/vinz01:38
dmsimardLook at that Gerrit UI. There be dragons.01:38
*** tphummel has quit IRC01:39
*** IlyaG has joined #openstack-infra01:40
*** IlyaG has quit IRC01:50
*** thorst has joined #openstack-infra01:55
*** thorst has quit IRC02:03
*** yamahata has quit IRC02:09
*** inc0 has joined #openstack-infra02:09
*** rhallisey has quit IRC02:11
*** sshnaidm has quit IRC02:22
*** inc0 has quit IRC02:24
*** amitgandhinz has joined #openstack-infra02:27
*** cody-somerville has quit IRC02:28
*** amitgandhinz has quit IRC02:32
*** zul_ has quit IRC02:34
*** jcoufal has joined #openstack-infra02:50
*** jcoufal has quit IRC02:55
*** thorst has joined #openstack-infra03:01
*** yamahata has joined #openstack-infra03:03
*** jamesmcarthur has joined #openstack-infra03:06
phschwartzdmsimard: yeah, I wish I was allowed to release the source for that. :( I don't even want to go there.03:07
*** cody-somerville has joined #openstack-infra03:07
*** cody-somerville has joined #openstack-infra03:07
*** thorst has quit IRC03:07
*** nwkarsten has joined #openstack-infra03:10
*** jamesmcarthur has quit IRC03:11
openstackgerritgreghaynes proposed openstack/diskimage-builder: WIP: Add support for remote element repos  https://review.openstack.org/33139203:20
greghaynesmordred: ^ You might enjoy that :)03:23
*** amitgandhinz has joined #openstack-infra03:28
phschwartzhmm. yay nodepool hating me today03:28
phschwartzI have 2 providers and 2 labels in my nodepool config with each provider named in the labels. But for some reason the second provider is not even attempting to build any nodes03:29
*** amitgandhinz has quit IRC03:32
phschwartzok, restarted nodepool and it works O.o03:35
phschwartzmy luck03:35
*** nwkarsten has quit IRC03:36
*** nwkarsten has joined #openstack-infra03:37
*** banix has quit IRC03:37
*** r-mibu has quit IRC03:37
*** r-mibu has joined #openstack-infra03:38
*** andreww has joined #openstack-infra03:38
*** flepied has joined #openstack-infra03:39
sdakeapprently when we retired kolla-mesos I did something wrong - we have a zombie review in our queue03:41
*** nwkarsten has quit IRC03:41
sdakecan someone abandon it and advise how I can fix this03:41
sdakehttps://review.openstack.org/#/c/315355/03:41
notmorgansdake: euuuw03:41
sdakenotmorgan ya not pretty :)03:41
notmorgansdake: my guess is if you no longer have ACL access on the repo a gerrit admin can easily handle it for you03:42
phschwartznova client does not support os-cloud-config :(03:42
notmorgansdake: (sadly not me)03:42
sdakenotmorgan yes I removed our acls03:42
notmorgansdake: (or actually, happily not me, I don't need that access :P)03:42
sdakenotmorgan do you happen to know where proposal bot code is stored?03:43
notmorgansdake: hmm. i am fairly certain its in either systemconfig or project-config. i don't remember, it's been a bit since i've looked for it03:44
notmorgans/systemconfig/system-config03:44
greghaynessdake: for requirements theres a script in project-config/tools03:45
greghaynessdake: and it is just kicked off by zuul03:45
notmorgansdake: greghaynes to the rescue03:45
notmorgan:)03:45
* greghaynes swoops in03:45
notmorgangreghaynes: ^_^03:45
sdakegreghaynes cool thanks but what i'm curious about is how does systemconfig know to look at kolla-mesos03:45
sdakeor did I just not abandon all the kolla-mesos changes03:46
sdakeorry project-config i mean03:46
greghaynesoh wait, I might have told you totally the wrong thing03:47
greghaynesI was just looking at this the other day...03:47
sdakenothing about propsal bot was in the retirement instructions03:47
sdakenote i'll submit a docs change (if I can find em:)03:48
greghaynesproject-config/jenkins/scripts/propose_update.sh03:48
greghaynessdake: it just cats the list of projects, maybe there was a race with removing the repo and proposal bot running?03:50
sdakepossible - or i missed abandoning the review03:50
sdakethanks greghaynes03:50
sdakei dont suppose you hae permission to abandon it?03:50
greghaynesI dont03:51
sdakecool i'll ask around monday when folks are working normal hours :)03:51
openstackgerritgreghaynes proposed openstack/diskimage-builder: WIP: Add support for remote element repos  https://review.openstack.org/33139204:03
*** thorst has joined #openstack-infra04:05
*** cody-somerville has quit IRC04:09
*** cody-somerville has joined #openstack-infra04:09
*** thorst has quit IRC04:13
*** nwkarsten has joined #openstack-infra04:18
*** yujunz has joined #openstack-infra04:19
*** nwkarsten has quit IRC04:23
*** amitgandhinz has joined #openstack-infra04:29
*** yujunz has quit IRC04:32
*** amitgandhinz has quit IRC04:33
*** _nadya_ has joined #openstack-infra04:44
*** dimtruck is now known as zz_dimtruck04:54
*** zz_dimtruck is now known as dimtruck04:58
*** jogo has joined #openstack-infra05:00
*** gouthamr has quit IRC05:00
openstackgerritgreghaynes proposed openstack/diskimage-builder: WIP: Add support for remote element repos  https://review.openstack.org/33139205:01
*** yamamoto has joined #openstack-infra05:10
*** yujunz has joined #openstack-infra05:11
*** thorst has joined #openstack-infra05:12
*** yujunz has quit IRC05:13
*** yujunz has joined #openstack-infra05:16
*** thorst has quit IRC05:18
*** yujunz has quit IRC05:20
openstackgerritgreghaynes proposed openstack/diskimage-builder: Add support for remote element repos  https://review.openstack.org/33139205:27
*** swat30 has quit IRC05:27
*** dimtruck is now known as zz_dimtruck05:28
*** swat30 has joined #openstack-infra05:29
*** netsin has joined #openstack-infra05:29
*** amitgandhinz has joined #openstack-infra05:29
odyssey4mejeblair pabelanger FYI the linters job is not the only one seeing the early timeout, for example this one happened on the functional test job: https://review.openstack.org/33129405:33
*** amitgandhinz has quit IRC05:34
odyssey4mejhesketh ^ (also see earlier discussions with jeblair & pabelanger)05:34
*** yujunz has joined #openstack-infra05:40
*** _nadya_ has quit IRC05:40
*** yujunz has quit IRC05:43
*** asselin_ has joined #openstack-infra05:45
openstackgerritKhai Do proposed openstack-infra/puppet-gerrit: Add configurations for gerrit its plugins  https://review.openstack.org/33092205:48
*** asselin_ has quit IRC05:50
*** kushal has joined #openstack-infra05:52
openstackgerritgreghaynes proposed openstack/diskimage-builder: Add support for remote element repos  https://review.openstack.org/33139205:52
*** Sukhdev has joined #openstack-infra05:53
openstackgerritKhai Do proposed openstack-infra/system-config: Configure its-storyboard plugin  https://review.openstack.org/33092505:55
*** kushal has quit IRC05:56
*** kushal has joined #openstack-infra05:58
*** yamamoto has quit IRC06:05
*** sdake has quit IRC06:06
*** armax has quit IRC06:09
*** linuxmaniac_ has quit IRC06:14
*** linuxmaniac has joined #openstack-infra06:15
*** thorst has joined #openstack-infra06:16
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/33140106:19
*** csomerville has joined #openstack-infra06:22
*** thorst has quit IRC06:23
*** zhurong has joined #openstack-infra06:24
*** zhurong has quit IRC06:24
*** somerville32 has joined #openstack-infra06:25
*** csomerville has quit IRC06:25
*** cody-somerville has quit IRC06:25
*** amitgandhinz has joined #openstack-infra06:30
*** amitgandhinz has quit IRC06:35
*** kushal has quit IRC06:38
*** Sukhdev has quit IRC06:41
odyssey4mehmm, infra-root is that intentional? I'm now seeing links like 'telnet://23.253.156.10:8088' showing in zuul for the jobs while running06:48
jheskethodyssey4me: that is correct.. if you do `nc 23.253.156.10 8088` you'll be able to see the running job06:49
jheskethand run grep and all the rest over it :-)06:49
odyssey4meah ok, I just thought that it was a little unusual06:49
jheskeththere are also browser plugins that'll open telnet streams that others were playing with which might make the telnet:// link more useful06:50
jheskeththere's also work to have a websocket to the running job allowing console streaming back in the browser again but that's still WIP06:50
odyssey4mecool :)06:50
nibalizeryah i have a bbs firefox plugin that ... sorta works07:02
*** nwkarsten has joined #openstack-infra07:06
*** yamamoto has joined #openstack-infra07:06
*** somerville32 has quit IRC07:06
*** nwkarsten has quit IRC07:10
*** yamamoto has quit IRC07:10
bkerobbs?07:11
nibalizerbkero: yep07:11
nibalizerbut it makes the telnet:// links clickable so thats nice07:11
bkeroAh07:12
bkerosounds like an in-browser mud cluent07:12
bkeroclient*07:12
*** sdake has joined #openstack-infra07:14
*** thorst has joined #openstack-infra07:20
*** thorst has quit IRC07:27
*** esikachev has joined #openstack-infra07:30
*** amitgandhinz has joined #openstack-infra07:31
AJaegerodyssey4me: did you see jeblair's announcements about Zuul on openstack-dev? That explains it...07:31
*** amitgandhinz has quit IRC07:35
AJaegerjeblair, pabelanger : There's a change for trove in the post queue that is now 13 hours old - it's waiting for trove-tox-doc-publishdocs to start. This is the only element in the post queue. Was the fix from yesterday not enough?07:37
openstackgerritAndreas Florath proposed openstack/diskimage-builder: Refactor: block-device handling (local loop)  https://review.openstack.org/31959107:39
*** yamamoto has joined #openstack-infra07:41
openstackgerritMerged openstack-infra/project-config: Normalize projects.yaml  https://review.openstack.org/33140107:42
*** nwkarsten has joined #openstack-infra07:44
*** nwkarste_ has joined #openstack-infra07:45
*** yamamoto has quit IRC07:46
*** nwkarst__ has joined #openstack-infra07:47
*** nwkarsten has quit IRC07:48
*** nwkarste_ has quit IRC07:49
*** nwkarsten has joined #openstack-infra07:50
*** nwkarst__ has quit IRC07:51
*** nwkarste_ has joined #openstack-infra07:55
*** nwkarsten has quit IRC07:55
*** nwkarste_ has quit IRC07:59
*** zzzeek has quit IRC08:00
*** zzzeek has joined #openstack-infra08:00
*** kzaitsev_mb has joined #openstack-infra08:04
*** Jeffrey4l has quit IRC08:04
*** Jeffrey4l has joined #openstack-infra08:05
*** kzaitsev_mb has quit IRC08:09
*** javeriak has joined #openstack-infra08:11
*** yamamoto has joined #openstack-infra08:15
*** yamamoto has quit IRC08:22
*** thorst has joined #openstack-infra08:25
*** nwkarsten has joined #openstack-infra08:30
*** amitgandhinz has joined #openstack-infra08:32
*** thorst has quit IRC08:33
*** amitgandhinz has quit IRC08:36
*** leifmadsen has quit IRC08:43
*** russellb has quit IRC08:43
*** abregman|afk has joined #openstack-infra08:43
esikachevAJaeger: hi! here?08:47
*** kzaitsev_mb has joined #openstack-infra09:04
*** winggundamth has quit IRC09:17
*** nwkarsten has quit IRC09:26
*** winggundamth has joined #openstack-infra09:29
*** thorst has joined #openstack-infra09:31
*** yamamoto has joined #openstack-infra09:32
*** amitgandhinz has joined #openstack-infra09:32
*** kzaitsev_mb has quit IRC09:35
*** amitgandhinz has quit IRC09:37
*** thorst has quit IRC09:37
*** sdake has quit IRC09:38
*** yamamoto has quit IRC09:58
*** esikachev has quit IRC10:02
*** mikelk has joined #openstack-infra10:25
odyssey4meAJaeger ah, I now see http://lists.openstack.org/pipermail/openstack-dev/2016-June/097595.html :)10:31
*** amitgandhinz has joined #openstack-infra10:33
*** thorst has joined #openstack-infra10:34
*** javeriak has quit IRC10:36
*** amitgandhinz has quit IRC10:38
*** thorst has quit IRC10:42
odyssey4meheh, neat - with iterm you can configure it to be the default handler for telnet - then clicking the link simply opens a new iterm tab10:44
*** flepied has quit IRC10:47
*** flepied has joined #openstack-infra10:49
*** yamamoto has joined #openstack-infra10:58
*** yamamoto has quit IRC11:00
*** yamamoto has joined #openstack-infra11:01
*** nwkarsten has joined #openstack-infra11:26
*** nwkarsten has quit IRC11:30
*** amitgandhinz has joined #openstack-infra11:34
*** amitgandhinz has quit IRC11:38
*** thorst has joined #openstack-infra11:39
*** mikelk has quit IRC11:43
*** mikelk has joined #openstack-infra11:44
*** mikelk has quit IRC11:47
*** mikelk has joined #openstack-infra11:47
*** amrith is now known as __amrith11:48
*** __amrith is now known as amrith11:48
*** thorst has quit IRC11:48
*** mikelk has quit IRC11:59
*** abregman|afk has quit IRC12:08
*** nwkarsten has joined #openstack-infra12:13
*** arif-ali has joined #openstack-infra12:13
*** nwkarsten has quit IRC12:17
openstackgerritMerged openstack-dev/cookiecutter: Change copyright year in sphinx conf file to 2016  https://review.openstack.org/31150212:21
openstackgerritMerged openstack-dev/cookiecutter: add reno to the default template  https://review.openstack.org/32609812:23
mordredphschwartz: question for you on 31767812:32
mordredphschwartz: also, https://review.openstack.org/#/c/245200/12:34
*** amitgandhinz has joined #openstack-infra12:35
*** nwkarsten has joined #openstack-infra12:38
*** amitgandhinz has quit IRC12:39
*** nwkarsten has quit IRC12:42
*** esikachev has joined #openstack-infra12:43
*** yamamoto has quit IRC12:45
*** nwkarsten has joined #openstack-infra12:46
*** thorst has joined #openstack-infra12:47
*** thorst has quit IRC12:48
*** thorst has joined #openstack-infra12:48
*** thorst has quit IRC12:49
*** jcoufal has joined #openstack-infra12:49
*** nwkarsten has quit IRC12:51
*** yamamoto has joined #openstack-infra12:58
openstackgerritJeffrey Zhang proposed openstack-infra/project-config: Enable the Ubuntu and CentOS gate voting for Kolla  https://review.openstack.org/33092713:05
*** nwkarsten has joined #openstack-infra13:07
*** esikachev has quit IRC13:08
*** nwkarsten has quit IRC13:11
*** zul_ has joined #openstack-infra13:12
openstackgerritMerged openstack-infra/project-config: Disable Tempest plugins gate for Telemetry in stable/liberty  https://review.openstack.org/33122513:13
*** asselin_ has joined #openstack-infra13:15
openstackgerritMerged openstack-infra/project-config: [trove] Add trove-dashboard to PROJECTS  https://review.openstack.org/33135213:15
*** javeriak has joined #openstack-infra13:16
*** arif-ali has quit IRC13:17
openstackgerritMerged openstack-infra/project-config: Run novaclient functional tests in Identity v3-only  https://review.openstack.org/31358813:17
openstackgerritMerged openstack-infra/project-config: Add murano periodic jobs with oslo master  https://review.openstack.org/32944013:18
*** baoli has joined #openstack-infra13:18
*** javeriak has quit IRC13:21
openstackgerritDean Troyer proposed openstack/os-client-config: Refactor fix magic in get_one_cloud()  https://review.openstack.org/33141813:23
*** baoli has quit IRC13:23
*** arif-ali_ has joined #openstack-infra13:30
openstackgerritMerged openstack-dev/pbr: Sort 'pbr freeze' output  https://review.openstack.org/29976613:31
zaroinfo-root i think gerrit could use another restart right now.  running slow again, http://cacti.openstack.org/cacti/graph.php?local_graph_id=27&rra_id=all13:34
*** amitgandhinz has joined #openstack-infra13:35
openstackgerritFerenc Horváth proposed openstack-infra/project-config: Add job which builds test images with watchdog support  https://review.openstack.org/32983213:37
*** amitgandhinz has quit IRC13:40
*** sdake has joined #openstack-infra13:44
*** sdake_ has joined #openstack-infra13:47
*** sdake has quit IRC13:49
*** armax has joined #openstack-infra13:49
*** thorst has joined #openstack-infra13:49
*** Wei_Liu has quit IRC13:56
*** Wei_Liu has joined #openstack-infra13:56
*** thorst has quit IRC13:58
*** baoli has joined #openstack-infra13:59
*** andreww has quit IRC14:03
Jeffrey4lis there any possible use a hight kernel version for the ubuntu trusty image in the CI?14:06
Jeffrey4l3.13 is a little old. A docker bug block kolla. We need 3.16 kernel at least.14:06
EmilienMJeffrey4l: what is provided by Canonical?14:07
Jeffrey4li installed a new ubuntu 14.04.2 LTS server, it has linux-image-3.16.0-30-generic for default kernel. EmilienM14:08
Jeffrey4lwhy the ubuntu image from  the ci is using a old one 3.13 kernel?14:08
EmilienMJeffrey4l: as a workaround for now, I would just run apt-get upgrade in kolla CI scripts14:09
Jeffrey4lEmilienM, but for the kernel, we need a reboot, right?14:09
EmilienMJeffrey4l: if you want to investigate how trusty image is built, download and dig http://nodepool.openstack.org/dib.ubuntu-trusty.log14:09
EmilienMJeffrey4l: yeah or soft reboot14:10
mordredyah - we use debootstrap for the base image - perhaps it's not installing the latest trusty updates kernel?14:10
Jeffrey4lEmilienM, is there any possible my ci scripts can could still running after a rebooting?14:11
*** thorst has joined #openstack-infra14:11
Jeffrey4lEmilienM, mordred thanks for the debootstrap info. I will try to find why the image still using a old kernel.14:12
mordredit looks like the updates repo is not enabled when the debootstrap command is run14:12
mordredit would be worth us discussing generally if we should enable it or not - maybe at the tuesday meeting14:13
*** thorst has quit IRC14:13
Jeffrey4lcool. thanks mordred14:14
*** baoli has quit IRC14:14
clarkbJeffrey4l: mordred that kernel is not installed by ubuntu by default. that is their hardware support kernel or whatever and you have to opt into it14:17
clarkbwe can do this but havent in the past. just a matter of deciding if it makes sense I think14:18
*** armax has left #openstack-infra14:19
*** armax has joined #openstack-infra14:20
Jeffrey4lclarkb, in my vm. using the same sources.list with dib. when running `apt-get dist-upgrade` the new kernel is installed. But in the dib's log, it only install the 3.13 kernel. any ideas for why?14:21
clarkbyou probably started with an image that included that newer kernel14:22
clarkbI do not think that the sources change instead the packagename does14:22
clarkbhttps://wiki.ubuntu.com/Kernel/LTSEnablementStack14:23
Jeffrey4lhmm..nice explanation14:25
*** Jeffrey4l has left #openstack-infra14:26
*** Jeffrey4l has joined #openstack-infra14:26
EmilienMI've got a lot of 502 on gerrit, for ~20 min14:26
Jeffrey4lclarkb, so is there any chance that the CI will use the latest kernel?14:26
Jeffrey4lnow the old kernel blocked one kolla's job.14:27
clarkbJeffrey4l: as mordred said we have the option we just have to decide if we want to do it and figure out if it will break anything14:27
Jeffrey4lcool. thanks.14:27
zaroEmilienM: yeah me too.  i’ve asked an infra-root to restart14:27
Jeffrey4lanother issue: in the rax-ord nodes, there are two Ethernets. sometimes, the second one ( eth1 ) has no ip address. http://logs.openstack.org/95/331395/2/check/gate-kolla-dsvm-build-oraclelinux-source/b8ea832/console.html#_2016-06-18_13_11_19_72713814:31
Jeffrey4lbut the /etc/nodepool/node_private has the ip address for eth114:32
Jeffrey4lhttp://logs.openstack.org/95/331395/2/check/gate-kolla-dsvm-build-oraclelinux-source/b8ea832/console.html#_2016-06-18_13_13_37_19654614:32
Jeffrey4lwhen ifdown eth1 && ifup eth1 the ip address show up.14:32
clarkbI thought that was addressed by removing the default dhcp for eth1 config when using glean14:35
clarkbmordred ^14:35
*** amitgandhinz has joined #openstack-infra14:36
*** esikachev has joined #openstack-infra14:38
*** amitgandhinz has quit IRC14:41
mordredclarkb: there's that, but there's also just the updated 3.x series kernel that's in the normal trusty-updates repo (which is 3.16)14:46
mordredclarkb: that said, I don't have much of a strong opinion on this either way14:46
mordredclarkb: and hrm, yeah. we should be configuring those second interfaces with glean - I really hope something hasn't broken14:47
mordredgreghaynes: ^^14:47
*** gouthamr has joined #openstack-infra14:47
*** kzaitsev_mb has joined #openstack-infra14:50
*** HenryG has quit IRC14:51
*** HenryG has joined #openstack-infra14:51
*** gouthamr_ has joined #openstack-infra14:53
clarkbmordred: aiui that updated kernel is the lts enablement kernel14:54
clarkbyou can install iy no problem but if ypu start on 3.13 it wont be automagic14:54
*** gouthamr has quit IRC14:56
*** baoli has joined #openstack-infra14:56
*** gouthamr_ is now known as gouthamr14:57
*** baoli has quit IRC15:00
jeblair[small] earthquake!15:05
jeblairhttp://earthquake.usgs.gov/earthquakes/eventpage/nc72652676#general15:05
*** esikachev has quit IRC15:06
*** kzaitsev_mb has quit IRC15:08
openstackgerritJim Rollenhagen proposed openstack-infra/project-config: Make ironic grenade voting  https://review.openstack.org/33142215:11
*** kzaitsev_mb has joined #openstack-infra15:12
SpamapSjeblair: oh my, close epicenter to you15:13
*** thorst has joined #openstack-infra15:13
jeblairmordred, clarkb: we intentionally disabled the hardware enablement repo on our systems in the providers where it did show up because it broke all our afs clients (which illustrates the poor testing and fit with the rest of the lts).  i expect you will find people that think either option is correct.  they are both 'supported' from a security pov at least.  as a lazy sysadmin, my view is "i dont want a new kernel rev unless i have to" ...15:16
jeblair... so i wouldn't enable it unless i needed to for hardware15:16
jeblairSpamapS: yes -- on the fault that has a 27% chance of a major quake in the next 20 years15:18
*** thorst has quit IRC15:22
openstackgerritJim Rollenhagen proposed openstack-infra/project-config: Use ironic-core for ironic-python-agent ACLs  https://review.openstack.org/33142515:23
SpamapSjeblair: 2.0's on those faults are supposedly good things, relieving pressure. Probably bought Oakland a few months more before an 8.0 throws it into the bay.15:24
*** sdake_ has quit IRC15:28
openstackgerritPaul Belanger proposed openstack-infra/zuul: Move job timeout into playbook as vars  https://review.openstack.org/33142615:30
pabelangerjeblair: mordred: odyssey4me: ^ Not that it fixes our current timeout issue, but a cosmetic change to add our initial timeout value into our playbooks.  Example playbook I tested with: http://paste.openstack.org/show/518882/15:31
pabelangerI also couldn't think of a good way to work with timeout, ourside set_facts15:32
*** nwkarsten has joined #openstack-infra15:33
jeblairpabelanger: i think that's fine -- but before we land that or the debug change, i want to figure out why the timeout callback plugin isn't working.  in the mean time, i think i've determined from local testing that the timeout for every task is the job timeout, so that doesn't help us understand odyssey4me's problem.  i think we should grab a node and re-run one of those jobs manually with -vvv (which will be SUPER easy to do to with ...15:35
jeblair... the uploaded playbooks :)15:35
phschwartzmordred: as to 317678 yes, it is still needed,fell to the side as no one else reviewed it.15:36
pabelangerjeblair: WFM15:36
odyssey4meJeffrey4l pabelanger hmm, let me find an unmerged one15:36
jeblairodyssey4me: also, does this impact every linter run?15:36
phschwartzmordred: As to 245200 what do you need from me for it? It seems to need to be rebased and a few touch ups. code looks solid like always15:36
odyssey4mejeblair nope, with enough rechecks and prayers to the gods of randomness it sometimes passes15:37
*** amitgandhinz has joined #openstack-infra15:37
odyssey4mewe got a bunch through the door over the last 48 hours15:37
odyssey4mewe have other failures on different jobs that are similar, but not the same15:37
jeblairyeah, i figured enough of the examples you linked were linters to make it worth considering the nature/content of the job15:38
odyssey4mesee the review comments in this one: https://review.openstack.org/329705 - I noted a 'failed to transfer file' error in the ansible log15:38
odyssey4meit also has to do with the async execution15:39
odyssey4methat's hit this one once too: https://review.openstack.org/331301 - you can test with it if you like as it's not merged15:40
odyssey4mehere's a linters one that seems to consistently be failing: https://review.openstack.org/33129815:41
odyssey4meargh, that's because it fails the actual lint test :p15:41
odyssey4menever mind15:41
*** amitgandhinz has quit IRC15:42
odyssey4mejeblair here's an unmerged patch which has failed the linters check before but passed on the last test: https://review.openstack.org/33129415:44
AJaegerjeblair, pabelanger : We have changes in the post queue - for 21h and 8h - still as queued ;(15:49
AJaegerIt's always the publishdocs and publishlang jobs15:50
jeblairi'm sorry, i'm not able to look into those issues now15:50
*** tpsilva has joined #openstack-infra15:52
*** zqfan has quit IRC15:53
odyssey4mejeblair need any more samples?15:53
jeblairodyssey4me: let's maybe collect them in an etherpad for further investigation: https://etherpad.openstack.org/p/zuul-timeouts15:54
AJaegerjeblair: sure - this can wait until Monday... We can life without publish jobs for a day or two...15:55
AJaegerjeblair: have a great weekend - without further earthquakes15:55
*** zz_dimtruck is now known as dimtruck15:55
jeblairthanks :)15:56
*** banix has joined #openstack-infra15:58
pabelangerAJaeger: jeblair: looks like another exception. I'll trying get a patch up for monday15:59
AJaegerthanks, pabelanger. A great weekend to you as well16:00
fungipabelanger: the one where build.start_time is None and we're trying to subtract build.launch_time (type float) from it?16:01
pabelangerfungi: Ya, I believe that is the reason for the failure16:02
pabelangerI don't understand why it is None at the moment16:02
fungishould we just coerce build.start_time to 0.0 if None?16:02
AJaegerpabelanger: are you talking about the post jobs that I mentioned or the timeout issue of odyssey4me ?16:03
pabelangerfungi: I am not sure, if understanding the code it would see that onBuildStarted was possible missed. Not sure how that is possible16:03
pabelangerAJaeger: your issue16:03
fungihrm, i guess the real question is why we're hitting onBuildCompleted when build.start_time is None16:03
fungiyeah16:03
pabelangerfungi: Agreed16:03
AJaegerpabelanger: I see - I was confused since you mentioned time ;)16:04
fungimaybe it's insta-aborting?16:04
pabelangerAJaeger: indeed16:04
fungitime is an illusion16:05
AJaegeralways16:05
fungion the launchers we're also gettign a lot of "Exception: Target path / is not below site root" from _makeFTPTask16:06
fungiwhich could be related i think16:07
pabelangerOh, maybe that is our path issue again for FTP tasks16:07
fungigiven the broken jobs are all ftp publisher jobs...16:07
AJaegeryes, they are ftp publisher jobs16:07
fungianyway, i'm done swooping in and making wild assertions for the moment... have some non-openstack weekend tasks i need to get back to16:08
pabelangerfungi: AJaeger: It is likely this issue https://review.openstack.org/#/c/329947/ which I abandoned. I thought we fixed it in zuul-launcher16:09
fungibut i suspect 1. we need to fix the configuration for those? 2. the launcher should be robust in the face of that exception? 3. the scheduler should probably not choke when it gets a completion event like that without any start time registered?16:09
AJaegerthe ftp publisher does not set root at all - should it?16:09
fungiAJaeger: / may be a default value16:10
*** Goneri has quit IRC16:10
pabelangerwill look more into it tonight / tomrrow16:10
AJaegeryes, that was supposed to be fixed16:10
fungithere was some discussion that for the rsync publisher / was relative to the site root, so maybe that assumption leaked into the ftp task?16:10
fungiaha, yes 329947 takes care of #2 and i guess #1 is irrelevant because it's falling back on a default value?16:12
fungioh, er other way around16:12
AJaegerhttps://review.openstack.org/#/c/330090/ was supposed to fix it16:12
pabelangerI think I see the issue, we lstrip('/') the wrong thing for FTP16:13
*** dimtruck is now known as zz_dimtruck16:13
pabelangerIn the review AJaeger posted, I think line 969 is the issue. target will still contain '/'16:14
jeblairpabelanger: yeah, that should be ftptarget16:15
pabelangeryup16:15
AJaegerwhenever you have something merged, tell me and I'll find some publishing job to approve ;)16:17
openstackgerritPaul Belanger proposed openstack-infra/zuul: Use ftptarget since leading / is stripped  https://review.openstack.org/33142816:17
pabelangerthat should fix it ^16:18
pabelangererr16:18
pabelanger1 sec16:18
pabelangerya, that is right16:19
AJaegerthanks, pabelanger16:19
pabelangercompared it scptask16:19
pabelangerAJaeger: enjoy the weekend!16:20
AJaegerthanks16:20
*** thorst has joined #openstack-infra16:21
*** banix has quit IRC16:22
fungii don't think the lstrip() makes any difference with os.path.join() in the mix, though the _substituteVariables() is likely still important and gets solved with 33142816:24
fungioh! it does16:25
fungios.path.join('foo', '/bar') returns '/bar'16:26
fungii never realized it had that behavior16:26
fungii approved 331428 but we're going to need to restart all the launchers once it merges16:27
fungii suppose we could just put half of them in graceful restart at a time given the current job volume16:27
*** thorst has quit IRC16:28
pabelangerfungi: I just did a recheck on 331362 as jeblair has some concerns with it16:28
pabelangerwe should land that before restart16:28
fungik16:28
AJaegerspeaking about restart: gerrit is quite slow at the moment ;(16:29
pabelangerfungi: any chance we can give 330184 a runs for the zuul-launcher restarts?16:29
odyssey4meAJaeger agreed - I've had several proxy errors16:29
AJaegerzaro asked at 13:34 for a gerrit restart..16:30
*** arif-ali_ has quit IRC16:31
fungioh yeah, the javamelody graphs show it's spending a lot of time doing java garbage collection16:31
AJaegerbye for now, enjoy the weekend16:32
fungionce memory usage climbed up around 30g it kicked in16:32
jeblairfungi, pabelanger: yeah, please land 331362 before restarting launchers.16:33
openstackgerritMerged openstack-infra/zuul: Use ftptarget since leading / is stripped  https://review.openstack.org/33142816:33
jeblairunless you want to experiment with that in production (not how i want to spend my weekend)16:33
pabelangeragreed16:33
fungii've queued up a gerrit restart for as soon as zuul isn't on the verge of reporting a result16:34
*** Wei_Liu has quit IRC16:34
*** Wei_Liu has joined #openstack-infra16:35
fungilooks like we're probably clear for a restart now16:35
fungi#status notice Gerrit is restarting now to relieve memory pressure and restore responsiveness16:35
openstackstatusfungi: sending notice16:35
fungiit should be back up again nowish16:36
-openstackstatus- NOTICE: Gerrit is restarting now to relieve memory pressure and restore responsiveness16:37
fungihrm, still 50316:37
fungithere it goes16:37
fungi331362 is slated to merge in ~4 minutes16:38
*** amitgandhinz has joined #openstack-infra16:38
openstackstatusfungi: finished sending notice16:38
fungifor some reason gertty won't load 33018416:38
fungi"Error: Git commits not present in local repository"16:39
fungitried several times16:39
fungiahh, finally worked16:39
openstackgerritMerged openstack-infra/zuul: Revert "Enabled host_key_checking"  https://review.openstack.org/33136216:40
fungipabelanger: 330184 looks like it will take quite a while, though i'm willing to give it a shot anyway16:40
fungii was figuring i'd graceful restart 4 launchers at a time16:41
fungianyway, need to step away for a few16:41
pabelangerfungi: Right, we default to 1 at a time in 330184 but maybe we can bump that to 2 or more based on the timeout day16:42
pabelangertime of*16:42
pabelangerserial: "30%" or something16:43
*** amitgandhinz has quit IRC16:43
*** Wei_Liu has quit IRC16:44
fungiright now the job volume is so low that we could get by on one zl (plus zlstatic01), so restarting half of them at a time should be perfectly safe16:44
*** Wei_Liu has joined #openstack-infra16:44
fungialso, that playbook will miss zlstatic0116:45
pabelangerYa, I'll need to add that16:46
openstackgerritPaul Belanger proposed openstack-infra/zuul: Move job timeout into playbook as vars  https://review.openstack.org/33142616:48
openstackgerritPaul Belanger proposed openstack-infra/zuul: Remove unused timeout args for _makeBuilderTask  https://review.openstack.org/33142916:48
openstackgerritPaul Belanger proposed openstack-infra/system-config: Add playbook to graceful restart zuul-launchers  https://review.openstack.org/33018416:52
*** zz_dimtruck is now known as dimtruck16:57
fungipabelanger: 330184 lgtm, but i'd suggest running it yourself from a local copy first as a dry run for this restart to double-check it so i didn't approve yet in case you find more bugs when testing16:57
pabelangerfungi: Agreed, I haven't tested these latest changes live yet. I limit the playbook to zl01.openstack.org for a test17:01
pabelangerI'll*17:01
*** ri0 has joined #openstack-infra17:02
pabelangerokay, restarting zl0117:03
*** nwkarsten has quit IRC17:03
pabelangergraceful that is17:04
*** dimtruck is now known as zz_dimtruck17:06
*** superdan is now known as dansmith17:11
*** davideagnello has joined #openstack-infra17:15
*** davideagnello has quit IRC17:17
pabelanger#status log zl01 currently graceful restarting via 33018417:21
openstackstatuspabelanger: finished logging17:21
fungipopping out to grab lunch and run a couple errands. bbiaw17:22
*** thorst has joined #openstack-infra17:25
*** arif-ali has joined #openstack-infra17:25
*** x00350071_ has joined #openstack-infra17:26
*** xiangxinyong has quit IRC17:26
*** thorst has quit IRC17:33
*** amitgandhinz has joined #openstack-infra17:39
*** amitgandhinz has quit IRC17:43
pabelanger#status log zl01 restarted properly17:44
openstackstatuspabelanger: finished logging17:44
pabelangerfungi: playbook worked as expected17:45
pabelangeralso stepping a way for a few hours, will check backin in a while17:45
*** andreww has joined #openstack-infra17:58
*** maishsk has joined #openstack-infra18:10
*** Sukhdev has joined #openstack-infra18:12
*** bmjen has quit IRC18:15
*** maishsk has quit IRC18:16
*** javeriak has joined #openstack-infra18:17
*** ri0 has quit IRC18:18
*** _bmjen has joined #openstack-infra18:18
*** _bmjen is now known as bmjen_18:18
*** javeriak_ has joined #openstack-infra18:21
*** javeriak has quit IRC18:21
*** tphummel has joined #openstack-infra18:25
*** thorst has joined #openstack-infra18:33
*** amit213 has quit IRC18:34
*** amit213 has joined #openstack-infra18:37
*** thorst has quit IRC18:38
*** amitgandhinz has joined #openstack-infra18:39
*** mtanino has joined #openstack-infra18:40
*** amitgandhinz has quit IRC18:44
*** maishsk has joined #openstack-infra18:45
*** mrmartin has joined #openstack-infra18:45
*** amit213 has quit IRC18:49
*** gouthamr has quit IRC18:49
*** amit213 has joined #openstack-infra18:49
*** amit213 has quit IRC18:56
*** amit213 has joined #openstack-infra18:57
*** zz_dimtruck is now known as dimtruck18:58
*** amit213 has quit IRC19:02
*** amit213 has joined #openstack-infra19:03
*** dimtruck is now known as zz_dimtruck19:07
*** banix has joined #openstack-infra19:08
*** bapalm has quit IRC19:08
*** ajmiller_ has quit IRC19:16
*** ajmiller has joined #openstack-infra19:17
*** nwkarsten has joined #openstack-infra19:28
*** Jeffrey4l_ has joined #openstack-infra19:31
*** _nadya_ has joined #openstack-infra19:32
*** nwkarsten has quit IRC19:33
*** Jeffrey4l has quit IRC19:34
*** banix has quit IRC19:37
openstackgerritMerged openstack-infra/system-config: Add playbook to graceful restart zuul-launchers  https://review.openstack.org/33018419:38
*** _nadya_ has quit IRC19:40
*** amitgandhinz has joined #openstack-infra19:40
*** mriedem1 has joined #openstack-infra19:44
*** amitgandhinz has quit IRC19:45
*** mriedem has quit IRC19:46
*** mriedem1 is now known as mriedem19:46
*** mriedem1 has joined #openstack-infra19:58
*** Sukhdev has quit IRC19:58
*** mriedem has quit IRC20:00
*** tphummel has quit IRC20:02
*** esikachev has joined #openstack-infra20:04
*** rwsu has quit IRC20:25
*** hongbin has joined #openstack-infra20:32
*** rwsu has joined #openstack-infra20:37
jpmaxmanfungi clarkb pabelanger I was going to take a look at the wiki.o.o machine - wondering how often the config file is being pushed by the puppet config - in other words if I make a change it will be overwritten right?20:37
*** thorst has joined #openstack-infra20:39
*** jcoufal has quit IRC20:40
*** amitgandhinz has joined #openstack-infra20:41
*** zeih has joined #openstack-infra20:45
*** amitgandhinz has quit IRC20:45
*** thorst has quit IRC20:47
*** nwkarsten has joined #openstack-infra20:49
*** hongbin has quit IRC20:52
fungijpmaxman: yes, every ~15 minutes _but_ we can put the server in our "emergency skip" list and it won't be updated until we take it back out20:53
pabelangerand back20:53
*** nwkarsten has quit IRC20:53
pabelangerfungi: did you by chance graceful restart other launchers?  If not, I was going to do that now20:54
*** mrmartin has quit IRC20:55
fungijpmaxman: or for a longer-term disablement, push a change to add it to the disable list in groups.txt as documented at http://docs.openstack.org/infra/system-config/sysadmin.html#disable-enable-puppet and one of our infra-core reviewers can merge it20:55
fungipabelanger: i did not, feel free to proceed20:55
*** esikachev has quit IRC20:56
jpmaxmanok cool - not sure it will be needed at this point, just wanted to make sure I understood how frequently it was being pushed.  Will keep you posted.20:58
*** zz_dimtruck is now known as dimtruck20:58
fungijpmaxman: great, thanks! let us know if you need anything20:59
jpmaxmanfungi I haven't ssh'd in yet, but was just looking at the front end web and tried to login with my launchpad creds20:59
jpmaxmanI get a 500 error20:59
fungijpmaxman: hah, you may not have an account on there yet20:59
jpmaxmanI see - so that's what happens if you try to login with launchpad that hasn't created an account21:00
fungijpmaxman: once you ssh in, comment out line 149 in /srv/mediawiki/Settings.php and try again21:00
*** sheeprine has quit IRC21:00
jpmaxmanhmm well I don't want to change a config file - just trying to login via the web as a regular user21:02
jpmaxmangetting: Verification of the OpenID URL failed. Error message: "Nonce already used or out of range"21:02
jpmaxmandid you disable logins?21:02
jpmaxmanwant to understand where things are currently before changing anything :)21:03
fungijpmaxman: we set $wgGroupPermissions['*']['createaccount'] = false; at http://git.openstack.org/cgit/openstack-infra/puppet-mediawiki/tree/templates/Settings.php.erb#n14921:03
pabelangerwe disable new users creation21:03
jpmaxmanahh ok21:03
jpmaxmanso this is expected as part of the spam prevention you've already put in place21:03
fungijpmaxman: so if you haven't logged into the wiki before, it will throw that error. the first time you log in (via launchpad openid) an account will be created and then you can log in via openid from that point on21:03
jpmaxmangot it - ok - thanks.21:04
fungiif that line of the config is commented out21:04
jpmaxmanthen I can create my account21:04
jpmaxmanok21:04
*** esikachev has joined #openstack-infra21:04
fungiright, it basically automatically creates your account on login, so trying to log in for the first time breaks with new account creation disabled21:04
*** PollyZ has joined #openstack-infra21:05
jpmaxmanok cool :)21:05
*** sheeprine has joined #openstack-infra21:05
fungiwe've been temporarily commenting that line out on request so legitimate users can create new accounts, without leaving it wide open for now21:06
fungiwhich seems to have put an end to the spam (coupled with blocking all previously spamming accounts), but is obviously inconvenient and not a good user experience21:07
*** dimtruck is now known as zz_dimtruck21:08
*** esikachev has quit IRC21:09
*** javeriak_ has quit IRC21:11
*** PollyZ has quit IRC21:13
*** mtanino has quit IRC21:22
jpmaxmanright21:25
*** ihrachys has joined #openstack-infra21:28
*** maishsk has quit IRC21:34
*** nwkarsten has joined #openstack-infra21:34
*** nwkarsten has quit IRC21:39
*** ffrank has joined #openstack-infra21:39
*** maishsk has joined #openstack-infra21:39
jpmaxmanok fungi I created the user - can you promote to adminstrator so I can see this page?21:40
jpmaxmanhttps://wiki.openstack.org/w/index.php?title=Special:SmiteSpam21:40
*** amitgandhinz has joined #openstack-infra21:42
*** thorst has joined #openstack-infra21:45
*** amitgandhinz has quit IRC21:46
*** jsavak has joined #openstack-infra21:50
*** thorst has quit IRC21:53
*** jsavak has quit IRC21:55
*** jsavak has joined #openstack-infra21:56
*** jsavak has quit IRC22:03
*** jsavak has joined #openstack-infra22:04
pabelangerfungi: okay, all launchers restarted22:05
*** tpsilva has quit IRC22:07
*** nwkarsten has joined #openstack-infra22:08
*** ihrachys has quit IRC22:09
*** nwkarsten has quit IRC22:12
fungijpmaxman: done, sorry was bolting things to walls and forgot to check back in until now22:13
fungipabelanger: awesome, thanks!22:14
*** jsavak has quit IRC22:14
jpmaxmanCool thanks.22:16
*** haaj has joined #openstack-infra22:16
*** PollyZ has joined #openstack-infra22:16
jpmaxmangoing to have to step away to watch some copa de america, but will pick it back up tomorrow.22:16
*** PollyZ has quit IRC22:21
*** gouthamr has joined #openstack-infra22:32
*** banix has joined #openstack-infra22:34
fungijpmaxman: have fun, and thanks again!22:34
*** haaj has quit IRC22:39
*** amitgandhinz has joined #openstack-infra22:42
*** amitgandhinz has quit IRC22:47
*** yamamoto has quit IRC22:48
*** thorst has joined #openstack-infra22:49
*** thorst has quit IRC22:58
*** zz_dimtruck is now known as dimtruck22:59
*** yamamoto has joined #openstack-infra23:00
*** yamamoto has quit IRC23:00
openstackgerritPaul Belanger proposed openstack-infra/system-config: Remove jjb playbook  https://review.openstack.org/33143923:01
openstackgerritPaul Belanger proposed openstack-infra/system-config: Remove jenkins from site.pp  https://review.openstack.org/33144023:05
*** claudiub has joined #openstack-infra23:06
*** abregman has joined #openstack-infra23:07
*** dimtruck is now known as zz_dimtruck23:09
openstackgerritMerged openstack-infra/system-config: Add #openstack-charms to statusbot  https://review.openstack.org/32915923:15
*** zz_dimtruck is now known as dimtruck23:20
*** openstackstatus has quit IRC23:40
*** openstackstatus has joined #openstack-infra23:41
*** ChanServ sets mode: +v openstackstatus23:41
*** amitgandhinz has joined #openstack-infra23:43
*** kzaitsev_mb has quit IRC23:44
*** amitgandhinz has quit IRC23:48
*** thorst has joined #openstack-infra23:52
*** thorst has quit IRC23:52
*** thorst has joined #openstack-infra23:53
*** dimtruck is now known as zz_dimtruck23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!