Monday, 2016-02-29

*** sdake has joined #openstack-infra00:00
nibalizer+2a on your patch00:00
jheskethokay looks like ansible is recovering00:02
jheskethdaemontool_: so once ansible finishes the jenkins masters should have your change and therefore a recheck should work00:03
nibalizerjhesketh: looks good00:07
jheskethI'll poke at the inventory shortly to not fail if a cloud is down00:07
nibalizerits copied out of ansible upstream iirc00:08
*** thorst has joined #openstack-infra00:08
*** amitgandhinz has quit IRC00:08
*** Qiming_ has quit IRC00:14
*** wmolicki has joined #openstack-infra00:16
*** roxanagh_ has quit IRC00:17
*** roxanaghe has joined #openstack-infra00:18
*** sflanigan has joined #openstack-infra00:19
*** thorst has quit IRC00:27
*** wmolicki has quit IRC00:28
*** wmolicki has joined #openstack-infra00:28
*** beekneemech has quit IRC00:29
*** bnemec has joined #openstack-infra00:31
*** roxanaghe has quit IRC00:32
*** roxanaghe has joined #openstack-infra00:34
*** zeih has joined #openstack-infra00:38
openstackgerritMerged openstack-infra/irc-meetings: Change Artifacts meeting chair & meeting ID
*** markvoelker has joined #openstack-infra00:43
*** zeih has quit IRC00:43
*** wmolicki has quit IRC00:44
*** sdake has quit IRC00:45
*** markvoelker has quit IRC00:47
*** fedexo has joined #openstack-infra00:47
*** sdake has joined #openstack-infra00:50
*** bgaifullin has quit IRC00:55
*** achanda has quit IRC01:00
*** hayato_ has joined #openstack-infra01:00
*** zeih has joined #openstack-infra01:00
*** Jeffrey4l has joined #openstack-infra01:01
*** Daisy has joined #openstack-infra01:03
*** x00350071 has joined #openstack-infra01:04
*** amitgandhinz has joined #openstack-infra01:04
*** zeih has quit IRC01:05
*** sdake has quit IRC01:06
*** sdake has joined #openstack-infra01:08
*** salv-orlando has joined #openstack-infra01:12
*** Qiming_ has joined #openstack-infra01:14
*** amitgandhinz has quit IRC01:17
*** aysyd has joined #openstack-infra01:20
*** salv-orlando has quit IRC01:20
*** camunoz has quit IRC01:26
*** hichihara has joined #openstack-infra01:28
*** sam_wan has joined #openstack-infra01:29
*** aysyd has quit IRC01:31
*** Daisy has quit IRC01:32
*** Daisy has joined #openstack-infra01:32
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: Enable network isolation on all the jobs.
*** claudiub has joined #openstack-infra01:35
*** Daisy has quit IRC01:36
*** x00350071 is now known as xiangxinyong01:36
*** sdake has quit IRC01:43
*** camunoz has joined #openstack-infra01:43
*** Daisy has joined #openstack-infra01:43
*** markvoelker has joined #openstack-infra01:44
*** Daisy has quit IRC01:46
*** Daisy has joined #openstack-infra01:46
*** markvoelker has quit IRC01:48
*** Daisy_ has joined #openstack-infra01:50
*** yamamoto_ has joined #openstack-infra01:50
*** Daisy has quit IRC01:51
*** gildub has joined #openstack-infra01:52
*** yamamoto_ has quit IRC01:55
*** Sukhdev has joined #openstack-infra01:56
*** zeih has joined #openstack-infra02:01
*** jpr has quit IRC02:04
*** zeih has quit IRC02:06
*** Daisy_ has quit IRC02:06
*** Qiming_ has quit IRC02:07
*** Qiming has joined #openstack-infra02:08
*** yamamoto_ has joined #openstack-infra02:08
*** Daisy has joined #openstack-infra02:09
*** yamamoto_ has quit IRC02:09
*** achanda has joined #openstack-infra02:10
*** Sukhdev has quit IRC02:13
*** Sukhdev has joined #openstack-infra02:13
*** amitgandhinz has joined #openstack-infra02:14
*** achanda has quit IRC02:14
*** sripriya has joined #openstack-infra02:15
*** achanda has joined #openstack-infra02:15
*** camunoz has quit IRC02:21
*** achanda has quit IRC02:22
*** dkranz has quit IRC02:24
*** ajmiller has quit IRC02:25
*** rguillebert has quit IRC02:27
*** amitgandhinz has quit IRC02:27
*** thorst has joined #openstack-infra02:28
*** Sukhdev has quit IRC02:31
*** sdake has joined #openstack-infra02:31
*** camunoz has joined #openstack-infra02:33
*** rhallisey has quit IRC02:33
*** yamamoto has joined #openstack-infra02:33
*** Keedya has joined #openstack-infra02:34
*** thorst has quit IRC02:36
*** salv-orlando has joined #openstack-infra02:37
*** Daisy has quit IRC02:38
*** Daisy has joined #openstack-infra02:38
*** binbincong_ has quit IRC02:39
*** yamamoto has quit IRC02:40
*** Keedya has quit IRC02:42
*** Daisy has quit IRC02:43
*** Keedya has joined #openstack-infra02:43
*** Daisy has joined #openstack-infra02:43
*** markvoelker has joined #openstack-infra02:44
*** adreznec has quit IRC02:45
*** cbader_ has quit IRC02:45
*** camunoz has quit IRC02:46
*** hichihara has quit IRC02:46
KeedyaHi there, can I get some reviews on
*** _joes_ has quit IRC02:46
*** lennyb__ has quit IRC02:46
*** camunoz has joined #openstack-infra02:46
KeedyaHappy Sunday :)02:46
*** adreznec has joined #openstack-infra02:47
*** jokke_ has quit IRC02:47
*** jokke_ has joined #openstack-infra02:47
*** lennyb__ has joined #openstack-infra02:47
*** _joes_ has joined #openstack-infra02:48
*** hichihara has joined #openstack-infra02:48
*** ianychoi has quit IRC02:49
*** markvoelker has quit IRC02:49
*** ianychoi has joined #openstack-infra02:49
*** Keedya has quit IRC02:49
*** Keedya has joined #openstack-infra02:50
*** boris-42 has joined #openstack-infra02:50
*** Sukhdev has joined #openstack-infra02:52
*** roxanaghe has joined #openstack-infra02:52
*** salv-orlando has quit IRC02:53
*** Keedya has quit IRC02:55
*** binbincong has joined #openstack-infra02:56
*** binbincong has quit IRC03:05
*** binbincong has joined #openstack-infra03:05
*** julim has quit IRC03:07
*** vgridnev has joined #openstack-infra03:07
*** nelsnels_ has joined #openstack-infra03:09
*** nelsnelson has quit IRC03:12
*** roxanaghe has quit IRC03:14
*** sdake has quit IRC03:19
*** baoli has joined #openstack-infra03:21
*** tojuvone_ has quit IRC03:21
*** yamamoto_ has joined #openstack-infra03:23
*** amitgandhinz has joined #openstack-infra03:24
*** tojuvone has joined #openstack-infra03:28
*** Daisy has quit IRC03:33
*** Daisy has joined #openstack-infra03:34
*** Daisy has quit IRC03:34
*** Daisy has joined #openstack-infra03:34
*** thorst has joined #openstack-infra03:36
*** amitgandhinz has quit IRC03:37
*** achanda has joined #openstack-infra03:37
*** roxanaghe has joined #openstack-infra03:39
*** hayato__ has joined #openstack-infra03:39
*** hayato_ has quit IRC03:41
*** thorst has quit IRC03:41
*** roxanaghe has quit IRC03:42
*** roxanaghe has joined #openstack-infra03:42
*** achanda has quit IRC03:42
*** Daisy has quit IRC03:43
*** links has joined #openstack-infra03:44
*** Daisy has joined #openstack-infra03:44
openstackgerritMerged openstack-infra/system-config: Infra-cloud-west is currently offline
*** sdake has joined #openstack-infra03:47
*** Daisy has quit IRC03:48
openstackgerritIan Wienand proposed openstack-infra/system-config: Reinstall requests on Fedora builds
*** baoli has quit IRC03:49
*** achanda has joined #openstack-infra03:49
*** Daisy has joined #openstack-infra03:52
*** stevemar has joined #openstack-infra03:53
*** Daisy has quit IRC03:54
*** Daisy has joined #openstack-infra03:54
*** achanda has quit IRC03:55
*** binbincong has quit IRC03:56
*** salv-orlando has joined #openstack-infra03:57
*** binbincong has joined #openstack-infra03:58
*** Daisy has quit IRC03:59
*** Daisy has joined #openstack-infra04:01
*** links has quit IRC04:01
*** zeih has joined #openstack-infra04:03
*** salv-orlando has quit IRC04:03
*** camunoz has quit IRC04:03
*** Sukhdev has quit IRC04:04
*** hayato_ has joined #openstack-infra04:05
*** Daisy has quit IRC04:06
openstackgerritgreghaynes proposed openstack/diskimage-builder: Fix PATH when being run from unactivated venv
*** zeih has quit IRC04:07
*** dimtruck is now known as zz_dimtruck04:07
*** hayato__ has quit IRC04:08
*** hayato__ has joined #openstack-infra04:09
*** hayato___ has joined #openstack-infra04:10
*** hayato_ has quit IRC04:10
*** hayato_ has joined #openstack-infra04:11
*** hayato__ has quit IRC04:11
*** hayato__ has joined #openstack-infra04:11
*** david-lyle has joined #openstack-infra04:13
*** hayato___ has quit IRC04:15
*** hayato_ has quit IRC04:15
*** camunoz has joined #openstack-infra04:16
*** claudiub has quit IRC04:17
*** links has joined #openstack-infra04:18
openstackgerritJoshua Hesketh proposed openstack-infra/shade: Catch failures with particular clouds
*** salv-orlando has joined #openstack-infra04:32
*** amitgandhinz has joined #openstack-infra04:34
*** Qiming_ has joined #openstack-infra04:35
*** Qiming has quit IRC04:35
*** david-lyle has quit IRC04:36
*** salv-orlando has quit IRC04:38
*** vgridnev has quit IRC04:42
*** markvoelker has joined #openstack-infra04:45
*** NTolerance has joined #openstack-infra04:46
openstackgerritIan Wienand proposed openstack-infra/system-config: Reinstall requests on Fedora builds
*** amitgandhinz has quit IRC04:47
openstackgerritgreghaynes proposed openstack/diskimage-builder: Prioritize venv python on hose
*** nt has quit IRC04:49
*** NTolerance is now known as nt04:49
*** markvoelker has quit IRC04:50
openstackgerritgreghaynes proposed openstack/diskimage-builder: Prioritize venv python on host
*** david-lyle has joined #openstack-infra04:50
*** thorst has joined #openstack-infra04:54
*** achanda has joined #openstack-infra04:55
*** david-lyle has quit IRC04:56
*** achanda has quit IRC05:01
openstackgerritgreghaynes proposed openstack/diskimage-builder: Make debootstrap cache opt-in
*** thorst has quit IRC05:03
openstackgerritYang Hongyang proposed openstack-infra/project-config: Add non-voting python34 job to magnum project
openstackgerritYang Hongyang proposed openstack-infra/project-config: Add voting python34 jobs for python-magnumclient
*** Daisy_ has joined #openstack-infra05:08
*** fabo_ is now known as fabo05:16
*** yuanying has quit IRC05:24
*** yuanying_ has joined #openstack-infra05:24
*** asselin__ has quit IRC05:25
*** exploreshaifali has joined #openstack-infra05:27
*** exploreshaifali has quit IRC05:34
*** [1]Thelo has joined #openstack-infra05:36
*** salv-orlando has joined #openstack-infra05:36
*** Thelo has quit IRC05:39
*** [1]Thelo is now known as Thelo05:39
*** maishsk has quit IRC05:39
*** sridhar_ram has joined #openstack-infra05:42
*** sridhar_ram has quit IRC05:42
*** watanabe_isao has joined #openstack-infra05:42
*** amitgandhinz has joined #openstack-infra05:44
*** roxanaghe has quit IRC05:44
*** roxanaghe has joined #openstack-infra05:45
*** salv-orlando has quit IRC05:47
*** fawadkhaliq has joined #openstack-infra05:54
*** abregman has joined #openstack-infra05:54
*** amotoki has joined #openstack-infra05:56
*** gongysh has joined #openstack-infra05:57
*** amitgandhinz has quit IRC05:57
*** exploreshaifali has joined #openstack-infra05:58
*** achanda has joined #openstack-infra05:58
*** thorst has joined #openstack-infra06:00
*** baoli has joined #openstack-infra06:01
*** achanda has quit IRC06:02
*** Daisy has joined #openstack-infra06:04
*** baoli has quit IRC06:05
*** Daisy_ has quit IRC06:07
*** thorst has quit IRC06:07
*** achanda has joined #openstack-infra06:11
*** Sukhdev has joined #openstack-infra06:13
*** zhurong has joined #openstack-infra06:15
*** dguitarbite has joined #openstack-infra06:20
*** calebb has joined #openstack-infra06:20
*** dguitarbite has quit IRC06:20
*** BobBall_AWOL has joined #openstack-infra06:21
*** Daisy has quit IRC06:22
*** Daisy has joined #openstack-infra06:22
*** dguitarbite has joined #openstack-infra06:23
*** dguitarbite has quit IRC06:24
*** dguitarbite has joined #openstack-infra06:24
*** calebb has quit IRC06:25
*** BobBall_1WOL has quit IRC06:25
openstackgerritabregman proposed openstack-dev/pbr: Fail on warnings when warnerrors is True
*** Daisy_ has joined #openstack-infra06:26
*** Daisy has quit IRC06:26
*** |-paul-| has joined #openstack-infra06:26
*** BobBall_1WOL has joined #openstack-infra06:26
openstackgerritJens proposed openstack-infra/git-review: Make it possible to configure draft as default push mode
*** BobBall_AWOL has quit IRC06:26
*** _nadya_ has joined #openstack-infra06:28
*** kushal has joined #openstack-infra06:28
*** _nadya_ has quit IRC06:29
*** zhurong has quit IRC06:29
*** zhurong has joined #openstack-infra06:30
*** Daisy has joined #openstack-infra06:31
*** nikhil has quit IRC06:32
*** korzen has joined #openstack-infra06:35
*** camunoz has quit IRC06:35
*** Daisy_ has quit IRC06:35
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints
*** camunoz has joined #openstack-infra06:38
*** salv-orlando has joined #openstack-infra06:42
*** zhurong has quit IRC06:43
*** sdake has quit IRC06:43
*** rcernin has joined #openstack-infra06:45
*** ianw has quit IRC06:45
*** markvoelker has joined #openstack-infra06:46
*** rcernin has quit IRC06:49
*** ianw has joined #openstack-infra06:50
*** markvoelker has quit IRC06:50
*** amitgandhinz has joined #openstack-infra06:54
*** AJaeger has quit IRC06:58
*** fedexo has quit IRC06:59
*** sdake has joined #openstack-infra06:59
*** korzen has quit IRC07:00
*** jaosorior has joined #openstack-infra07:01
*** mrunge_ is now known as mrunge07:01
*** gildub has quit IRC07:01
*** AJaeger has joined #openstack-infra07:02
*** hayato__ has quit IRC07:04
openstackgerritabregman proposed openstack-dev/pbr: Fail on warnings when warnerrors is True
*** thorst has joined #openstack-infra07:05
*** hichihara has quit IRC07:07
*** amitgandhinz has quit IRC07:08
*** yolanda has joined #openstack-infra07:08
*** thorst has quit IRC07:12
*** _nadya_ has joined #openstack-infra07:13
*** fawadk has joined #openstack-infra07:15
*** fawadkhaliq has quit IRC07:15
*** fawadkhaliq has joined #openstack-infra07:15
*** roxanaghe has quit IRC07:17
*** sdake has quit IRC07:17
*** amotoki has quit IRC07:19
*** fawadk has quit IRC07:20
*** exploreshaifali has quit IRC07:24
*** rcernin has joined #openstack-infra07:24
*** vgridnev has joined #openstack-infra07:27
*** vgridnev has quit IRC07:27
*** scheuran has joined #openstack-infra07:30
*** amotoki has joined #openstack-infra07:31
*** nikhil has joined #openstack-infra07:32
*** rossella_s has quit IRC07:32
*** rossella_s has joined #openstack-infra07:33
*** amotoki has quit IRC07:35
*** Daisy has quit IRC07:35
*** Daisy has joined #openstack-infra07:36
*** claudiub has joined #openstack-infra07:37
AJaegerinfra-root, it looks like we're not building any nodes anymore ;(07:38
openstackgerritBoris Pavlovic proposed openstack-infra/project-config: Add experimental pylint job to Rally
AJaegerinfra-root, please check
yolandahi AJaeger, i'm back but i need to step out for some errands, for a pair of hours07:39
yolandai hope that there is any other infra-root that can take care, if not i'll look when i come back07:39
*** gongysh has quit IRC07:39
AJaegeryolanda: thanks! Let's see whether jhesketh is still awake...07:40
*** Daisy has quit IRC07:41
AJaegerLooking at timing, I have a theory...07:41
openstackgerritBoris Pavlovic proposed openstack-infra/project-config: gate-rally-install-devstack-centos7 job is ready for check & gate queue
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Remove infracloud-west
AJaegerjhesketh, infra-root, please check ^07:43
*** pcaruana has joined #openstack-infra07:43
*** fawadkhaliq has quit IRC07:43
*** sripriya has quit IRC07:46
AJaegerjhesketh, infra-root: 285917 will not get any node due to these problems, you need to ninja merge it. But first please check the nodepool logs whether this is really the problem.07:47
jheskethAJaeger: looking07:47
AJaegerjhesketh: If I look at grafana, it looks like after your change "Infra-cloud-west is currently offline" merged no new nodes where build...07:48
AJaegerthanks, jhesketh07:48
jheskethAJaeger: hmm probably broke nodepool :-s07:48
*** kozhukal` has joined #openstack-infra07:48
jheskethas it probably expects the credentials07:49
*** amotoki has joined #openstack-infra07:49
jheskethAJaeger: nice pick up07:49
*** amotoki has quit IRC07:49
*** gus has quit IRC07:50
*** Daisy has joined #openstack-infra07:52
*** gus has joined #openstack-infra07:52
openstackgerritMerged openstack-infra/project-config: Remove infracloud-west
*** gongysh has joined #openstack-infra07:55
AJaegerjhesketh: do you want to apply the change manually to nodepool - or are you confident that puppet/ansible will pick it up?07:55
jheskethI'm confident it'll pick it up... the run starts in 5min07:55
jheskethI'm contemplating running it or applying it manually though07:55
jheskethI'm not sure if nodepool needs a restart to reload a cloud...07:55
* jhesketh checks07:56
abregmanhi. can anyone help with reviewing this: ?07:57
*** maishsk has joined #openstack-infra07:57
*** kozhukal` has quit IRC07:57
*** kozhukal` has joined #openstack-infra07:58
AJaegerabregman: please explain a bit what your change is so that the right people can look at it.07:58
*** kozhukal` is now known as kozhukalov`07:58
*** kozhukalov` has quit IRC07:59
abregmansure. at the moment the docs gate for every project in OpenStack is not really working since pbr skip warnings, although warnerror is set in setup.cfg07:59
AJaegerjhesketh: looking at grafana, I see that some jenkins are out of nodes, others are not - and jenkins03 is deleting 50+ nodes for quite some time. Do we have some node leakage on them?07:59
*** maishsk has quit IRC07:59
abregmanso this patch is fixing it by setting it to True with warnerrors = True07:59
*** max_lobur has joined #openstack-infra07:59
AJaegerabregman: So, that's a pbr change? Please say so - and talk to pbr cores.07:59
abregmanAJaeger: hmm who are the pbr cores?08:00
AJaegerabregman: check it yourself - go to review.o.o, projects -> List , search for pbr and then look at at the access08:01
abregmanAJaeger: ack. thanks08:01
*** maishsk has joined #openstack-infra08:01
*** Daisy has quit IRC08:02
*** pasquier-s_ has quit IRC08:03
*** mattymo has quit IRC08:03
*** dkaigarodsev has quit IRC08:04
*** amitgandhinz has joined #openstack-infra08:04
jhesketh(looks like nodepool reloads configs itself... ansible is running currently so hopefully the new change will be adopted shortly)08:04
* jhesketh watches it08:04
*** dkaigarodsev has joined #openstack-infra08:05
*** pasquier-s has joined #openstack-infra08:05
*** Daisy has joined #openstack-infra08:05
jheskethAJaeger: not sure what's going on with jenkins03 there... am poking08:05
*** mattymo has joined #openstack-infra08:05
*** ihrachys has joined #openstack-infra08:06
max_loburHi Everyone! Can someone from project-config review please (adding the new proj, already +2)08:06
*** jtomasek has joined #openstack-infra08:07
*** wwwbukolaycom has quit IRC08:09
*** thorst has joined #openstack-infra08:11
*** zeih_ has joined #openstack-infra08:11
*** zeih_ has quit IRC08:12
*** zeih_ has joined #openstack-infra08:12
*** kushal has quit IRC08:13
*** bgaifullin has joined #openstack-infra08:15
*** hayato_ has joined #openstack-infra08:15
*** bgaifullin has quit IRC08:15
*** [HeOS] has joined #openstack-infra08:15
*** hayato_ has quit IRC08:15
*** zhurong has joined #openstack-infra08:16
*** hayato_ has joined #openstack-infra08:16
jheskethAJaeger: okay new nodes building08:16
jheskethAJaeger: I think nodepool was also stuck from cleaning up nodes, so I'll see if the nodes fix themselves from jenkins0308:17
*** amitgandhinz has quit IRC08:17
*** HeOS has quit IRC08:17
*** thorst has quit IRC08:17
*** exploreshaifali has joined #openstack-infra08:18
*** jed56 has joined #openstack-infra08:20
*** arxcruz has joined #openstack-infra08:21
*** vincentll has joined #openstack-infra08:22
*** jlanoux has joined #openstack-infra08:23
*** hashar has joined #openstack-infra08:24
*** dizquierdo has joined #openstack-infra08:25
*** ifarkas has joined #openstack-infra08:25
*** matrohon has joined #openstack-infra08:25
*** vgridnev has joined #openstack-infra08:25
*** achanda has quit IRC08:26
*** fawadkhaliq has joined #openstack-infra08:27
AJaegerjhesketh: thanks, yeah, see them building now (had to step out)08:27
*** exploreshaifali has quit IRC08:28
*** vgridnev has quit IRC08:28
*** gongysh has quit IRC08:29
bogdandoAJaeger, could you please accept the revert of the change ?08:29
*** bgaifullin has joined #openstack-infra08:29
*** rcernin has quit IRC08:29
bogdandoit fails for unknown tox reasons :) and blocks all patches to the affected repo , like this ;(08:30
*** vgridnev has joined #openstack-infra08:30
AJaegerbogdando: you really didn't test docs building locally? ;(08:31
bogdandoif you know how to fix this, I'd be very happy though08:31
bogdandoI do, but may be I should have done it to the *clean* VM08:32
bogdandoalso I'm not sure how to trigger the exactly gate job at my env. Tox worked for me, for example08:33
AJaegerbogdando: invocation is " tox -evenv -- python build_sphinx"08:33
AJaegeryour tox.ini looks very strange, compare with other repos...08:34
bogdandoas there is no python things to test, only docs08:34
AJaegerbogdando: problem is you create:08:34
AJaegervenv create: /home/jenkins/workspace/gate-fuel-noop-fixtures-docs/.tox/2.708:35
AJaegerBut it should be venv08:35
*** amotoki has joined #openstack-infra08:35
bogdandoOk, I will put -1 WIP for a while, may be I will be lucky to fix this in couple of hours ...08:35
bogdandothank you08:36
*** amotoki has quit IRC08:36
*** achanda has joined #openstack-infra08:37
*** rcernin has joined #openstack-infra08:37
AJaegerbogdando: your tox.ini was far too sophisticated, I'm for small is beautiful ;)08:38
*** fawadkhaliq has quit IRC08:38
bogdandoAJaeger, oh, thank you! You've saved my couple of hours08:38
AJaegerbogdando: change is untested ;)08:38
bogdandoAJaeger, looks working08:41
AJaegerbogdando: then let's wait for Zuul to finish testing and merge it... Our systems are busy right now, might take an hour or two.08:43
bogdandookay, thank you08:43
AJaegerjhesketh: can you reach jenkins03? I cannot ;(08:44
*** exploreshaifali has joined #openstack-infra08:44
*** amotoki has joined #openstack-infra08:45
*** gildub has joined #openstack-infra08:45
*** d0ugal_ has quit IRC08:46
*** d0ugal has joined #openstack-infra08:46
*** gildub has quit IRC08:46
*** markvoelker has joined #openstack-infra08:47
AJaegerjhesketh: ah, now it's there - slow...08:47
*** gildub has joined #openstack-infra08:48
*** watanabe_isao has quit IRC08:49
*** markvoelker has quit IRC08:51
jheskethAJaeger: hmm the load looks normal08:53
jheskethloading the front page of jenkins for any masters is usually slow though08:53
jheskethas for why it has so many in delete state, that's unclear08:54
jheskeththey are also across many clouds08:54
*** maishsk has quit IRC08:55
openstackgerritLenny Verkhovsky proposed openstack/diskimage-builder: Get max retries for the interface from environment
jheskethAJaeger: I might do a safe restart on it...08:55
*** fhubik has joined #openstack-infra08:55
*** nmagnezi has joined #openstack-infra08:56
*** dingyichen has quit IRC08:56
*** notnownikki has joined #openstack-infra08:57
*** lezbar has joined #openstack-infra08:58
*** maishsk has joined #openstack-infra08:58
*** maishsk has quit IRC08:58
*** cody-somerville has quit IRC08:59
*** jordanP has joined #openstack-infra09:02
*** shardy has joined #openstack-infra09:03
*** maishsk has joined #openstack-infra09:04
openstackgerritMarton Kiss proposed openstack-infra/groups: Add a standalone map page
*** esikachev has joined #openstack-infra09:05
*** mrmartin has joined #openstack-infra09:06
*** achanda has quit IRC09:08
openstackgerritEvgeny Sikachev proposed openstack-infra/project-config: Add pylint, coverage, py34 to sahara-tests
*** acabot has joined #openstack-infra09:09
*** _degorenko|afk is now known as degorenko09:10
*** amotoki has quit IRC09:10
xiangxinyonghi anteaya09:11
*** achanda has joined #openstack-infra09:12
*** amotoki has joined #openstack-infra09:13
*** Sukhdev has quit IRC09:13
*** amitgandhinz has joined #openstack-infra09:14
*** thorst has joined #openstack-infra09:15
*** dizquierdo has quit IRC09:15
*** Daisy has quit IRC09:15
*** nmagnezi has quit IRC09:15
*** Daisy has joined #openstack-infra09:15
*** asettle has joined #openstack-infra09:19
*** jistr has joined #openstack-infra09:20
*** Daisy has quit IRC09:20
*** Hal has joined #openstack-infra09:21
*** Hal is now known as Guest571409:21
*** derekh has joined #openstack-infra09:21
*** korzen has joined #openstack-infra09:22
*** thorst has quit IRC09:22
*** amotoki has quit IRC09:24
*** patrickeast has quit IRC09:24
*** patrickeast has joined #openstack-infra09:25
*** zeih_ has quit IRC09:26
*** amitgandhinz has quit IRC09:27
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Disable Liberty translations for Server projects
*** nmagnezi has joined #openstack-infra09:28
*** sorantis has joined #openstack-infra09:31
*** zeih_ has joined #openstack-infra09:31
*** kzaitsev_mb has joined #openstack-infra09:32
*** zhurong has quit IRC09:34
*** amotoki has joined #openstack-infra09:34
*** BobBall_1WOL is now known as BobBall09:35
*** e0ne has joined #openstack-infra09:43
*** sheeprine has quit IRC09:47
yolandaAJaeger, i'm back. How are things ?09:48
AJaegeryolanda: jhesketh and I fixed nodepool to produce nodes again09:48
AJaegerit's slowly recovering.09:49
yolandaAJaeger, what was the problem?09:49
*** kushal has joined #openstack-infra09:49
*** sheeprine has joined #openstack-infra09:49
AJaeger285917 fixed it - jhesketh removed infracloud-west in system-config but it was still in project-config09:50
AJaegerlooking at I'm not happy with the large number of deletes for jenkins0309:50
yolandaoh, i thought there was a change from crinkle from saturday, to remove infracloud-west09:50
AJaegerjhesketh was considering a safe restart, not sure whether he did it09:50
jheskethyeah I actually knew it would be in nodepool, I just didn't think it would cause the scheduler to stop working... I might try and make nodepool more resilient to that09:50
jheskethAJaeger: yeah in progress09:51
jheskethit needs to finish the building jobs09:51
AJaegerjhesketh: indeed...09:51
jheskethyolanda: quite likely, but it never merged and I didn't look for it when debugging this morning09:51
*** sdake has joined #openstack-infra09:51
*** Daisy_ has joined #openstack-infra09:52
*** achanda has quit IRC09:53
*** boris-42 has quit IRC09:54
*** ihrachys has quit IRC09:54
AJaegeryolanda, jhesketh: I would have expected us to use all nodes by now but looking at zuul graph, there's still room left, so something is still odd (might be jenkins03)09:54
*** e0ne has quit IRC09:55
yolandaAJaeger, i'm trying to enter into jenkins03 but looks super slow09:56
jheskethyolanda: I have it doing a graceful restart fyi09:57
yolandajhesketh, so that change landed on the weekend
yolandait sees max-servers to -1 for weset09:57
*** Daisy_ has quit IRC09:57
yolandaisn't enough?09:57
jheskethyolanda: right, so the problem was that clouds.yaml still had infra-west in it09:58
jheskethwhich made ansible fail09:58
jheskethso no infra changes were being applied09:58
jheskethso I removed infra-west from clouds.yaml, but nodepool also uses that09:58
yolandaoh... i didn't think that an extra entry on clouds.yaml could make it fail, if nodepool didn't use it09:58
jheskethand nodepool does not gracefully handle not having the right configuration or a missing cloud09:58
jheskethyolanda: well it wasn't nodepool, it was ansible doing the puppet runs.. nodepool failed due to my ansible fix09:58
*** salv-orl_ has joined #openstack-infra09:59
*** ihrachys has joined #openstack-infra09:59
yolandaah i see. This should be more resilient to failures09:59
*** thorst has joined #openstack-infra09:59
yolandajhesketh, i started some work on making that ansible puppet more resilient:
yolandai can take a look at this ansible-puppet issue10:01
jheskethyolanda: already have10:02
*** salv-orlando has quit IRC10:02
yolandaah cool10:02
lucasagomeshi folks, if you have a time mind taking a look at this devstack-gate patch? this is a fairly small (3LOC) patch that will allow Ironic to enable drivers such as pxe_ipmitool to run in the gate. Thank you10:03
*** thorst has quit IRC10:06
*** sdake has quit IRC10:06
*** wznoinsk__ is now known as wznoinsk10:07
*** rguillebert has joined #openstack-infra10:07
yolandaAJaeger, i'm looking at What's the reasoning for start using ubuntu-trusty vs bare-trusty ?10:12
*** craige has quit IRC10:14
AJaegeryolanda: this is part of fungi's move from bare-trusty to ubuntu-trusty. Marked it as WIP since I want to coordinate appropriate time with him.10:14
AJaegeryolanda: bare-trusty only lives in RAX clouds, we want to move away from those images and use only ubuntu-trusty10:14
yolandaAJaeger, so the final goal is to increase the nodes available for that tests?10:15
AJaegerand have ubuntu-trusty in all clouds available10:15
*** gildub has quit IRC10:15
AJaegerincrease nodes available - and remove single point of failure10:15
AJaegerEnd goal is consolidating the number of different images10:16
*** dtantsur|afk is now known as dtantsur10:16
yolandai see, so a single ubuntu-trusty generated by dib, and used everywhere10:16
AJaegeryolanda: fungi has done a couple of bindep changes to move us in that direction - but those only run experimental jobs10:17
AJaegerI think we're ready to make the switch now but want fungi to coordinate that - therefore WIP. So, reviews welcome10:18
yolandaAJaeger how is status on those tests? have they good success rate?10:18
AJaegeryolanda: pabelanger and fungi hunted down one problem where the two nodes gave different results - and now they look fine.10:19
AJaegerBut too early to speak about success rate.10:19
AJaegerThey do the same now ;)10:19
*** sputnik13 has joined #openstack-infra10:20
*** Qiming_ has quit IRC10:20
openstackgerritMerged openstack-infra/project-config: Add dashboard and client for Smaug
*** daemontool_ has quit IRC10:23
openstackgerritMerged openstack-infra/project-config: Add broadview-ui project
openstackgerritMerged openstack-infra/project-config: Remove tempest-lib-src experimental jobs
openstackgerritMerged openstack-infra/project-config: Update zuul layout for Puppet OpenStack jobs on Trusty
*** amitgandhinz has joined #openstack-infra10:24
yolandai had a very long review queue, but trying to unlock things...10:24
AJaegeryolanda: thanks a lot!10:24
AJaegerjhesketh: got an 502 proxy error on
yolandaAJaeger, please let me know if you need reviews on something in particular, i may loose something with that week without reviews10:25
AJaegeryolanda: nothing critical right now, thanks10:26
jheskethAJaeger: I think that's a timeout thing10:26
AJaegerjhesketh: got now 33050 which according to jenkins03 is finished10:26
AJaegerjhesketh: but looking at it's not finished10:26
jheskethI think it's struggling to keep up10:27
AJaeger(that's the top change in the gate)10:27
openstackgerritMerged openstack-infra/project-config: Make tweaks to neutron-plus job
jheskethso it's probably dropping things10:27
yolandajhesketh, i added a fix to the play to run jjb on hosts, to try avoid cache corruption
openstackgerritMerged openstack-infra/project-config: Promote lbaas namespace driver to check, non-voting
AJaegerjhesketh: that keystone change failed, can you kill it somehow?10:27
AJaegerkeystone will not merge anything today - they have a leap year bug:10:28
AJaegerexpires_at = datetime.datetime.utcnow().replace(year=2031)10:28
AJaegerValueError: day is out of range for month10:28
yolandaoh, leap years! a nightmare10:28
*** daemontool has joined #openstack-infra10:29
jheskethAJaeger: I'm trying to load jenkins03 to take a look, but it's slow10:30
jheskethI'll need to log into it to kill it gracefully which probably isn't going to be fun10:30
openstackgerritMerged openstack-infra/project-config: Remove gate-openstack-ansible-commit-nv job
openstackgerritMerged openstack-infra/project-config: Add a new job to run neutron_vpnaas tempest tests
*** sputnik13 has quit IRC10:34
openstackgerritMerged openstack-infra/project-config: Remove redundant job to stop running the same tests
openstackgerritMerged openstack-infra/project-config: add reno jobs for oslo.db
openstackgerritDmitry Tantsur proposed openstack/requirements: Bump python-ironic-inspector-client to 1.5.0
*** amitgandhinz has quit IRC10:37
*** sfinucan has joined #openstack-infra10:37
*** hayato_ has quit IRC10:39
*** daemontool_ has joined #openstack-infra10:39
openstackgerritMerged openstack-infra/project-config: Retire cloudv-ostf-adapter repo
openstackgerritMerged openstack-infra/project-config: Remove magnumclient bandit job
openstackgerritMerged openstack-infra/project-config: Add senlin-dashboard to PROJECTS
*** daemontool has quit IRC10:41
*** daemontool_ is now known as daemontool10:42
*** salv-orlando has joined #openstack-infra10:42
*** salv-orlando has quit IRC10:43
*** salv-orl_ has quit IRC10:43
*** sdake has joined #openstack-infra10:43
*** salv-orlando has joined #openstack-infra10:43
openstackgerritMerged openstack-infra/storyboard: Add Due Dates to relevant WSME models
openstackgerritMerged openstack-infra/storyboard: Update `PUT /v1/worklists/:id/items/:id` to allow assigning a due date
*** markvoelker has joined #openstack-infra10:47
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Add mirror server
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Used the mirror server for the centos image and git repos
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Use instack-qcow2 from cache if available and appropriate
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Upload the instack qcow image after periodic jobs
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: [NO MERGY] Test a fake periodic job
*** markvoelker has quit IRC10:52
*** yamamoto_ has quit IRC10:53
*** sdake has quit IRC10:53
*** mrmartin has quit IRC10:53
*** andreykurilin__ has joined #openstack-infra10:54
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Archive cards instead of deleting them
*** sdague has joined #openstack-infra10:55
*** daemontool__ has joined #openstack-infra10:56
*** flaper87 has quit IRC10:56
*** flaper87 has joined #openstack-infra10:56
*** daemontool has quit IRC10:57
*** korzen_ has joined #openstack-infra10:58
*** korzen has quit IRC10:58
*** sam_wan has quit IRC10:59
*** amotoki has quit IRC11:00
*** amotoki has joined #openstack-infra11:03
*** thorst has joined #openstack-infra11:06
*** exploreshaifali has quit IRC11:06
*** sdake has joined #openstack-infra11:06
*** tpsilva has joined #openstack-infra11:09
*** craige has joined #openstack-infra11:09
*** thorst has quit IRC11:11
*** amotoki has quit IRC11:11
*** sdake has quit IRC11:16
*** Qiming has joined #openstack-infra11:16
Qimingpath submitted 7 hours ago, it is still not triggering Ci, any hints?11:18
ttxQiming: the gate is pretty loaded right now, there is a long backlog11:20
openstackgerritMerged openstack-infra/project-config: Add non-voting python34 job to magnum project
Qimingthanks ttx, will find some other things to work on, :)11:20
ttxCurrent estimates on the check queue ~ 5 hours11:21
openstackgerritMerged openstack-infra/project-config: Add voting python34 jobs for python-magnumclient
sdagueleap day11:21
ttxsdague: fun heh11:21
*** asselin__ has joined #openstack-infra11:22
*** fhubik is now known as fhubik_brb11:22
ttxThis morning I almost tweeted "make use of the extra day to get things in before th FF crunch, then I looked up zuul status and decided it wasn't the best day to do that after all11:22
sdagueno... not even that11:22
sdaguekeystone doesn't do leap days -
ttxsdague: right, I just tweeted about that one instead :)11:23
sdagueah, ok, I haven't looked at the twitters yet11:23
*** daemontool_ has joined #openstack-infra11:24
* ttx likes leap day blunders11:24
*** achanda has joined #openstack-infra11:24
ttxthat one just sounds perfectly alright (except it starts failing as you get closer to 2031 but meh)11:24
sdaguehas anyone posted that fix yet?11:24
ttxsdague: yolanda AJaeger and jhesketh were discussing it earlier11:25
ttxThey may not have11:25
jheskethI wasn't looking for a fix sorry11:25
*** asselin__ has quit IRC11:26
sdagueok, I'll post the fix11:27
ttxsdague: looks like the answer is "no"11:27
*** daemontool__ has quit IRC11:27
* ttx lunches11:28
yolandai haven't sent any fix11:28
*** achanda has quit IRC11:29
*** dims has joined #openstack-infra11:29
*** yamamoto has joined #openstack-infra11:31
*** maishsk has quit IRC11:32
*** dims has quit IRC11:32
*** fhubik_brb is now known as fhubik11:32
*** dims has joined #openstack-infra11:32
*** rossella_s has quit IRC11:32
max_lobursdague: Hi! Could you take a look please (adding a proj to project-config)11:32
*** bgaifullin has quit IRC11:32
*** rossella_s has joined #openstack-infra11:33
*** bgaifullin has joined #openstack-infra11:33
*** yamamoto_ has joined #openstack-infra11:33
*** amitgandhinz has joined #openstack-infra11:33
*** korzen has joined #openstack-infra11:34
*** korzen_ has quit IRC11:35
*** sorantis has quit IRC11:35
*** maishsk has joined #openstack-infra11:35
*** otsuka_ has joined #openstack-infra11:36
*** yamamoto has quit IRC11:36
*** Daisy has joined #openstack-infra11:36
sdagueyolanda / jhesketh did zuul just flip out an not do anythign over the weekend. the nodepool pattern + the check queue looks like what happens when work doesn't process for a long time then flushes all at once11:37
jheskethsdague: nodepool was offline due to infra-cloud credentials being unavailable11:38
jheskethI should have picked up on it earlier :-(11:38
AJaegersdague: the keystone fix is open for grabs ;)11:38
sdagueAJaeger: I pushed it11:38
sdaguejhesketh: ah11:39
sdaguejhesketh: sounds like a good enhancement :)11:39
*** otsuka has quit IRC11:39
sdagueespecially given the vast number of clouds we get under it now11:40
dmelladoHi is there any cause about the CI being *really* slow today?11:40
AJaegerdmellado: it has a long backlog, have patience, please11:40
openstackgerrityolanda.robla proposed openstack-infra/glean: Fix hostname entries in glean
dmelladoAJaeger: ack, I was thinking about if there was something broken, as it has been waiting way longer than almost ever xD11:40
AJaegersdague: nova did not publish any tarballs or docs for a week due to constraints changes, please see
dmelladoand I was seeing every gate but one still in 'queued' state11:41
dmelladobut thanks for letting me know AJaeger ;)11:41
sdaguedmellado: there was something broken over the weekend, now there is a lot flushing11:42
dmelladosdague: I see, thanks!11:42
*** ldnunes has joined #openstack-infra11:43
AJaegersdague: downtime was only this morning for 6 hours or so...11:43
*** Daisy has quit IRC11:46
sdagueAJaeger: right, but we're coming up on M3, and we're already way over capacity11:46
sdagueI'll be surprised if the check queue is less than 6 hours delay this entire week11:46
*** Daisy has joined #openstack-infra11:46
*** bgaifullin has quit IRC11:47
*** exploreshaifali has joined #openstack-infra11:47
*** amitgandhinz has quit IRC11:47
sdagueAJaeger: you have WIP this11:49
AJaegersdague: Yes, wanted to give lifeless, Nakato and you a chance to review ;)11:50
sdaguecan you unWIP it and I'll approve it11:50
*** Daisy has quit IRC11:51
sdagueI think on stuff like that keeping it mergable is good. Few nova cores are going to weigh in on a bit like that that don't understand what's going on11:51
AJaegersdague: I wanted review from outside of nova ;)11:52
* AJaeger removed -1 from both nova and glance changes now11:52
*** cody-somerville has joined #openstack-infra11:53
*** bgaifullin has joined #openstack-infra11:53
sdagueyolanda: is it possible to kill all the kolla changes in check queue. Looking at sdake's email they all will fail11:54
AJaegersdague: could you remove two more constraint related changes: and , please?11:54
sdagueand the kolla rush isn't helping the capacity11:54
yolandasdague i can be possible yes11:54
sdaguewell except his fix for it -
AJaegersdague: now I get it - the kolla change did not merge yet ;(11:55
*** marcusvrn_ has joined #openstack-infra11:55
sdagueAJaeger: the kolla change hasn't even gotten tests run on it yet11:57
sdaguebecause all the other kolla changes rushing the world11:57
*** htruta` is now known as htruta11:58
sdagueand other changes11:58
sdagueit's not just them11:58
sdaguebut we are just way over capacity11:59
yolandai killled one kolla service, the others have not even reached jenkins11:59
*** skolekonov_ has joined #openstack-infra12:00
sdagueas long as  stays in the queue to get tests run on12:01
sdaguethat's their important fix12:01
skolekonov_Hi folks. Is works ok?  I see a long chain in gate pipeline and I can't open a job on this jenkins12:01
AJaegerskolekonov_: jhesketh is shutting down jenkins03 currently12:02
AJaegerskolekonov_: we have a long backlog, please have patience12:02
skolekonov_AJaeger, ok, thanks12:03
jheskethyeah jenkins03 is clearing it's queues12:03
*** zeih_ has quit IRC12:03
openstackgerritMerged openstack-infra/project-config: Fix pylint to work with constraints
AJaegeryolanda, sdague, jhesketh : What about sending out a status notice like "Our systems have a long backlog, please have patience and avoid rechecks."12:03
jheskethAJaeger: yep, good idea12:04
jhesketh#status notice Infra currently has a long backlog. Please be patient and where possible avoid rechecks while it catches up.12:05
openstackstatusjhesketh: sending notice12:05
openstackgerritMerged openstack-infra/project-config: Remove cinder constraints jobs
*** lucasagomes is now known as lucas-hungry12:06
-openstackstatus- NOTICE: Infra currently has a long backlog. Please be patient and where possible avoid rechecks while it catches up.12:06
*** sorantis has joined #openstack-infra12:07
openstackstatusjhesketh: finished sending notice12:07
*** jpr has joined #openstack-infra12:08
*** rfolco has joined #openstack-infra12:08
*** amrith is now known as _amrith_12:11
odyssey4meAJaeger yolanda we have a time sensitive review (adding new repositories) which needs attention - could you please take a look at it when you have a moment?12:12
bgaifullinHi. Folks please review and merge if it is OK12:12
*** otsuka has joined #openstack-infra12:13
*** fhubik is now known as fhubik_brb12:14
*** bgaifullin has quit IRC12:15
*** otsuka_ has quit IRC12:15
*** bgaifullin has joined #openstack-infra12:16
*** bgaifullin has quit IRC12:18
*** zeih has joined #openstack-infra12:18
*** daemontool_ has quit IRC12:18
openstackgerritxiangxinyong proposed openstack-infra/project-config: Add smaug-dashboard and python-smaugclient to gerritbot
openstackgerritMatthew Thode proposed openstack/diskimage-builder: Add Gentoo to the list of supported distributions
*** daemontool has joined #openstack-infra12:20
*** khappone has quit IRC12:21
*** khappone has joined #openstack-infra12:21
*** baoli has joined #openstack-infra12:22
*** baoli has quit IRC12:22
kozhukalovAJaeger: could you please take a look at once again (Vitaly has answered Dmitry's question)?12:23
*** otsuka_ has joined #openstack-infra12:23
*** hashar is now known as hasharLunch12:25
*** achanda has joined #openstack-infra12:26
*** otsuka has quit IRC12:26
*** zeih has quit IRC12:28
*** amotoki has joined #openstack-infra12:30
*** zeih has joined #openstack-infra12:30
*** achanda has quit IRC12:30
*** bgaifullin has joined #openstack-infra12:31
*** gordc has joined #openstack-infra12:32
*** thorst has joined #openstack-infra12:32
*** daemontool has quit IRC12:32
*** jaosorior has quit IRC12:33
openstackgerritMerged openstack-infra/project-config: Create openstack/packetary-specs repository
*** jaosorior has joined #openstack-infra12:34
*** daemontool has joined #openstack-infra12:34
*** smarcet has joined #openstack-infra12:35
*** rhallisey has joined #openstack-infra12:36
smarcetfungi: mordred: clarkb: morning guys, please review when u get a chance :)12:37
*** otsuka has joined #openstack-infra12:37
*** exploreshaifali has quit IRC12:38
*** sorantis has quit IRC12:39
*** sorantis has joined #openstack-infra12:40
*** otsuka_ has quit IRC12:41
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Added new projects for the OSA role break out
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Added new projects for the OSA role break out
*** amitgandhinz has joined #openstack-infra12:43
*** amotoki has quit IRC12:45
*** sheeprine has quit IRC12:45
*** sheeprine has joined #openstack-infra12:46
*** markvoelker has joined #openstack-infra12:48
*** fhubik_brb is now known as fhubik12:52
*** markvoelker has quit IRC12:53
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Added new projects for the OSA role break out
*** pcaruana has quit IRC12:53
jamespageAJaeger, hello - we're ready to move forward on the migration of openstack charm development under
jamespageany chance we can get that moving today?12:54
*** baoli has joined #openstack-infra12:55
*** daemontool has quit IRC12:55
*** coreyob has joined #openstack-infra12:55
*** daemontool has joined #openstack-infra12:55
*** ldnunes has quit IRC12:56
*** daemontool has quit IRC12:56
*** jpr has quit IRC12:56
*** julim has joined #openstack-infra12:56
*** amitgandhinz has quit IRC12:57
*** rlandy has joined #openstack-infra12:57
jheskethyolanda: I might delete all the nodes on jenkins03 from nodepool12:58
jheskeththen restart jenkins03 forcefully12:58
*** exploreshaifali has joined #openstack-infra12:58
jheskeththe soft-restart doesn't appear to be working12:58
*** daemontool has joined #openstack-infra12:59
*** ldnunes has joined #openstack-infra12:59
*** amotoki has joined #openstack-infra13:02
*** zeih has quit IRC13:02
openstackgerritJames Page proposed openstack-infra/project-config: Add Juju Charms for OpenStack
*** andreykurilin__ has quit IRC13:03
*** mrmartin has joined #openstack-infra13:03
yolandajhesketh, need any help ?13:04
jheskethnope, done already13:04
yolandaalso, i think there is a play for jenkins restarts13:04
jheskethhad to kill the jenkins process13:04
*** alivigni has joined #openstack-infra13:05
jheskethyolanda: yes, I was using that for the soft restart13:05
jheskethwhich at least stopped it from getting more jobs while it was finishing off the ones it had... but it had completed all of that and was still not very responsive13:05
jheskethand it was hogging nodeds13:05
*** lucas-hungry is now known as lucasagomes13:06
*** dtardivel has joined #openstack-infra13:07
*** pcaruana has joined #openstack-infra13:08
*** sheeprine has quit IRC13:09
*** sheeprine has joined #openstack-infra13:10
*** ociuhandu has quit IRC13:11
*** yaume has joined #openstack-infra13:12
hasharLunchif soft restart does not work, you might want to get a thread dump to identify the lock13:13
*** hasharLunch is now known as hashar13:13
hasharon my setup I had multiple issues with the IRC notification plugins. Of course YMMV13:14
*** rfolco_ has joined #openstack-infra13:14
*** toabctl has quit IRC13:14
*** dkranz has joined #openstack-infra13:15
jheskethokay that appears to have unstuck jenkins0313:15
*** rfolco has quit IRC13:16
*** trash has quit IRC13:16
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Archive cards instead of deleting them
*** otsuka has quit IRC13:18
*** otsuka has joined #openstack-infra13:18
*** links has quit IRC13:18
*** toabctl has joined #openstack-infra13:18
*** otsuka has left #openstack-infra13:18
*** trash has joined #openstack-infra13:20
*** dprince has joined #openstack-infra13:23
*** |-paul-| has quit IRC13:23
*** fhubik is now known as fhubik_brb13:23
*** _amrith_ is now known as amrith13:23
*** edmondsw has joined #openstack-infra13:25
*** skolekonov_ has quit IRC13:26
*** kushal has quit IRC13:27
*** exploreshaifali has quit IRC13:28
*** dmellado is now known as dmellado|lunch13:28
*** dmellado|lunch is now known as dmellado13:28
*** fhubik_brb is now known as fhubik13:29
*** sheeprine has quit IRC13:30
*** sheeprine has joined #openstack-infra13:30
*** regXboi has joined #openstack-infra13:31
*** ociuhandu has joined #openstack-infra13:31
*** amrith is now known as _amrith_13:33
openstackgerritOleksandr Kyrylchuk proposed openstack-infra/project-config: Remove cloudv-ostf-adapter from Infrastructure Systems
*** annegentle has joined #openstack-infra13:35
*** jtomasek_ has joined #openstack-infra13:35
*** keedya has joined #openstack-infra13:35
*** amotoki has quit IRC13:39
*** tongli has joined #openstack-infra13:40
*** jpr has joined #openstack-infra13:40
*** fawadkhaliq has joined #openstack-infra13:43
*** aysyd has joined #openstack-infra13:45
openstackgerritAntoine Musso proposed openstack-infra/nodepool: node deletion delay is now configurable
*** markvoelker has joined #openstack-infra13:47
*** baoli has quit IRC13:47
*** sambetts has joined #openstack-infra13:48
*** daemontool has quit IRC13:48
*** baoli has joined #openstack-infra13:48
*** aysyd has quit IRC13:52
*** amotoki has joined #openstack-infra13:53
*** achanda has joined #openstack-infra13:53
*** daemontool has joined #openstack-infra13:54
*** aysyd has joined #openstack-infra13:55
*** jtomasek_ has quit IRC13:55
*** daemontool_ has joined #openstack-infra13:55
*** links has joined #openstack-infra13:59
*** daemontool has quit IRC13:59
*** bswartz has joined #openstack-infra14:01
openstackgerritVincent proposed openstack-infra/project-config: add Synergy project
max_loburHi Everyone! Can someone from project-config review please (adding the new proj, already +2)14:04
tobiash_hi, just two small questions about the zuul-merger14:05
tobiash_is it processing the merges for different projects single threaded or in parallel?14:05
tobiash_how many instances do you have on the openstack ci?14:06
*** otsuka has joined #openstack-infra14:06
tobiash_our (currently single instance) zuul-merger seems to get slow when rebasing larger patch series14:07
*** otsuka has quit IRC14:07
*** otsuka has joined #openstack-infra14:08
*** calebb has joined #openstack-infra14:08
odyssey4meAJaeger I see your email about a recheck automatically doing the check on the latest base. Is that really so? I'm under the impression that for the check queue it does a checkout, whereas in the gate queue it does a cherry-pick on top of the latest HEAD?14:11
*** mriedem has joined #openstack-infra14:11
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Collect status of all nested stacks in resource-list and event-list
*** zz_dimtruck is now known as dimtruck14:12
*** achanda has quit IRC14:13
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Use netiso in the ha job
openstackgerritBen Swartzlander proposed openstack-infra/project-config: Add experimental Manila LVM job with minimal services
AJaegerodyssey4me: AFAIK we always merge the changes14:15
AJaegerThat's why a change that would create a merge-conflict when rebased will not get tested in the check queue at all...14:15
*** zeih has joined #openstack-infra14:16
*** links has quit IRC14:16
*** boris-42 has joined #openstack-infra14:16
AJaegerodyssey4me: rebasing on master is only usefull if you have merge-conflicts. OTherwise it's not needed, it will make comparison between patchsets really hard14:17
dhellmannfungi : I have another pypi upload failure we didn't notice from a few weeks ago. Do you think you'll have time to retry it for us today?14:17
odyssey4meAJaeger yes, that's why I'm interested - this is good feedback and something that I can share with our team who is under a different impression14:17
dhellmannfungi :
*** calebb has quit IRC14:18
AJaegerodyssey4me: In the mail, the change did not merge - so it should have asked to rebase *on top of the unmerged change*.14:18
*** ajmiller has joined #openstack-infra14:18
*** Qiming has quit IRC14:19
*** Qiming has joined #openstack-infra14:19
*** tiswanso has joined #openstack-infra14:20
kozhukalovAJaeger: could you please take a look once again ?14:21
*** dizquierdo has joined #openstack-infra14:21
AJaegerkozhukalov: what is that one about, please give details14:21
kozhukalovmoving javascript code from fuel-web to a separate repo fuel-ui14:22
kozhukalovAJaeger: we have answered Dmitry's question about tests14:22
AJaegerkozhukalov: will review next time I go through the queue...14:22
kozhukalovnice, thanks a lot14:23
*** otsuka has left #openstack-infra14:23
*** denisra_ has quit IRC14:25
*** ajmiller has quit IRC14:25
*** denisra_ has joined #openstack-infra14:25
*** jcoufal has joined #openstack-infra14:26
*** andymaier has joined #openstack-infra14:26
*** vgridnev has quit IRC14:26
*** vgridnev has joined #openstack-infra14:26
*** mrmartin has quit IRC14:28
*** jsavak has joined #openstack-infra14:30
korzenHi, is there anyone able to get my patch +2? I've got +1 from armax, ihrachys and sc68cal, it is the DVR multinode grenade job for Neutron upgrade tests14:30
korzensdague ^ ?14:31
*** kevinsho_ has joined #openstack-infra14:31
*** eharney has joined #openstack-infra14:34
*** sdake has joined #openstack-infra14:35
*** asselin__ has joined #openstack-infra14:35
*** dimtruck is now known as zz_dimtruck14:37
anteayainfra-root: looks like jenkins 02 and 04 are off doing their own thing14:37
anteayawould be great to have them playing along14:37
*** pcaruana has quit IRC14:38
anteayaand bluebox14:39
anteaya's error node launch attempts graph looks fairly full:
anteayamordred: ^14:39
*** zz_dimtruck is now known as dimtruck14:39
annegentlewhat's the date for cutoff of summit atc invites?14:39
annegentlecutoff date would be simpler, heh14:39
anteayaannegentle: fungi would know that14:40
*** asselin__ has quit IRC14:40
annegentleanteaya: figures since fungi is probably running the script14:40
anteayaso far I haven't seen him online yet today, should be about soon14:40
anteayaannegentle: yup14:40
annegentleanteaya: ok, thanks14:40
anteayaI believe he has two meetings in about 20 minutes14:40
anteayaannegentle: np14:40
keedyaGood Morning all, can I get some reviews on ?14:40
*** sigmavirus24_awa is now known as sigmavirus2414:42
*** x00350071 has joined #openstack-infra14:42
max_loburand here please :) needs one more +214:43
*** mrmartin has joined #openstack-infra14:44
pabelangerAJaeger: yolanda: I talked a little to fungi last week about bindep.  I think we might be ready for some jobs. Will wait for fungi to take point, but haven't seen anything other issues14:44
yolandaanteaya, will take a look at jenkins14:44
*** xiangxinyong has quit IRC14:45
AJaegerpabelanger, fungi: I just put together a small etherpad with reviews:
openstackgerritSergey Belous proposed openstack-infra/project-config: do not merge, test job that install os-vif
*** dimtruck is now known as zz_dimtruck14:47
pabelangerAJaeger: cool, looking14:48
*** woodster_ has joined #openstack-infra14:49
AJaegerpabelanger, fungi: I'd like fungi to say which jobs to merge first (and review whether he's happy with changes as they are)14:49
*** _amrith_ is now known as amrith14:49
*** amitgandhinz has joined #openstack-infra14:49
anteayayolanda: thank you14:49
yolandawe also had problems with jenkins03 before14:50
anteayayeah I saw on the graph14:50
anteayathanks for getting jenkins 03 back in the game!14:50
*** pradk has joined #openstack-infra14:50
AJaegeranteaya: which graph are you lookin at?14:50
anteayajenkins 03 has some history there14:51
AJaegerok - that's what I did earlier as well...14:51
anteayagreat, haven't read backscroll yet14:51
pabelangerAJaeger: agreed14:51
AJaegeryou're right - 2 and 4 have a straight line, didn't notice earlier ;((((14:52
*** pcaruana has joined #openstack-infra14:52
lucasagomeshi folks, if you have a time mind taking a look at this devstack-gate patch? this is a fairly small (3LOC) patch that will allow Ironic to enable drivers such as pxe_ipmitool to run in the gate. Thank you14:53
lucasagomessdague, sorry to ping you directly, but if you have a min later on today ^14:53
AJaegerpabelanger: thanks for review14:54
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Use ubuntu-trusty for project-config
*** asselin has joined #openstack-infra14:54
*** yamahata_ has quit IRC14:55
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Use ubuntu-trusty for project-config
yolandajenkins04 is totally broken, cannot be restarted from the play14:55
AJaegerpabelanger: now fixed ^14:56
*** doug-fish has joined #openstack-infra14:56
*** thiagop has joined #openstack-infra14:56
pabelangerAJaeger: \o/14:56
AJaegeryolanda: ;(14:57
AJaegeryolanda: and 02?14:57
yolandait's restarting now14:57
*** sdake has quit IRC14:58
AJaegerpabelanger: did I list all changes on the etherpad or are you aware of more?14:59
anteayaAJaeger: oh that's okay, thanks for getting 03 back online15:00
anteayayolanda: thanks for taking a look at them and getting 02 coming back15:01
pabelangerAJaeger: no, I think we are good.15:02
*** lennyb__ is now known as lennyb15:03
yolandathey should be online now15:03
anteayathank you15:03
yolandai'm going to check the remaining nodepool nodes there, as the plays did fail due to 502 error15:03
anteayathank you15:03
*** matbu is now known as perrin15:05
*** kgiusti has joined #openstack-infra15:05
*** perrin is now known as traffyy15:05
*** rbrndt_ has joined #openstack-infra15:05
*** traffyy is now known as matbu15:05
*** vgridnev has quit IRC15:05
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Install dstat after we set the fedora mirror
*** jpr has quit IRC15:08
*** kushal has joined #openstack-infra15:11
*** asselin__ has joined #openstack-infra15:12
*** annegentle has quit IRC15:12
*** sorantis has quit IRC15:13
*** vgridnev has joined #openstack-infra15:13
*** asselin has quit IRC15:15
*** ajmiller has joined #openstack-infra15:16
*** vgridnev has quit IRC15:16
*** mtanino has joined #openstack-infra15:16
*** asselin has joined #openstack-infra15:18
*** mestery has quit IRC15:19
*** asselin__ has quit IRC15:19
*** mestery has joined #openstack-infra15:19
*** asselin has quit IRC15:22
*** asselin has joined #openstack-infra15:23
*** vgridnev has joined #openstack-infra15:25
*** vgridnev has quit IRC15:26
anteayayolanda: can you take another look at jenkins 02? currently it has over 80 ready nodes but none in use15:26
*** fabio_ has joined #openstack-infra15:27
openstackgerritBogdan Dobrelya proposed openstack-infra/project-config: Adjust acls for fuel-noop-fixtures
*** MarkAtwood has joined #openstack-infra15:28
*** mrmartin has quit IRC15:28
*** vgridnev has joined #openstack-infra15:28
*** kingia has quit IRC15:28
BobBallLooking at the stable/liberty test run at can someone explain why devstack logs several repos as using master (e.g. "2016-02-26 09:43:17.524 | ++ GITBRANCH["ceilometermiddleware"]=master") rather than using stable/liberty?15:28
*** kingia has joined #openstack-infra15:28
*** mriedem is now known as mriedem_meeting15:30
*** zz_dimtruck is now known as dimtruck15:30
*** sripriya has joined #openstack-infra15:31
anteayaBobBall: if I look at that I see all the services using stable/liberty and all the clients and libs using master15:32
openstackgerritFrancesco Longo proposed openstack-infra/project-config: Added IoTronic project.
anteayaI believe the stable jobs use the stable/foo branch of the services and the master branch of clients and libs by design15:33
anteayamriedem_meeting: can you confirm, after you have finished your meeting?15:33
*** amrith is now known as _amrith_15:33
*** esikachev has quit IRC15:33
*** mizar has joined #openstack-infra15:35
*** annegentle has joined #openstack-infra15:35
*** mdenny has joined #openstack-infra15:35
*** mizar is now known as flongo15:35
*** jaosorior is now known as jaosorior_away15:37
*** [1]Thelo has joined #openstack-infra15:37
*** pblaho_ is now known as pblaho15:38
sripriyainfra-cores: could you please help review a minor update for tacker project, Thank you15:38
*** Thelo has quit IRC15:40
*** [1]Thelo is now known as Thelo15:40
*** annegentle has quit IRC15:42
*** annegentle has joined #openstack-infra15:42
*** _amrith_ is now known as amrith15:44
*** kushal has quit IRC15:45
*** kdas_ has joined #openstack-infra15:45
*** Sukhdev has joined #openstack-infra15:46
odyssey4mepabelanger do you have a minute to discuss how works? (can you pop into #openstack-ansible to chat) ?15:46
pabelangerodyssey4me: sure, let me grab a coffee15:46
*** vgridnev has quit IRC15:47
*** vgridnev has joined #openstack-infra15:48
*** jlanoux has quit IRC15:48
*** sorantis has joined #openstack-infra15:48
bogdandoAJaeger, your tox.ini fix's working like a charm. Thank you! Could you please also take a look this one ?15:49
*** denisra_ is now known as denisra15:49
*** armax has joined #openstack-infra15:50
*** mriedem_meeting is now known as mriedem15:51
*** julim has quit IRC15:51
*** Qiming has quit IRC15:52
mriedemanteaya: the dsvm jobs use pypi released clients/libs15:53
anteayayolanda: any report on what jenkins 02 is doing?15:53
*** julim has joined #openstack-infra15:53
mriedemanteaya: unless they are the -src- jobs for the client/lib repo itself15:53
*** wolsen has joined #openstack-infra15:53
anteayamriedem: thank you15:53
yolandai'm removing stale nodes there, is that behaving better?15:53
anteayaBobBall: ^^15:53
*** dtantsur is now known as dtantsur|brb15:53
anteayayolanda: well grafana says 02 has 95 ready nodes and 0 running nodes:
yolandai see jobs running on jenkins0215:54
anteayaso it is sucking up the nodes but not putting them back into play15:54
*** amrith is now known as _amrith_15:54
anteayayolanda: perhaps grafana is having a derp15:54
*** _amrith_ is now known as amrith15:54
yolanda , there are plenty of jobs running15:55
*** dkehn has quit IRC15:55
anteayayolanda: awesome, I'd go by that link then and say the grafana graph for jenkins 02 used nodes is not behaving as I would expect15:56
anteayapabelanger: any idea why grafana has jenkins 02 used nodes at 0 and shows jobs running?15:56
*** zeih has quit IRC15:56
*** vgridnev has quit IRC15:57
*** angdraug has joined #openstack-infra15:57
*** vgridnev has joined #openstack-infra15:57
*** zeih has joined #openstack-infra15:57
*** kdas_ has quit IRC15:58
jeblairanteaya, yolanda: nodepool reports that data to grafana, but no one has checked nodepool.  perhaps someone should do that before we decide the graph is in error?15:58
*** vgridnev has quit IRC15:58
anteayajeblair: ah good point15:58
*** roxanaghe has joined #openstack-infra15:58
*** maishsk has quit IRC15:58
mriedemclarkb: fungi: can you tell if the e-r daemon is processing results?15:58
jeblairanteaya: based on what i'm seeing, i suspect that jenkins02 is not reporting start or finish events to nodepool via zeromq15:58
anteayaah, that would make sense15:59
anteayathank you15:59
anteayajeblair: and good morning15:59
*** salv-orl_ has joined #openstack-infra16:00
jeblairthat will cause a significant waste of resources.  jenkins02 should be removed or restarted again.16:00
anteayaI didn't know that16:00
anteayathank you16:00
jeblairanteaya: all those ready nodes are really used nodes which should be re-used, but won't be for 8 hours.16:00
yolandai did a restart like an hour ago. The play crashed with 502 and left lots of stale nodes, but i cleaned16:01
*** sc68cal has joined #openstack-infra16:01
jeblairyolanda: apparently zmq did not restart correctly16:01
anteayajeblair: oh16:01
yolandashall i restart again?16:01
*** dkehn has joined #openstack-infra16:02
yolandajeblair, restarting again16:02
*** arxcruz has quit IRC16:03
*** salv-orlando has quit IRC16:03
yolandajeblair also, what's better way to operate when the restart is not clean? what's best way to clean the stale nodes?16:03
*** jsavak has quit IRC16:03
*** jsavak has joined #openstack-infra16:04
*** nmagnezi has quit IRC16:05
pabelangeranteaya: sorry, no16:05
fungiyolanda: take the jenkins service on the master down completely (no java processes remaining in the process table), then mark all its nodes for deletion in nodepool and delete all the slave entries from its master config.xml on disk, then start jenkins back up again16:05
yolandafungi thanks16:06
yolandathat should be documented...16:06
fungithat way there are no unaccounted for ready/used nodes in nodepool and no risk that used nodes might get reused when jenkins puts them back online when it starts up16:06
anteayapabelanger: thanks it looks like issue is that jenkins 02 is not reporting correctly to nodepool via zeromq, yolanda is working on fixing the issue16:07
*** asselin__ has joined #openstack-infra16:07
anteayapabelanger: well understanding first and then fixing16:07
yolandai ran the play to restart jenkins02 again16:07
*** fabio_ has quit IRC16:08
*** pedroalvarez_ is now known as pedroalvarez16:08
*** asselin has quit IRC16:10
*** kushal has joined #openstack-infra16:10
openstackgerritSergey Belous proposed openstack-infra/project-config: Initial job for Nova with os-vif
openstackgerritKevin Carter proposed openstack-infra/project-config: Added new projects for the OSA role break out
*** abregman has quit IRC16:13
*** zeih has quit IRC16:13
openstackgerritSergey Belous proposed openstack-infra/project-config: Initial job for Nova with os-vif
openstackgerritThomas Herve proposed openstack-infra/devstack-gate: Remove double timestamp from console logs
*** jlanoux has joined #openstack-infra16:16
*** rcernin has quit IRC16:16
*** esikachev has joined #openstack-infra16:17
*** fawadkhaliq has quit IRC16:17
*** pcaruana has quit IRC16:17
*** stevelle_ is now known as stevelle16:18
clarkbodyssey4me: we always merge with the appropriate target ref before trsting. in check that is head of the branch proposed against and in gate that is merged atop the theoretical future state that is being tested16:18
pabelangeranteaya: sounds good16:18
odyssey4meclarkb really good to know16:19
yolandafungi, jeblair, i have to leave now, but left jenkins02 restarting. maybe is better if i stop that and some of you force the stop and clean up the nodes?16:19
*** jpr has joined #openstack-infra16:20
fungiyolanda: i'm going to make a pass through all the jenkins masters shortly and see what needs cleaning up16:22
fungithanks for working on that16:22
yolandafungi ok16:23
yolandai will leave for a pair of hours16:23
anteayayolanda: thank you16:23
anteayafungi: thank you too16:23
yolandaanteaya, fungi, anytime. Bye!16:23
fungiand yeah, was in morning meetings which ran long, so mostly caught up on scrollback but that's why i was basically silent16:24
anteayafungi: yup, figured as much16:24
anteayahappy meetings16:24
fungiworking on fixing the ironic-inspector-client 1.4.0 release16:24
anteayacacti says gerrit and zuul look okay16:24
hrubiHi, zuul combines change queues of project when they share the same job. I have a common job which I want to run for several projects, but it is not of an integration nature, thus I don't want it to trigger the shared change queue behaviour. Would it make sense to have an option for turning this behaviour off for certain job?16:25
openstackgerritOleg Gelbukh proposed openstack-infra/project-config: Add project 'fuel-cfgdb'
*** sigmavirus24 is now known as sigmavirus24_awa16:25
anteayaand the other jenkinsii appear to be jenkining16:25
anteayaand the gate is moving16:25
*** sigmavirus24_awa is now known as sigmavirus2416:25
fungiand to the earlier comments that the ci system was "broken" or "down" all weekend, not true. it was humming along and completely caught up when i was looking at it yesterday afternoon. the nodepool config issue didn't merge and starve us for resources until utc monday16:26
fungithanks jhesketh and yolanda for working through that16:26
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for ceph pure virtual method called bug 1551305
openstackbug 1551305 in Cinder "backup service crashes in ceph job with "pure virtual method called"" [Medium,Confirmed]
*** abitha has joined #openstack-infra16:27
clarkbhrubi: you can just use a different job name16:27
fungihrubi: we do that fairly trivially with jenkins-job-builder by making the job a job-template which takes a name suffix parameter, and then using different suffixes when instantiating it for different projects16:28
*** lxsli has joined #openstack-infra16:28
fungiyou could even just use the {name} parameter to differentiate it completely on project name16:28
fungiwe do that with the majority of our non-integration-test jobs in fact (unit tests, static analysis, documentation builds, et cetera)16:29
*** esikachev has quit IRC16:29
*** lxsli has quit IRC16:29
*** tiswanso has quit IRC16:30
*** lxsli has joined #openstack-infra16:30
openstackgerritMateusz Matuszkowiak proposed openstack-infra/project-config: Added new repository for fuel-plugin-murano
*** kushal has quit IRC16:31
*** ashleighfarnham has joined #openstack-infra16:31
*** korzen_ has joined #openstack-infra16:31
*** tiswanso has joined #openstack-infra16:31
*** fhubik is now known as fhubik_brb16:31
*** austin81 has joined #openstack-infra16:32
*** korzen has quit IRC16:32
*** jtomasek_ has joined #openstack-infra16:33
*** kushal has joined #openstack-infra16:33
*** lxsli has quit IRC16:33
hrubiclarkb, fungi: thanks. I thought this was overkill for my use case when having ~40 projects and wanted to add one common job to all of them. I would now have 40 new jobs for the same exact thing. would a patch with such option I mentioned be voted down? :)16:33
*** jtomasek has quit IRC16:34
*** sputnik13 has joined #openstack-infra16:34
*** kushal has quit IRC16:34
*** lxsli has joined #openstack-infra16:34
*** kushal has joined #openstack-infra16:34
*** electrofelix has joined #openstack-infra16:36
clarkbhrubi considering that zuulv3 will address this and this is how we deal with it today I doubt zuulv2 would want a patch like that16:36
jklareAJaeger hey, still around?16:37
clarkb(better to fix in zuulv3 which is part of the spec than cobble a workaround in zuulv2 when a workaround already exists)16:37
hrubiclarkb: alright, thanks. I'll go with separate job names then16:37
*** markus_z has joined #openstack-infra16:37
*** sbalukoff has quit IRC16:38
*** sputnik13 has quit IRC16:38
*** keedya has quit IRC16:39
prometheanfiresomeone mind +w this (dib) has two +2 already
*** keedya has joined #openstack-infra16:39
jklaredoes anybody know a good way to run a jenkins job for multiple projects always on centos AND ubuntu without duplicating a lot of code16:39
fungijklare: sure, job-templates16:40
jklarefungi sure, but i think the {node} is usually handed over in the project16:41
jklarefungi or can i use an array here?16:41
fungijklare: you can use an array. see for an example16:42
fungijklare: the gate-bindep-fallback-{node} template instantiation there16:42
stevemarShrews: mordred: btw, we reverted the silly truncated thing for ksc and released a new version:
fungii need to clean that up too. someone added an unnecessary {suffix} parameter16:43
*** calebb has joined #openstack-infra16:43
jklarefungi and the job will be run on all of them always or just on one of them by random16:43
*** thorst is now known as thorst_afk16:44
fungijklare: that's just defining the jobs. to run the jobs you need to tell zuul to do that, so for example this project-template in its layout added to projects which need it
jklarefungi ahhh i see, thanks16:45
*** kushal has quit IRC16:45
max_loburanteaya: sdague are you around? do you have a sec to review (needs another +2)16:45
*** sorantis has quit IRC16:45
fungijklare: in that example we're voting/gating on devstack-centos7 and ubuntu-trusty nodes, and running the fedora-23 version of the job only in experimental16:45
*** asilenkov_ has joined #openstack-infra16:46
*** katyafervent_awa has quit IRC16:46
*** asilenkov has quit IRC16:46
*** pkarikh has quit IRC16:46
markus_zZara: Hi, we talked some weeks ago about my "requirements" of a bug tracker and I wrote some lines down, maybe we can discuss it shortly?16:46
*** katyafervent_awa has joined #openstack-infra16:47
*** pkarikh has joined #openstack-infra16:47
*** vgridnev has joined #openstack-infra16:48
markus_zZara: It's at the bottom of under the section "temp/misc/stuff". Not very elaborate, but maybe it helps.16:49
mriedemclarkb: check it out
jklarefungi it there something like a bare-centos7 node, like the bare-trusty one?16:49
mriedemthat debug log is in each test run over 65K times16:49
*** sputnik13 has joined #openstack-infra16:51
*** scheuran has quit IRC16:51
fungijklare: no, we're in the process of moving off the bare-.* nodes anyway16:51
*** sorantis has joined #openstack-infra16:51
Zaramarkus_z: ah, thanks for those, reading now16:51
jklarefungi ok, i will try the devstack-centos7 one :)16:51
*** annegentle has quit IRC16:52
fungijklare: in the near term we expect to have all jobs running on devstack-.* nodes (which are likely to also be renamed)16:52
*** thingee has quit IRC16:53
fungijklare: the "bare" and "devstack" prefixes are misleading anyway. "bare" means we don't run devstack there so preinstall a ton of packages different jobs might need. "devstack" means we're going to run jobs which, like devstack, install their own dependencies16:53
*** thingee has joined #openstack-infra16:53
*** fhubik_brb is now known as fhubik16:53
jklarefungi ah ok16:53
*** mrmartin has joined #openstack-infra16:53
fungijklare: so the devstack-.* nodes are a fairly minimal blank slate16:53
fungithat "ubuntu-trusty" node is really just "devstack-trusty" under a new name16:54
jklarefungi i guess i will have to just give it a try to figure out which packages we need for the chef-stuff we are doing16:54
*** sputnik13 has quit IRC16:54
*** yamamoto_ has quit IRC16:54
jklarefungi thanks for explaining :)16:55
*** yamamoto has joined #openstack-infra16:55
fungijklare: yep. if you want something with the equivalent of the packageset from the bare-.* workers, try adding the install-distro-packages macro like we're doing in our experimental jobs for the switch off bare-.* nodes:
*** daemontool_ has quit IRC16:55
*** dtantsur|brb is now known as dtantsur16:56
mordredstevemar: woot!16:56
fungijklare: those are equivalents of jobs which run on bare-.* but instead install packages and in some cases do database setup too16:56
*** matrohon has quit IRC16:57
*** apoorvad has joined #openstack-infra16:57
pleia2good morning16:59
bkeroGood morning17:00
*** sbalukoff has joined #openstack-infra17:00
mrmartingood morning17:00
*** vincentll has quit IRC17:00
anteayaa crowd17:01
mrmartinso wiki.o.o fix, what is the plan?17:02
*** Jeffrey4l has quit IRC17:03
*** hashar has quit IRC17:04
clarkbmrmartin: is htat all stdout?17:04
clarkbmriedem: ^17:04
fungidhellmann: python-ironic-inspector-client-1.4.0.tar.gz is up at now. did you also need the release announcement job run for it?17:04
jpmaxmanWell let's start what if anything has been done?17:04
mriedemclarkb: yeah17:04
mriedemi think a logging fixture in the cinder tests is enabling debug logging for the libs so it steps on oslo.log's default levels17:04
*** yamahata has joined #openstack-infra17:04
mriedemi'm writing a test to see17:04
jpmaxmanWhen I dropped off on Saturday people were going down lots of paths simultaneously17:05
*** exploreshaifali has joined #openstack-infra17:05
jpmaxmanDid any changes actually make it to the server?17:05
Zaramarkus_z: that is helpful, thank you. some of those things are either implemented or on the todo list, but a lot of it is new (pretty much everything involving more sophisticated commenting), so that's interesting. it's nearly the end of the working day for me, and I expect to be busy tomorrow, but Wednesday might be a good day to talk more; what (UTC) time would work for you?17:05
jpmaxmanI noticed that the spam is still coming in as of last night.17:05
anteayamrmartin jpmaxman I suggest taking a look at the logs from the weekend:
mrmartin@pleia2, jpmaxman: I suggested to disable the password auth to prevent spam account registration:
anteayaI haven't had a chance myself yet so don't know the status17:05
*** sdake has joined #openstack-infra17:06
jpmaxmanOk I suggest we do one thing at a time to identify if it is the right fix or not17:06
mrmartinanteaya: I checked the logs, and also checked the wiki database17:06
* anteaya continues to consume the split the design summit mailing list thread17:06
lucasagomeshi folks, if you have a time mind taking a look at this devstack-gate patch? this is a fairly small (3LOC) patch that will allow Ironic to enable drivers such as pxe_ipmitool to run in the gate. Thank you17:06
markus_zZara: I'm in UTC+1, Wednesday is perfect for me, very few meetings. I usually start at 9am17:06
Sam-I-Amanteaya: i'm sorry?17:06
anteayamrmartin: great well if you read the logs you know the status better than I17:06
fungipabelanger: had a solution which restricted account creation to wiki admins17:06
anteayaSam-I-Am: what did I do?17:06
*** kushal has joined #openstack-infra17:07
jpmaxmanSo did anything actually make it to the server ?17:07
Sam-I-Amanteaya: reading the summit split thread17:07
anteayaSam-I-Am: did I ping you by mistake?17:07
Sam-I-Amanteaya: no, i was just making a joke-ish17:07
anteayaSam-I-Am: yes, I am reading the summit split thread, hence have not read the backlog from the weekend17:07
openstackgerritJan Klare proposed openstack-infra/project-config: run integration testing for chef-cookbooks on centos7 and trusty
anteayaSam-I-Am: oh I see, sorry had my busy hat on and I missed it17:07
fungijpmaxman: i think solutions were still under discussion, since it's assumed we'll have to script some massive cleanup regardless of whether it goes on for another day or two at this point17:07
*** jaosorior_away is now known as jaosorior17:08
fungii'm personally fine with password auth temporarily breaking the ability of #status and #success to update wiki pages until we get the stray accounts deleted and can switch to disallowing account creation by non-admins instead of completely disallowing password authentication17:09
Sam-I-Amanteaya: tis ok17:09
Zaramarkus_z: ah, great! :) I'm also relatively free all day Weds (well, we have the storyboard meeting at 3pm UTC, but the conversation may be relevant to that anyway :)). will talk in more detail then!17:09
mrmartinpabelanger's patch seems to be ok for me.17:09
*** dizquierdo has quit IRC17:10
*** yamamoto has quit IRC17:10
fungii may not be completely caught up on the wiki discussion, but i think the path forward is probably to land mrmartin's change to disallow password auth and pabelanger's change to disallow account creation by non-admins, then delete all accounts with no openid link besides our two bot accounts, then reenable password auth17:10
zaro  morning17:11
fungisince the latest substantiated theory seems to be that these are not acutally from compromised launchpad user accounts but rather just non-openid accounts17:11
anteayaSam-I-Am: thanks17:11
mrmartinok, I need to rebase the auth patch17:11
anteayazaro: morning17:11
*** fhubik has quit IRC17:11
fungias such, it should be fairly easy to spot them and remove them from the database17:12
openstackgerritMerged openstack-infra/project-config: Create fuel-ui project
fungihowever, we likely also want to retain a list of the account ids so that we can attempt to identify any edits they performed prior to their deletion17:13
*** amotoki has quit IRC17:13
openstackgerritClark Boylan proposed openstack-infra/system-config: Add OSIC clouds.yaml details
fungiRyan_Lane: i saw you appeared over the weekend too. do you have any input on whether there's a sane way to find and unwind edits/moves performed by those accounts?17:13
jpmaxmanFungi instead of removing I think we can change permissions so they can't create or edit17:14
*** boris-42 has quit IRC17:14
jpmaxmanSo I am fine to make that change and see what happens17:14
fungisure, that's also fine17:14
fungidelete or disable17:14
*** sridhar_ram has joined #openstack-infra17:14
*** sorantis has quit IRC17:14
jpmaxmanGiven that we have captcha in place and the spam continues I am still concerned this may be an exploit17:14
*** sputnik13 has joined #openstack-infra17:15
jpmaxmanMy main point on Saturday is before we account for all the side effects of a change let's see if the change actually fixes the problem.17:15
fungii wouldn't be surprised to find that they're updating via the api, so bypassing any captcha enforcement. i don't know that would strictly qualify as an exploit17:15
jpmaxmanI still think making the captcha impossible to answer for a period of time would be a telling change too17:15
fungibut once they can't create password-auth accounts they should cease to be able to use the api too17:16
*** esikachev has joined #openstack-infra17:16
jeblairfungi: oh, when was it substantiated that they are non-openid accounts?17:16
openstackgerritMarton Kiss proposed openstack-infra/puppet-mediawiki: Disable standard password based auth
jeblairfungi: certainly when we started looking at this, they had openid urls associated with them17:16
fungijeblair: when mrmartin started looking in the db all the accounts he found had no openid associated17:17
jpmaxmanTrue. But good to know. Anyway my overarching advice is simply to quickly apply one change at a time to identify the fix more quickly then come up with a proper patch17:17
mrmartinyes, it is definietly with password auth17:17
mrmartinok so the patch got a rebase now.17:17
jpmaxmanIf we want to start with user account creation being disabled that is fine with me17:17
*** korzen_ has quit IRC17:17
*** andreykurilin__ has joined #openstack-infra17:18
*** tphummel has joined #openstack-infra17:18
*** openstackgerrit has quit IRC17:18
*** openstackgerrit has joined #openstack-infra17:18
*** vgridnev has quit IRC17:19
*** sc68cal has quit IRC17:22
*** sdake has quit IRC17:22
jeblairfungi, mrmartin: there may now be non-openid users, probably because we have enabled their creation by removing the impossible-to-answer question.  but when we started looking into this, the new accounts were definitely openid accounts.17:22
jeblairfungi, mrmartin: as an example, look at user 6498.17:22
*** sdake has joined #openstack-infra17:23
*** ayoung has quit IRC17:23
jeblairselect * from user, user_openid where user_id=6498 and uoi_user=user_id;17:23
pleia2when I looked they were also all openid accounts17:23
fungiyeah, i agree turning on the captcha by adding guessable q&a (via mechanical turk, et cetera) has probably increased spam by allowing non-openid account creation17:23
pleia2so this is new17:23
*** asettle has quit IRC17:23
Ryan_LaneI added special:nuke for that17:24
Ryan_LaneFor removing old edits17:24
Ryan_LaneIt should let you remove all edits by a user or ip I think17:24
fungiso seems like a more sane way to disable non-openid account creation than our old solution of having an unguessable answer17:25
mrmartinyeah, but 6498 don't have a valid user_password field17:25
Ryan_LaneSomeone was mentioning that launchpad has an issue with people making fake openid accounts?17:25
jeblairmrmartin: yes, that's my point.  it is an openid account from launchpad.17:25
mrmartinso, the infected accounts, have a non-empty user_password field.17:25
fungiRyan_Lane: yeah, pleia2 said the ubuntu wiki is overrun with spam in recent weeks as well17:26
jeblairmrmartin: that is only the case recently, since the account creation question was changed17:26
mrmartinthe field value for openid one's are ''17:26
Ryan_LaneYou can setup autoconfirmed17:26
*** sigmavirus24 is now known as sigmavirus24_awa17:26
*** jcoufal has quit IRC17:26
*** daemontool_ has joined #openstack-infra17:26
Ryan_LaneSo that new users can't create new articles17:26
*** jistr has quit IRC17:26
Ryan_LaneThere's also the thing I linked the other day, which will block bad ips17:26
Ryan_LaneCan use that in combination to disallow autoconfirmed users in the ip blacklist from editing at all17:27
fungimost of the spam seems to be they'll edit a page (often a "talk" page) to replace its content with spam, then redirect other high-traffic pages to it17:27
fungihrm, have we possibly done anything to disable listing of the special pages? is blank for me now even though i'm logged in17:29
*** vgridnev has joined #openstack-infra17:29
openstackgerritMerged openstack-infra/tripleo-ci: Install dstat after we set the fedora mirror
*** vgridnev has quit IRC17:29
pabelangerfungi: clarkb needs to revert the chmod 0000 to RunJobs.php17:30
*** david-lyle has joined #openstack-infra17:30
clarkbpabelanger: we decided that wasn't helpful?17:30
*** sripriya has quit IRC17:30
clarkbthats a simple chmod 644 $file17:30
jpmaxmanWhy don't we then start by making the answer unguessable. This will stop the new account creation. And we can see if it stops the spam.17:30
*** kushal has quit IRC17:30
pabelangerclarkb: not sure honestly, maybe we can revert it now.  Do that current changes and see if runjobs.php is used again17:31
jpmaxmanIf it does not we can start disabling extensions until we find what vector the spammers are using17:31
fungijpmaxman: well, it won't entirely stop the spam, because it was unguessable up until a few days ago17:31
fungijpmaxman: it was made guessable in an attempt to stem the flow of spam17:31
jpmaxmanFungi not exactly17:31
fungijpmaxman: please elaborate17:31
jpmaxmanIt was enabled for page edit and creation in an attempt to stop the spam. It was made guessable to allow humans to still edit the wiki17:32
fungijpmaxman: oh, you mean leave the q&a enforced for page edits but put it back to being unguessable17:32
fungiso basically turn off edits for the whole wiki to see if that turns off edits for the whole wiki?17:32
*** electrofelix has quit IRC17:32
jpmaxmanNot for long17:32
jpmaxmanJust maybe 30 min17:33
jpmaxmanTo see if it stops the spam17:33
*** kushal has joined #openstack-infra17:33
jpmaxmanIf it doesn't we know they are getting in via a different path17:33
*** degorenko is now known as _degorenko|afk17:33
*** jsavak has quit IRC17:33
pabelangerif that is the case, then we are in worst shape right?17:34
*** jsavak has joined #openstack-infra17:34
mrmartindon't we like to turn off puppet runs on wiki.o.o and do Settings.php modification manually until we find out what is working?17:34
mrmartinpabelanger: if that is the case, we still need to see something in access logs17:34
pleia2we've resisted doing that because having an editable wiki is important to the project, but if people feel it's valuable as a test, we can try it and just let people know17:34
*** electrofelix has joined #openstack-infra17:35
*** sputnik13 has quit IRC17:35
fungiit basically boils down to "let's test our mediawiki installation for an unknown vulnerability"17:35
*** abitha has quit IRC17:35
pabelangerI think there is 2 paths we are working on. 1st, what jpmaxman is suggestion, disable edits, if we are still getting spammed, then we have some security issue. 2nd, is try to configure wiki.o.o how we needed it too, which is taking longer.17:35
fungiwhat is the cadence of spam updates? we'd need to leave it broken long enough to be reasonably sure it's preventing spamming. how long is long enough?17:35
fungii'm not understanding where the 30-minute estimate is coming from17:36
jpmaxmanthey are coming in almost minute by minute17:36
fungithat's helpful17:36
*** sputnik13 has joined #openstack-infra17:36
*** fawadkhaliq has joined #openstack-infra17:36
jpmaxmanI just think it is somewhere to start that seems logical17:36
jpmaxmanI'm open to other suggestions17:36
Shrewsmordred: i think we're probably due for a shade release:
jpmaxmanbut we need to start narrowing this down17:37
pabelangerI am also not sure out blacklist spam is working properly, it would be nice to have an admin confirm in the logs we are rejecting some pages17:37
fungipabelanger: i can do that now17:37
pabelanger now exists17:37
pabelangerso we can start adding URLs into that17:37
pabelangerto see if page creation is affected17:38
pabelangerbut, I also think there is a download process from wikipedia but unsure how to confirm that17:38
jpmaxmanmy problem with blacklists is that even if we successfully block it still doesn't tell us how they were getting in in the first place17:38
mrmartinlet's try to prevent spam account creation first.17:39
*** sfinucan has quit IRC17:39
pabelangerwell, we have a few patch up to stop user creation. which will force people to launchpad.net17:39
*** flongo has quit IRC17:39
fungiat which point we're back to where we were last week, only compromised/fake lp accounts are able to spam17:39
*** flongo has joined #openstack-infra17:40
*** davideagnello has joined #openstack-infra17:40
fungiwhich i think eliminates their use of the api as long as we also disable all the password-based auth accounts which got added in the interim17:40
fungiand forces them to have to bypass the captcha17:40
*** abregman has joined #openstack-infra17:41
pabelangerright. We can then start the process to purge accounts17:41
pabelangerif we are considering the possibility of a security exploit, we should also bring wiki-dev.o.o online for precise see that is needed to start the migration.17:42
fungihere's an example post from the apache access log:17:42
fungi63.141.249.202 - - [28/Feb/2016:07:01:12 +0000] "POST /wiki/File:Support_Help_@@@_(((1-800-860-9230!_QuickBooks_((TechNICAL))_Support_phone_number,,_QuickBooks_PRO_Customer_support_phone_number.pdf/trackback/ HTTP/1.1" 302 705 "!_QuickBooks_((TechNICAL))_Support_phone_number,,_QuickBooks_PRO_Customer_support_phone_number.pdf"17:42
fungi"Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko"17:42
pabelangerwe need to do that anyways, but will depend on how far we can kick it down the road17:43
mrmartinbefore the purge we need to disable fake account creation.17:43
pabelangermrmartin: yes, my patch will help with that17:43
pabelangeranother though it to maybe run awstats on our http logs and see if there is a pattern17:44
pabelangeror what ever is the latest and greatest http log stats program17:44
*** ihrachys has quit IRC17:45
mrmartinok, then someone should accept pabelanger's patch, and go on17:46
mriedemi'm trying to find the thing in project-config that only runs unit tests/pep8 on unit test only changes17:46
mriedemnvm, found it17:46
pabelangermriedem: zuul/layout.yaml17:46
fungii'm looking for patterns in the access log right now, mostly for a clue to what's being used to do this17:47
*** achanda has joined #openstack-infra17:47
fungithe user agent string, which is most likely faked, is typical of ie11 on win717:47
kozhukalovguys could you please add me to the group,members? I am an author of the patch (kozhukalov Vladimir Kozhukalov
*** harlowja_at_home has joined #openstack-infra17:48
*** rbradfor_ is now known as rbradfor17:49
openstackgerritKaiyan Sheng proposed openstack-infra/project-config: Set persister to only run monasca tempest tests
*** ajmiller has quit IRC17:50
anteayafungi: are you saying a browser has opened an exploit in wiki access?17:51
fungianteaya: i am not saying anything, just looking at the details of the post calls in the apache access log17:51
pabelangerfungi: not sure I want to, but we could add some logic into apache to limit access based on User-Agent17:51
openstackgerritMatt Riedemann proposed openstack-infra/project-config: Skip dsvm jobs on cinder docs/reno/unit test changes
fungiit's more likely a non-browser spambot which is faking its user agent string to look like ie11/win717:52
anteayafungi: very good, sorry to jump to conclusions17:52
*** rkukura has joined #openstack-infra17:52
*** ajmiller has joined #openstack-infra17:52
*** annegentle has joined #openstack-infra17:53
fungithe usual tactic for attackers in cases like this is to claim to be the most common user agent possible, so that blocking by user agent is an unattractive option17:53
*** jsavak has quit IRC17:53
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Collect status of all nested stacks in resource-list and event-list
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Use netiso in the ha job
*** derekh has quit IRC17:53
pleia2and it's trivial to do17:54
*** tiswanso has quit IRC17:54
openstackgerritOleksandr Kyrylchuk proposed openstack-infra/project-config: Remove cloudv-ostf-adapter from Infrastructure Systems
fungiooh, they're doing file uploads too17:54
*** exploreshaifali has quit IRC17:54
pleia2(there's verification that what the user agent says is true)17:54
*** sdake has quit IRC17:54
pleia2er NO verification17:54
pabelangerfungi: ya, PDFs17:54
nibalizerhow can I help?17:55
*** david-lyle has quit IRC17:55
*** david-lyle_ has joined #openstack-infra17:55
*** dkranz_ has joined #openstack-infra17:56
*** kencjohnston has joined #openstack-infra17:56
funginibalizer: mostly we're trying to gather as much information about the spam situation as possible on wiki.o.o while we also try to limit the damage being caused17:56
*** dkranz has quit IRC17:56
*** tongli has quit IRC17:56
pabelangerargument could be made, PDFs on wiki.o.o is bad to start with.  Meaning we could filter our file extensions we support:$wgFileExtensions17:56
*** sdake has joined #openstack-infra17:57
*** claudiub has quit IRC17:57
fungiso there's a patch now approved to disable new non-openid account creation except for sysops. another patch soon to merge i think to temporarily disable password-based auth while we work on disabling accounts which were created via non-openid means17:57
*** jsavak has joined #openstack-infra17:57
fungiand discussing other avenues we can close off, filters we can get into place, cleanup tools at our disposal17:58
anteayais there an etherpad up for this work?17:58
anteayaor a storyboard board?17:58
anteayaor can I create one?17:58
pabelangeranteaya: I'd be open to creating an etherpad for this17:58
pleia2there is one that jpmaxman made, sec17:58
anteayagreat, would you like to or shall I?17:58
anteayapleia2: ah great17:58
*** kushal has quit IRC17:58
*** krtaylor has quit IRC17:59
*** annegentle has quit IRC17:59
anteayathank you17:59
*** dkranz_ is now known as dkranz-meeting17:59
*** dtantsur is now known as dtantsur|afk17:59
openstackgerritMerged openstack-infra/puppet-mediawiki: Prevent new user registrations except by sysops
*** claudiub has joined #openstack-infra18:01
*** jlanoux has quit IRC18:01
*** Guest67703 is now known as me_18:01
*** me_ is now known as med_18:01
*** med_ has quit IRC18:01
*** med_ has joined #openstack-infra18:01
*** baoli has quit IRC18:01
*** esikachev has quit IRC18:02
*** baoli has joined #openstack-infra18:03
*** baoli has quit IRC18:03
*** tiswanso has joined #openstack-infra18:03
*** kencjohnston has quit IRC18:04
*** achanda has quit IRC18:04
*** sridhar_ram1 has joined #openstack-infra18:05
*** lucasagomes is now known as lucas-dinner18:05
*** sridhar_ram has quit IRC18:07
*** annegentle has joined #openstack-infra18:07
openstackgerritMerged openstack-infra/tripleo-ci: Get more info from ps command
openstackgerritMerged openstack-infra/tripleo-ci: Remove unused urllib
openstackgerritPaul Belanger proposed openstack-infra/puppet-mediawiki: Disable PDF uploads by users
openstackgerritMerged openstack-infra/tripleo-ci: Run top in batch mode
*** sarob has joined #openstack-infra18:09
fungistill working on correlating the recent changes page to apache access logs18:09
*** maishsk has joined #openstack-infra18:10
*** matrohon has joined #openstack-infra18:10
*** baoli has joined #openstack-infra18:10
*** krtaylor has joined #openstack-infra18:10
fungiaha, they're in the general access.log not the mediawiki-access.log18:11
mrmartinyeah, the mediawiki-access.log is for the runjobs only18:11
fungithis is looking much more like what i was expecting at least18:11
fungiand different user agent strings on these too18:12
*** bgaifullin has quit IRC18:13
*** Swami has joined #openstack-infra18:14
*** maishsk has quit IRC18:15
*** maishsk has joined #openstack-infra18:15
fungiip addresses seem to be from hosting and internet service provider networks all over the world, pretty typical botnet pattern18:15
*** yamamoto has joined #openstack-infra18:16
dhellmannfungi : thanks for dealing with that release. I can run the announce script myself locally.18:17
*** _nadya_ has quit IRC18:18
armaxanteaya: ping18:18
anteayaarmax: hey there18:19
armaxanteaya: hi, question18:19
* anteaya listens18:19
armaxanteaya: the other day you showed me this page18:19
anteayaI did so18:19
armaxdo job pop here automatically?18:19
armaxwhat I mean is18:19
*** sc68cal has joined #openstack-infra18:19
armaxI see for instance gate-neutron-dsvm-api18:19
anteayaany that don't have an elastic recheck query filed for them, yes18:19
*** baoli_ has joined #openstack-infra18:19
armaxhow do we dismiss those that are due to infra failures18:20
anteayahow do you mean?18:20
armaxfor instance job has 318:20
armaxanteaya: hang on18:21
*** yamamoto has quit IRC18:21
armaxanteaya: that’s not the link I wanted to share18:21
armaxanteaya: this one
armaxboth unclassified failures don’t seem due to intermittent neutron issues18:22
*** baoli has quit IRC18:22
*** electrofelix has quit IRC18:22
anteayathat may be true18:22
armaxperhaps you have some guide for me to read?18:22
anteayahowever there is still a part of the sytem that is not working as expected18:22
anteayafor elastic-recheck?18:22
* anteaya gets the link18:22
fungiRyan_Lane: any guesses what this would be trying to accomplish? "GET /w/load.php?debug=false&lang=en-gb&modules=ext.echo.badge%7Cext.uls.nojs%7Cmediawiki.legacy.commonPrint%2Cshared%7Cmediawiki.ui.button&only=styles&skin=strapping&* HTTP/1.1"18:23
fungiRyan_Lane: i see it being tried by one of the ip addresses which was also spamming pages18:23
anteayaarmax: also dougwig had a good chat with mriedem on friday about elastic-recheck, and kevinbenton was there too18:23
*** thorst_afk is now known as thorst18:23
armaxanteaya: ok18:23
anteayaarmax: they might remember some helpful stuff as well18:24
anteayaarmax: thanks for asking18:24
nibalizerfungi: im gonna pull some of these longer log files locally so I can peer at them better18:24
*** Guest63721 is now known as mfisch18:24
anteayaarmax: if a job fails and it is due to an infra issue, we still need an e-r query filed about it, so we can track it18:24
*** maishsk has quit IRC18:24
armaxanteaya: ok18:24
anteayaarmax: sometimes our hosts differ or change things, but we don't know about it unless we can track it18:24
anteayaarmax: thanks :)18:25
*** mfisch is now known as Guest200418:25
armaxanteaya: that makes sense18:25
anteayagreat, thank you18:25
*** _nadya_ has joined #openstack-infra18:25
fungiRyan_Lane: nevermind, i think this is just how css is linked in the theme18:25
fungithough that does point to the attacks likely using actual browsers18:25
armaxanteaya: I am still somewhat wondering how come the neutron functional job doesn’t show up on this dashboard18:25
fungii'm seeing http post calls for action=edit, and then followup http get calls for the resulting page _and_ linked css18:26
armaxanteaya: on the unclassified dashboard I mean18:26
anteayaarmax: good question, well we noticed on friday that the elastic-recheck bot isn't behaving as mriedem expects, but we haven't had much time to dig into it with clarkb yet18:26
anteayaarmax: yes, we expected more unclassified results than we were seeing on friday18:27
mriedemarmax: does the neutron functional job run in the gate queue or only check queue?18:27
armaxanteaya: ok, I’ll look into those are reported18:27
armaxmriedem: check only18:27
mriedemand is it voting?18:27
armaxmriedem: voiting18:27
anteayaarmax: currenting infra is focused on addressing our wiki spam issue that has been going on for about 2 weeks18:27
*** krtaylor has quit IRC18:27
mriedemarmax: the uncategorized bugs page is only gate queue failures18:27
armaxmriedem: ack18:27
anteayaso after that is over was hoping to find clarkb and ask him about it18:27
anteayamriedem: ah thank you18:28
clarkbmriedem: armax anteaya all of the e-r html /json files are up to date18:28
clarkbI think only the bot is having trouble?18:28
mriedemclarkb: yeah18:28
armaxmriedem: the functional job was on the gate queue at one point, how long do these go back?18:28
mriedemarmax: 10 days18:28
*** Guest2004 is now known as mfisch18:28
armaxmriedem: that explains it18:28
*** mfisch has quit IRC18:28
*** mfisch has joined #openstack-infra18:28
armaxmriedem: thanks18:28
*** bpokorny has joined #openstack-infra18:28
clarkbmriedem: freenode has been flapping I am almost certain that the bot got lost in that18:28
openstackgerritBeth Elwell proposed openstack-infra/storyboard: Updated documentation for installing Storyboard
mriedemarmax: np, that's why you can only filter back to 10 days on the page18:29
mriedemGenerated at: 2016-02-29T18:20 (View: 24 hours, 2 days, 7 days, 10 days)18:29
*** baoli_ has quit IRC18:29
armaxmriedem: oh, I see that now18:29
funginibalizer: picking the client ip address of a particular offending spam and then filtering the log for that and analyzing the flow of http requests is interesting18:29
clarkbthough looks like it is connected now according to the PING PONG log messages18:29
* armax blind on monday morning18:29
*** boris-42 has joined #openstack-infra18:30
*** baoli has joined #openstack-infra18:31
mriedemsdague: in case you want to reduce unnecessary test node usage
*** thorst is now known as thorst_afk18:31
*** abitha has joined #openstack-infra18:31
*** doug-fish has quit IRC18:32
*** marekd has joined #openstack-infra18:32
anteayaarmax: thanks for asking about it :)18:32
*** ashleighfarnham has quit IRC18:32
armaxanteaya: thank you18:32
funginibalizer: on the one i'm looking at (), the first we ever hear from this client is a get request for one of the existing spam files in the wiki, with a referrer claimed to be
*** ashleighfarnham has joined #openstack-infra18:32
funginibalizer: it's not until a couple minutes later that it issues its first post, which is an upload of a different file18:33
anteayaarmax: welcome :)18:33
funginibalizer: er, () was supposed to be (
*** sridhar_ram1 is now known as sridhar_ram18:34
*** jamesmcarthur has joined #openstack-infra18:34
*** notnownikki has quit IRC18:35
pabelangeraccording to only admins can use createaccount now18:36
pabelangerwe need to test if accounts are still able to be created18:36
pabelangerI think I can test that with one of my bot launchpad accounts18:36
*** annegentle has quit IRC18:37
*** cloudtrainme has joined #openstack-infra18:37
mrmartinI can check the last spam account user id in the database.18:37
funginibalizer: so this is the history of the page in question:!!!_Pogo_tech_Support_phone_number,,_Pogo_techNIcal_support_phone_number._usa.pdf&action=history18:37
nibalizeris it weird that I tend to think of pogo as a real company?18:38
mrmartinok, so just for the record the last password based user account is: 7791 | KatieBatista1218:38
funginibalizer: it looks like the file was uploaded a couple days ago, with a different account at a different ip address, and this new ip address retrieved the page for it today, then updated the page description18:38
*** sripriya has joined #openstack-infra18:38
fungithis is pretty sophisticated18:38
*** krtaylor has joined #openstack-infra18:39
anteayamrmartin: is there a timestamp associated with account creation on that account?18:40
anteayafungi: so they might be updating via email notifications?18:40
mrmartinanteaya: user_touched: 2016022918033818:40
*** yaume has quit IRC18:41
anteayamrmartin: thank you18:41
fungimrmartin: anteaya: that would be just shy of 40 minutes ago then18:41
*** kevinsho_ has quit IRC18:41
fungiassuming that's a squashed iso-8601 timestamp in utc, which it looks to be18:41
*** baoli has quit IRC18:42
anteayaiso 8601 seems to stick a T in there according to:
anteayaI might be reading that wrong18:43
pabelangerclarkb: fungi: do you mind chmod 0644 runjobs.php?18:43
*** sdake has quit IRC18:43
fungipabelanger: happen to know the full path?18:43
clarkbpabelanger: I can do it18:43
*** flongo has quit IRC18:43
fungiahh, clarkb would know since it's likely in his command history18:43
*** baoli has joined #openstack-infra18:44
pabelangerso, login into wiki.o.o for the first time with a new id, can only see the following:18:44
pabelangerAll users need a nickname; you can choose one from the options below.18:44
pabelangerUsername and account choice18:44
pabelangerLog in / create account with OpenIDCancel18:44
pabelangerclicking loging doesn't do much but loop back to the page18:44
pabelangerreading up on openid extensions for suggestions18:44
*** angdraug has quit IRC18:45
clarkball done18:45
*** pvaneck has joined #openstack-infra18:45
*** Sukhdev has quit IRC18:45
*** doug-fish has joined #openstack-infra18:46
*** nmagnezi has joined #openstack-infra18:46
*** doug-fish has quit IRC18:46
*** doug-fish has joined #openstack-infra18:47
fungianteaya: the note in section 4.3.2 says the T is effectively optional18:48
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Added summit_type_id/event_type_id filter to events endpoint
*** notnownikki has joined #openstack-infra18:49
fungiin the 8601:2004(E) revision anyway18:49
*** kencjohnston has joined #openstack-infra18:49
fungior so i gather from context, since i haven't bought a copy of the standard18:50
*** e0ne has joined #openstack-infra18:51
anteayathank you18:51
*** baoli has quit IRC18:52
fungigotta love standards bodies who charge inordinate sums of money for copies of their standards (even digital copies)18:52
*** thorst_afk is now known as thorst18:52
anteayathere it is footnote 19, thanks18:52
clarkbfungi: you know the standard is good when you have to pay lots of money to read it18:53
clarkbsee also SQL18:53
*** sigmavirus24_awa is now known as sigmavirus2418:53
pabelangerSo, reading I think we need some updates to our config.  By setting createaccount = False, we removed the ablity for the user to pick there user name.  So, I think we need to set $wgOpenIDUseEmailAsNickname = True and possible $wgOpenIDProposeUsernameFromSREG = True too18:53
fungiin the past i've gotten iso standards in digital form on the cheap through ansi since they have an agreement to allow ansi to distribute at least some iso standards documents to american companies/citizens at a fraction of the iso direct cost18:53
*** baoli has joined #openstack-infra18:54
fungibut still, paying for access to standards seems counter to the idea of having a standard18:54
*** jaosorior has quit IRC18:54
*** baoli has quit IRC18:55
pabelangermrmartin: looking at I see Helen as latest user created18:55
fungipeople will often either break copyright law to code to an illegally-obtained copy of the standard, or just declare access to the standard unworkable and make up their own nonstandard implementation18:55
*** maishsk has joined #openstack-infra18:56
clarkbjeblair: new raspi 3 has integrated wireless18:56
anteayapabelanger: timestamp?18:56
pabelangeranteaya: after KatieBatista1218:56
pabelanger29 Feb 2016 @ 18:0618:56
mrmartinpabelanger yeah, but Helen is an openid account18:56
fungipabelanger: mrmartin: Helen also updated a page at 18:54:05 according to
fungiwith spam content18:57
pabelangermrmartin: Ah, good to know18:57
mrmartinthe last password auth account is still KatieBatista1218:57
fungiso new openid account, spamming the page18:57
pabelangerfungi: see my comments above about openid creation18:57
pabelangerright now, I cannot create a new users using it18:57
*** baoli has joined #openstack-infra18:58
mrmartinyou should ask the spammer's tool to create a new account.18:58
*** jaosorior has joined #openstack-infra18:58
*** _nadya_ has quit IRC18:58
fungii'm looking in the apache logs now for an indication of what urls they hit for that18:59
mrmartinbut this means that launchpad is full with fake accounts18:59
anteayaso thus far we have had zero success closing any spammer account creation vector18:59
*** andymaier has quit IRC18:59
pabelangerRight, if is full of spammers, then we need to up the game for blacklisting URLs18:59
pabelangerI'll be surprised if QuickBooks is using openstack19:00
*** maishsk has quit IRC19:00
fungii also don't think quickbooks needs to be linking to their website from our wiki even if they are using openstacjk19:01
fungier, openstacj19:01
fungii give up typing19:01
*** rockyg has joined #openstack-infra19:01
anteayait is overrated anyway19:02
pabelangerso far new users look good19:03
pabelangerbut, pretty sure we broke any user creation :)19:03
*** doug-fis_ has joined #openstack-infra19:03
clarkbif passes testing this time around it would be great to get that in19:04
clarkbthen I can work on getting osic up and running19:04
jpmaxmanjust fYI still spam coming in19:04
mrmartinwhat I missed during the registration was the user confirmation email19:04
*** asselin__ has quit IRC19:05
pabelangerso, lets land: to disble PDF uploads19:05
*** arif-ali has quit IRC19:06
*** doug-fi__ has joined #openstack-infra19:06
*** doug-fi__ has quit IRC19:06
*** doug-fi__ has joined #openstack-infra19:07
*** julim has quit IRC19:07
*** jaosorior has quit IRC19:07
*** doug-fish has quit IRC19:07
*** doug-fis_ has quit IRC19:08
*** maishsk has joined #openstack-infra19:08
*** julim has joined #openstack-infra19:10
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Added summit_type_id/event_type_id filter to events endpoint
fungithat was probably added because some people were actually legitimately wanting to upload pdf content for reference, but it's not actually searchable in mediawiki so i agree that's probably safe19:10
fungior at least something we'd want to discourage anyway19:11
fungiso i don't object19:11
*** dims_ has joined #openstack-infra19:11
*** dims has quit IRC19:11
*** annegentle has joined #openstack-infra19:12
*** flongo has joined #openstack-infra19:12
*** annegentle has quit IRC19:13
*** angdraug has joined #openstack-infra19:13
*** mtanino has quit IRC19:14
anteayaI'm open to even reducing the ability of our users to use the wiki temporarily until we can identify what will stop the spamming19:15
mrmartinwould be great to identify a pattern here19:16
*** doug-fi__ is now known as doug-fish19:16
*** annegentle has joined #openstack-infra19:16
fungiyeah, i'm trying to analyze Helen's use now19:16
*** ifarkas has quit IRC19:16
fungicurious to see whether Helen is coming from more than one ip address for example19:16
*** jsavak has quit IRC19:16
*** tiswanso has quit IRC19:16
*** jsavak has joined #openstack-infra19:17
pabelangerA lot of the changes I see now are related to PDF files19:17
*** baoli has quit IRC19:18
fungiand whether other users than Helen are coming from the same address too19:18
mrmartinbut they are also creating pages19:18
anteayapabelanger: can you take your workflow -1 off of 286189?19:18
*** baoli has joined #openstack-infra19:18
pabelangeranteaya: done19:18
openstackgerritAustin Clark proposed openstack-infra/project-config: Create stackviz element and script in nodepool (WIP)
prometheanfireany DIB cores around to give this +W? it has the two +2 it needed, not sure why it didn't get workflow.
fungiso... at the same time Helen created i see a get of that url from in the apache logs. i'm having trouble tracking down the actual file upload in the logs19:19
pabelangerIf I can figure out how to add 1-800-919-0992 to spam19:19
pabelangerthat will get a lot of quickbooks pages19:19
*** tiswanso has joined #openstack-infra19:19
fungibut that ip address was making other requests from the wiki well before Helen's account was created19:19
*** ihrachys has joined #openstack-infra19:20
*** e0ne has quit IRC19:20
anteayapabelanger: thank you19:20
fungiso as i would expect, confirmation that there are multiple accounts being used from the same client addresses19:20
anteayaclarkb: openstackjenkins-osic is only mentioned in this file: what is it?19:21
clarkbanteaya: it is the account that nodepool uses19:21
*** _nadya_ has joined #openstack-infra19:21
anteayaclarkb: ah thank you19:21
clarkbanteaya: in each of our clouds we have two accounts, the account for services (mirrors etc) and the account to run tests in19:22
clarkbopenstackjenkins == run tests in and openstackci == services19:22
anteayathank you, I didn't know that19:22
anteayaclarkb: can you review 286189?19:23
*** kzaitsev_mb has quit IRC19:23
anteayawe are hoping that may help19:23
*** baoli has quit IRC19:23
*** baoli has joined #openstack-infra19:23
clarkbanteaya: have we seen people POSTing pdfs as part of the spamming?19:23
fungianteaya: another way of looking at it is that any machines booted by nodepool go in their own isolated project (openstackjenkins) so that we don't have to trust nodepool with credentials to our other servers19:23
clarkbwhere people == spam bots?19:23
openstackgerritAustin Clark proposed openstack-infra/project-config: Create stackviz element and script in nodepool (WIP)
anteayaclarkb: yes19:24
fungiclarkb: a majority of the activity is uploads of pdfs19:24
anteayafungi: that is a good way of looking at it, thank you19:24
fungithough i've no doubt that as soon as we stop allowing pdf uploads, they'll just fall back to doing more spam content in pages19:24
fungiwhich they're already doing quite a bit of anyway19:24
pleia2just +Aed the pdf one19:24
pleia2but yeah, I'm sure inline spam will come back very fast19:25
anteayapleia2: thank you19:25
*** ybathia has joined #openstack-infra19:25
*** maishsk has quit IRC19:25
* anteaya eats some food19:25
*** baoli has quit IRC19:26
openstackgerritMerged openstack-infra/openstackid-resources: Added summit_type_id/event_type_id filter to events endpoint
*** dkehn has quit IRC19:27
openstackgerritJames E. Blair proposed openstack/gertty: Make topic selection a select list
openstackgerritJames E. Blair proposed openstack/gertty: Indent projects under topics
openstackgerritJames E. Blair proposed openstack/gertty: Add configuration information to docs
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Remove obsolete dvipng install
openstackgerritAndreas Jaeger proposed openstack-infra/system-config: Remove obsolete dvipng install
clarkbthere are a bunch of older ready nodes in nodepool that jenkinses don't seem to know about, I am going to mark them delete so that we can put that quota to use19:31
*** rossella_s has quit IRC19:32
*** austin81 has left #openstack-infra19:32
fungifun. so looking at the openstack_wiki db, Helen (user.user_id 7792, created today at 18:06:28) has a user.user_email of and user_openid.uoi_openid url which i think corresponds to but someone probably needs to use the launchpad.people.getByOpenIDIdentifier() api method to confirm19:33
*** ybathia has quit IRC19:33
*** bgaifullin has joined #openstack-infra19:33
fungiclarkb: oh, yeah, i was going to start digging into seeing which jenkinses are broken and in need of cleanup a couple hours ago before i got distracted by wikispam19:34
*** IlyaG has joined #openstack-infra19:34
fungii'll get back to that19:34
*** IlyaG has quit IRC19:34
clarkbfungi: just open that login url in your browser19:34
clarkbfungi: it will confirm that it belongs to suraj19:34
mrmartinmember since: 2010-07-15 ?19:34
AJaegerfungi, can I distract you for two mins about bindep/ubuntu-trusty?19:34
fungiclarkb: all that tells you is an identifier, but i've seen there be duplicates19:34
fungiAJaeger: probably not today, no19:35
*** rossella_s has joined #openstack-infra19:35
AJaegerok, fungi19:35
clarkbfungi: you cna have multiple openids that map to a single ~user19:35
fungiclarkb: the openid url returns what they set their visible name to, but it's not necessarily == to their lp username19:35
clarkbfungi: but  Idon't think you can have multiple accounts with different openids and ~users19:35
*** mtanino has joined #openstack-infra19:35
clarkbfungi: huh19:35
fungiclarkb: it's not actually a proper lookup, which is why we got them to add that reverse lookup api call for us19:35
*** sambetts is now known as sambetts|afk19:36
fungiclarkb: for example digging my openid url out of the gerrit db and loading it in a browser, it says "OpenID Identity URL for Jeremy Stanley" but the launchpad.people.getByOpenIDIdentifier() should properly reverse that to
*** dkehn has joined #openstack-infra19:38
*** baoli has joined #openstack-infra19:39
fungiand there's no unique key in lp for the account display name afaik19:39
fungiso this could be any random lp account with display name set to suraj19:39
pabelangerokay, I'm going to start deleting old PDF files on the wiki19:40
mrmartina real captcha at the new user registration page (even from openid) would solve this issue.19:40
clarkbmost of the "ready" nodes belonged to jenkins0219:40
clarkbso I am ssuming we removed them from the jenkins master but not nodeool when it got restarted19:40
clarkbfungi: huh fun19:40
fungiclarkb: yep, that was a known problem then. when yolanda tried to restart jenkins02 earlier nodepool wasn't seeing zmq from it19:40
fungior that's what it seemed like anyway19:40
*** jordanP has quit IRC19:41
fungiso used or finished slaves weren't transitioning out of ready state in nodepool19:41
anteayapabelanger: ack19:41
kevinbentonanteaya: hey, i remember you mentioned infra had an issue with neutron security groups. who was it?19:42
anteayakevinbenton: hi yes thank you for rememebering19:42
nibalizerkevinbenton: it was in infracloud19:42
clarkbfungi: they were all at least 3 hours old and some were 5 days old so I went ahead and set them all to delete19:42
anteayacan we try again perhaps tomorrow or wednesday19:42
*** tongli has joined #openstack-infra19:42
anteayakevinbenton: the channel is focused on trying to address spam in the wiki19:42
fungijenkins03 looks like it's got a ton of cruft nodes. going to start working on cleaning it up next19:42
anteayakevinbenton: but we would really like to have the neutron default security groups chat19:43
fungialso checking bluebox for leaked floating ips19:43
*** jordanP has joined #openstack-infra19:43
kevinbentonanteaya: ack. maybe tomorrow?19:43
*** jordanP has quit IRC19:43
anteayajenkins03 was also restarted today according to AJaeger and yolanda, I never did get the backscroll read19:43
anteayakevinbenton: awesome thank you, let's try tomorrow19:43
kevinbentonanteaya: sounds good19:43
fungianteaya: in that case it never got its list of workers cleaned up or something19:43
anteayakevinbenton: thanks for remembering19:43
*** ihrachys has quit IRC19:44
anteayafungi: that sounds likely19:44
AJaegerjhesketh restarted jenkins0319:44
fungianteaya: it seems to have lots of offline slaves that refer to nodepool nodes which were deleted some time ago19:44
fungiso i'm guessing he didn't clean out the slave entries in its config when he restarted it19:44
pabelangerI am sad that is empty19:44
anteayanot the most efficient19:44
anteayafungi: likely19:44
pabelangerNeed to look into why19:45
anteayafungi: I wonder if he knows to do that?19:45
AJaegeranteaya: commented on 286242 - see
anteayapabelanger: I can't see it, I'm not an administrator19:45
anteayaAJaeger: okay thanks19:45
fungianteaya: no idea. the closest thing we have to documentation of a manual restart process is clarkb's ansible playbook19:45
anteayaafter the fun dies down perhaps we could maybe look at some docs for this process19:46
clarkbwhich does not remove things from nodepool because it relies on things to coalesce when in shutdown mode19:46
anteayaAJaeger: ah wonderful19:47
fungiyep, though it _does_ remove remaining slaves from jenkins, which someone doing this manually might not think about19:47
fungire: jenkins0319:47
clarkbyup it will remove offline slaves once all jobs are done running19:47
anteayaI think both jhesketh and yolanda were caught by not knowing that19:47
pabelangerSo we haven't had any new pages since 19:19 UTC19:47
pabelangerand that was a valid user19:47
*** achanda has joined #openstack-infra19:47
pabelangerSo, the spammers appear to be hitting existing pages / files19:48
fungii'm deleting 45 leaked floating ips in bluebox now19:48
anteayafungi: thank you19:48
pabelangerHelen is spamming like a champ19:48
anteayahopefully she reveals her pattern19:49
fungipabelanger: a.k.a. "suraj"19:49
pabelangerI haven't blocked anybody today, just deleting pdf files19:49
pabelangerhopefully once our upload patch lands, traffic will slow19:49
*** hashar has joined #openstack-infra19:49
fungialso keep in mind the Special:Nuke Ryan_Lane mentioned earlier, which should undo actions taken by a specific user or possibly also ip address19:50
jeblairdo not nuke ryan lane ;)19:50
*** MarkAtwood1 has joined #openstack-infra19:50
fungithough i don't see it at so we may need an extension for it?19:50
*** dalgaaf has joined #openstack-infra19:51
pabelangeris how I am deleting spam19:51
pabelangerwhich is nuke I believe19:51
fungioh, it does exist19:51
*** MarkAtwood1 has quit IRC19:51
*** MarkAtwood has quit IRC19:51
pabelangerMass delete is what wiki.o.o calls it19:51
fungiso anyway, we'll want a list of known spammer accounts to feed into that19:52
openstackgerritMerged openstack-infra/elastic-recheck: Add query for ceph pure virtual method called bug 1551305
openstackbug 1551305 in Cinder "backup service crashes in ceph job with "pure virtual method called"" [Medium,Confirmed]
zarojeblair: i have run a few tests with regards to
zarojeblair: let me know when you are available to discuss.19:55
*** gtmanfred has quit IRC19:56
*** gtmanfred has joined #openstack-infra19:57
*** achanda has quit IRC19:57
fungii've finished looking through the other jenkins masters and they seem fine, so once 03 is dealt with we should be back in good shape there19:57
anteayafungi: thank you19:58
*** dims_ has quit IRC19:58
anteayathe graphs in grafana look as I expect now19:58
mrmartinwe have 607 user accounts without passwords, but how will we identify the infected openid ones?19:59
*** armax has quit IRC19:59
clarkbmrmartin: I think you can start with ones that were created recently, then filter through changes they have made to identify which are spamming?19:59
fungimrmartin: we might need to work backwards from known spam patterns to put together a list of which accounts made/uploaded them19:59
*** Guest71383 is now known as redrobot20:01
*** ashleighfarnham has quit IRC20:02
*** baoli has quit IRC20:02
*** amitgandhinz has quit IRC20:03
*** sdake has joined #openstack-infra20:04
*** amitgandhinz has joined #openstack-infra20:05
mrmartinok, so we need to find the spam pattern first.20:07
mrmartinare we the only mediawiki users who met with this kind of attack?20:07
*** dtardivel has quit IRC20:08
*** cznewt has quit IRC20:08
clarkbmrmartin: aiui many wikis have come under similar attack recently and not just mediawiki20:09
*** dkranz-meeting is now known as dkranz20:09
clarkbubuntu for example uses moinmoin and ran into the same thing20:09
clarkb(which makes me think a mediawiki vulnerability is less likely as a source)20:09
*** gtmanfred has quit IRC20:09
mrmartinand do we know what was their solution for this problem?20:09
mrmartinhave they found any pattern in the attacks?20:10
clarkbmrmartin: they went read only20:10
mrmartinso we should close all the user accounts, and give away write-only access for trusted people only?20:11
*** achanda has joined #openstack-infra20:11
*** angdraug has quit IRC20:11
*** dims has joined #openstack-infra20:12
*** david-lyle_ is now known as david-lyle20:12
*** hockeynut_afk is now known as hockeynut20:13
pc_mHi all. I did a Jenkins run and one of the tests that produced a log before, is passing, but not providing any log. Any thoughts on what is going on and what I should do?20:13
*** cznewt has joined #openstack-infra20:14
*** eharney_ has joined #openstack-infra20:15
clarkbpc_m: it typically helps to link to the run in question20:15
pc_mclarkb: Sure. Was seeing if anyone would respond.
fungiyay... "Forbidden launching node id: 8386189 in provider: rax-iad error: [...] Forbidden: Quota exceeded for ram: Requested 8192, but already used 1531904 of 1536000"20:17
fungiso why are we over quota in rax-iad?20:17
*** maishsk has joined #openstack-infra20:17
*** gtmanfred has joined #openstack-infra20:18
*** yamahata has quit IRC20:18
*** exploreshaifali has joined #openstack-infra20:18
*** eharney has quit IRC20:19
fungi says "max-servers: 190"20:19
fungii wonder if they lowered our quota there?20:19
*** _nadya_ has quit IRC20:20
anteayajroll: who would know if infra's rax quota was changed recently?20:20
*** ZZelle_ has joined #openstack-infra20:20
fungiit's been that since raised it from 85 back in early december20:21
jrollanteaya: I could probably find out20:21
anteayajroll: you are awesome20:21
fungiit's possible we've had it set too high this entire time20:21
jrollfungi: tenant ID?20:21
*** sdake has quit IRC20:22
*** fawadkhaliq has quit IRC20:22
*** sshnaidm has quit IRC20:22
fungijroll: OS_PROJECT_ID=63777620:23
pabelangerfungi: do you mind promoting 286189 for zuul? that is the disable pdf uploads for wiki.o.o20:23
pabelangerfungi: currently 2h15mins in check queue20:23
*** doug-fish has quit IRC20:24
fungipabelanger: sounds like it just needs to be enqueued, not promoted20:24
pabelangerfungi: ah, didn't know right syntax20:24
jrollfungi: taking a look20:25
*** nmagnezi has quit IRC20:26
fungipabelanger: it's in the gate now, and getting workers assigned20:26
*** kzaitsev_mb has joined #openstack-infra20:27
jrollfungi: that quota is accurate, I'm not sure where to find logs on quotas though :/20:27
fungianyway, i'm inclined to lower the max-servers there to 180 (effective quota of 187 minus 4 for template instances during snapshot image builds, pls a few breathing room)20:28
pleia2fungi: makes sense to me20:29
*** dims has quit IRC20:29
*** doug-fish has joined #openstack-infra20:30
*** _nadya_ has joined #openstack-infra20:30
anteayaif we avoid the over quota errors I'm fine with lowering it20:30
*** annegentle has quit IRC20:31
clarkbpc_m: which test?20:31
*** annegentle has joined #openstack-infra20:32
jpmaxmanHey guys on wiki.o.o any summary update ?   Sorry I had to step away for a bit.20:32
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Lower max-servers in rax-iad to 180
fungipleia2: anteaya: ^20:32
openstackgerritgordon chung proposed openstack/requirements: bump aodhclient upper-constraints
pc_mclarkb: networking-cisco-coverage20:32
*** doug-fis_ has joined #openstack-infra20:32
jpmaxmansorry guys and gals :D20:32
*** sshnaidm has joined #openstack-infra20:32
fungipabelanger: eta 6 minutes to merge on the pdf blocking change20:33
pc_mclarkb: Says "File Not Found".20:33
pc_mclarkb: Produced output on previous versions.20:33
*** dizquierdo has joined #openstack-infra20:33
*** tongli has quit IRC20:34
clarkbpc_m: "ran out of time uploading files"20:34
pabelangerfungi: woot20:34
fungijpmaxman: so we confirmed there are definitely compromised/throwaway lp accounts involved. we've temporarily disabled new account creation and are working on characterizing some means of making things harder for them (rejecting pdf uploads, url/pattern blacklisting is the current thrust)20:34
clarkbjpmaxman: a change is going in to prevent pdf uploads20:34
*** doug-fish has quit IRC20:34
clarkbjpmaxman: and I think fungi is still working to identify a pattern of users/spamming20:34
pc_mclarkb: Can you point me to info on how I could find the above link (so I don't have to ask)?20:35
fungiyeah, took a break to try to figure out what our current ci node launch errors were all about20:35
*** keedya has quit IRC20:35
*** keedya has joined #openstack-infra20:35
clarkbpc_m: you hvae to go to the individual jenkins masters (01-07) and look at the job logs there20:35
*** pots72 has joined #openstack-infra20:36
*** salv-orlando has joined #openstack-infra20:36
clarkbpc_m: they list what change triggered them and when they ran and from that you can find the one you are interested in typically20:36
*** |-paul-| has joined #openstack-infra20:36
anteayafungi: thanks for that20:36
fungi5 jobs remaining before jenkins03 has quiesced and then i can take it back down and clean it up20:37
*** doug-fis_ has quit IRC20:37
*** mizar has joined #openstack-infra20:37
*** flongo has quit IRC20:37
fungiworking now on reversing our lp spammer case study openid url to an actual lp account20:37
*** xyang1 has joined #openstack-infra20:38
*** keedya has quit IRC20:38
*** maishsk has quit IRC20:38
*** keedya has joined #openstack-infra20:39
*** salv-orl_ has quit IRC20:39
*** exploreshaifali has quit IRC20:39
pleia2off for some lunch + lunchtime errands, bbiab20:40
*** ybathia has joined #openstack-infra20:40
*** Sukhdev has joined #openstack-infra20:40
openstackgerritMerged openstack-infra/puppet-mediawiki: Disable PDF uploads by users
pc_mclarkb: Thanks. I'm running coverage test comparison. Similar to Rally project. seeing differences run to run with coverage reporting. :(20:40
AJaegeryeah, check queue is getting shorter - and the Zuul Job Queue is going down as well - so jenkins cleanup efforts have helped. Thanks!20:41
jpmaxmanso to me the question still remains if I'm using a comrpomised/throwaway lp account am I succefully guessing / scripting the captcha or are the edits/updates being made some other way (API, etc)?20:41
*** ybathia has quit IRC20:41
boris-42pc_m: hi20:41
pc_mboris-42: hi20:41
clarkbpc_m: keep in mind that we merge against the target branch for each run20:41
*** eharney_ is now known as eharney20:41
clarkbpc_m: so if the underlying base changes coverage can change20:41
pabelangerjpmaxman: right, but once we do the first round of patches / clean up, it will be easier to see where those POSTs are coming from20:42
anteayapleia2: enjoy20:42
pc_mclarkb: On this commit, it runs HEAD vs HEAD^ (which would be the change set for this commit and recent, and this commit is only script changes).20:42
pabelangerI haven't see any HTTP logs, but everything appears to just be a POST to know page20:42
clarkbpc_m: right and HEAD^ may change on each run20:42
clarkbpc_m: because zuul merges the change into latest master and uses that ref for testing20:43
jpmaxmanright but that POST could be happening as a result of a user/script hitting the submit button and properly answering the captcha - or it could be a direct post bypassing...20:43
mtreinishclarkb, fungi, nibalizer, pleia2: is there anything in the subunit gearman worker logs? There have been 3 results added to the db since clarkb restarted it last week20:43
fungipabelanger: jpmaxman: right, i haven't been able to adequately distinguish browser-based and api-based calls from the apache logs. for example, both seem to use action=edit20:43
jpmaxmananyway, hopefully this next round of passes will make it more clear :D20:43
*** pots has joined #openstack-infra20:43
pc_mclarkb: See different # lines missing/partial covered. so not a difference from run to run, but on a single run.20:44
jpmaxmanwhat role does disabling the PDF upload play?20:44
pabelangerFor the most part, the spam I am deleting is all the same.  Either quickbooks or microsoft care. So, exposing the tools to allow admins to quick blacklist those keywords will also help too20:45
pabelangerright now, we are just flooded with users posting junk20:45
*** maishsk has joined #openstack-infra20:45
fungipabelanger: jpmaxman: that's on my to do list is to figure whether there's even a difference in the "api" and the web browser urls, though i see evidence that the attacker retrieves the entire page after the post results in a 302, and also pulls any css and images linked in the target confirmation page. even includes a referrer from the post url, making it seem authentically browseresque20:45
*** dprince has quit IRC20:45
openstackgerritErno Kuvaja proposed openstack/requirements: Bump up glance_store to 0.12.0
jpmaxmanis the write API enabled for a specific use?20:45
jpmaxmanYou can disable all write modules by setting $wgEnableWriteAPI = false; in LocalSettings.php. The write API is enabled by default as of MediaWiki 1.14, and disabled by default in older versions.20:45
clarkbjpmaxman: we have a small numbre of bots that do legit updates to the wiki20:46
pabelangerWe've plugged the new users being created, since Helen is the latest user.  But I also think any new user via openid is blocked too20:46
fungiwell, i don't mind breaking bot writes to the wiki for a little while as a troubleshooting measure20:46
fungitheir use is infrequent and informational in nature20:46
*** baoli has joined #openstack-infra20:47
fungiobviously disabling the api indefinitely is probably not something we want, but finding a way to lock api use down to teh bots account group would be cool20:47
mtreinishclarkb: hmm, a None is getting passed into subunit2sql for the stream20:47
jpmaxmanagain I think any of these efforts would be great troubleshooting steps.20:47
mtreinishclarkb: is that just once or a bunch of them?20:47
jpmaxmandisabling write api to see if it does anything20:47
jpmaxmanwould be telling for sure20:48
*** baoli has quit IRC20:48
clarkbright the question was is it enabled for a reason, that is the reason. I don't care if we lock it down temporarily20:48
clarkbif I hda more time to devote to this I owuld just make the entire wiki read only20:48
*** baoli has joined #openstack-infra20:48
clarkband undo that in a week and see if they have moved on20:48
*** jsavak has quit IRC20:48
jpmaxmankind of like an impossible to answer captcha question ;)20:49
jpmaxmanbut they would be back20:49
clarkba few meeting agendas would suffer but other than that I don't think it would hurt us too badly20:49
jpmaxmanIMHO we need to find how they're getting in and plug the hole20:49
*** ociuhandu has quit IRC20:49
pabelangerwell, right now it is only existing users that are spamming. Since we stopped new20:49
jpmaxmanbut I think we need to iterate faster on enabling disabling20:49
pabelangerand I am simply blocking them as they create new pages20:49
jpmaxmanthe nice part is they are hitting it so hard20:49
jpmaxmanyou will know in a few minutes if it works :D20:50
mrmartinI'll do some pattern analysis tomorrow if we don't have an ultimate solution until next morning.20:50
jpmaxmanI mean here
jpmaxmanit is literally 20:49 pabeleanger deletes20:51
jpmaxman20:50 new spam20:51
pabelangerya, but if you look at their contribs, they are old accounts.  So, mass delete is nuking a fair bunch of stuff ATM20:52
pabelangerwe still haven't landed the password auth disable20:52
pabelangerthat is in the gate20:52
pabelangerwe could enqueue that too20:52
fungipabelanger: project name, change id and patchset number?20:52
jpmaxmanwas just more commenting on the speed20:52
fungii can kick it over now20:52
pabelangerfungi: 28566920:53
pabelangerfrom what I see, it is file uploads ATM20:53
pabelangeror existing pages20:53
pabelangernew page creation is down20:53
pabelanger19:19 UTC was last new page, and that was l18n team20:53
*** IlyaG has joined #openstack-infra20:54
*** jsavak has joined #openstack-infra20:54
fungipabelanger: so openstack-infra/puppet-mediawiki 285669,2 looks like, but we need a second core reviewer20:54
pabelangerAh, right20:54
fungipleia2: you had reviewed that one previously, looks like20:54
pabelangerthat will break StatusBot I believe20:54
pabelangerbut maybe we just notify people of the breakage20:55
fungipabelanger: break it, or just prevent it from successfully uploading the log on the wiki?20:55
*** Sukhdev has quit IRC20:55
pabelangerfungi: I believe it won't post to the wiki20:55
fungier, updating the log on the wiki20:55
fungiso i feel like wiki updating features of #status and #success are probably okay to temporarily suspend while we're working on this as long as we're sure to solve for them soon20:56
pabelangerI started looking at the code last night, but didn't figure out how it auth'd.20:56
pabelangerI don't think it would be hard to convert it to openid20:57
*** IlyaG has quit IRC20:57
pabelangeror API key (if that exists)20:57
clarkbI think it auths with an api key like thing today20:57
pabelangeror allow password auth based on user group20:57
pabelangerokay, so then it might still work20:57
*** angdraug has joined #openstack-infra20:59
tonyblifeless: Are you able to look at the fixtures thing?  The neutron gate is still blocked and it'd be good to do or anadon the fixtures path so we can work on alternates if required.21:00
tonyblifeless: sorry to keep hasseling you ....21:00
*** chlong_ has joined #openstack-infra21:01
fungiokay, bad news. the openid for Helen is somehow blocked from reverse lookup in lp i think. if i pass in my openid url (connecting with our gerrit lp creds on review.o.o) it returns the url to my lp account, but if i pass in Helen's openid url i get back a None object from people.getByOpenIDIdentifier()21:02
fungithough that may also just mean that you can create a account which doesn't map back to a account21:03
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: DO NOT MERGE: Print output of ovs-vsctl show from hosting node
fungigiven that they've been described as distinct and unrelated systems21:03
clarkbfungi: could be an ubuntu one only account21:03
clarkbright that21:03
*** davideagnello has quit IRC21:03
*** davideagnello has joined #openstack-infra21:04
fungithough i'll go ahead and push up this lookup utility script anyway for future use21:04
clarkbjenkins came back +1 on finally, if I could get that approved real soon now that would be great (it is first step in starting to use osic for testing)21:05
clarkbonce that is in I will fix security groups and attempt launching a mirror host21:05
*** sputnik13 has quit IRC21:05
pabelangerfungi: interesting, maybe something long term is to expect wiki.o.o users to have created accounts in review.o.o21:06
anteayapabelanger: or at least foundation accounts21:06
pabelangeranteaya: right21:06
clarkbpabelanger: or maybe need to explicitly ack users befor ethey get write access?21:07
*** sputnik13 has joined #openstack-infra21:07
anteayaas not every person who wants to offer a wiki page/edit understands gerrit21:07
pabelangerI think somebody suggested but unsure what the migration to that looks like21:07
clarkb(that is a lot of work for humans though)21:07
anteayaclarkb: yeah, it would be21:07
pabelangerclarkb: right, I was looking at some sort of moderation extensions, but not sure humans want to do that21:07
anteayabut we could enact it as spam measures21:07
clarkbfungi: we can ask for help on the canonical/ubuntu one side21:08
pabelangerif we did moderation, then captcha goes away21:08
anteayaso not having a human approve during normal use, but bring it in in times of spam21:08
*** salv-orlando has quit IRC21:08
*** amrith is now known as _amrith_21:08
anteayahow many wiki users do we have?21:08
anteayaany ideas?21:08
mrmartinif they could filter spam accounts, our case can be much simpler.21:08
*** salv-orlando has joined #openstack-infra21:09
anteayaand what is the rate of creation during non-spam times21:09
clarkbanteaya: if id numbers are an indication ~7k21:09
fungipabelanger: worth considering, but though i suspect the page update latency might be terrible if we had every edit moderated21:09
anteayaclarkb: okay, that's not bad21:09
anteayahave we any kind of rate of creation numbers from say, last month?21:09
pabelangerfungi: maybe some promotion system.  First edit is moderated, if valid you go into good users and by pass moderation21:09
*** baoli_ has joined #openstack-infra21:10
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Add a simple LP OpenId reverse lookup utility
mrmartinanteaya: 1351 new user in the last month21:10
*** baoli_ has quit IRC21:10
fungipabelanger: these also seem like longer-term solutions, since the availability, efficacy and security of existing extensions for this are likely quite different after we upgrade mediawiki, which means after we move it to ubuntu trusty21:11
anteayamrmartin: yeah okay so, about 6K up until now21:11
pabelangerfungi: agreed21:11
anteayaabout the same as gerrit, I think21:11
mrmartin1351 new21:11
anteayaI wonder if there is anything that would track the rate of new user creation?21:12
*** _nadya_ has quit IRC21:12
fungiwe could trivially graph it from the account registration timestamps in the openstack_wiki.user table21:13
ianwwhere are the rules about what goes on ?21:13
*** baoli_ has joined #openstack-infra21:13
fungiianw: documentation for official (tc-recognized) project-team deliverables21:13
fungiand documentation for other efforts by official project-teams21:13
*** baoli has quit IRC21:14
*** ybathia has joined #openstack-infra21:14
fungiianw: so basically, no publishing documentation for unofficial teams is about the only rule i guess, and i don't think that's explicitly documented anywhere21:14
*** ldnunes has quit IRC21:15
ianwfungi: right, so something like "bashate" falls where?  and gerrit-dashboard-creator?21:15
*** amitgandhinz has quit IRC21:15
ianwi don't think they're official, but widely used21:15
*** bpokorny has quit IRC21:16
*** rfolco_ has quit IRC21:16
openstackgerritTim Buckley proposed openstack-infra/devstack-gate: Install and run StackViz during devstack-gate host cleanup. (WIP)
fungiianw: according to bashate is maintained by the qa team21:16
mrmartinI'm leaving now, trying to do something tomorrow that helps the find infected openid accounts.21:16
anteayamrmartin: thanks for your help today21:16
fungimrmartin: thanks for your assistance, and have a great evening!21:17
*** ayoung has joined #openstack-infra21:17
mrmartinthnx, bye21:17
*** mrmartin has quit IRC21:17
fungiianw: i agree that gerrit-dashboard-creator is not listed there21:17
openstackgerritTim Buckley proposed openstack-infra/subunit2sql: Add API methods for getting tests by prefix
*** hashar has quit IRC21:18
fungiianw: so presumably an unofficial project, though could either ask the tc to make its maintainers an official team or ally with an existing official team21:18
anteayaianw: if it can't go on docs.o.o it goes to readthedocs:
fungiwell, rtd is a common solution to the problem, but it's far from the only option21:18
anteayaokay fair21:18
anteayaI don't think I've seen docs go anywhere else21:18
*** cloudtrainme has quit IRC21:19
anteayagerrit-dash-creator is sdague's,members21:19
fungithere are plenty of webservers out there which you could use to host documentation21:19
*** jsavak has quit IRC21:19
ianwok, cool, yeah i started a change to fix dash-creator for rtd21:19
*** jsavak has joined #openstack-infra21:19
ianwthat one would be particularly useful to have documentation, because each dashboard actually builds itself into a useful info page21:20
fungiianw: as anteaya points out, you might want to confer with sdague to see if he has a preference21:20
ianwmaybe we can stick it under qa21:20
ianwyeah, will do21:20
*** kgiusti has left #openstack-infra21:20
fungihis unoffical project, so his rules21:20
fungier, unofficial21:21
*** cloudtrainme has joined #openstack-infra21:21
fungiokay, where was i?21:21
openstackgerritMerged openstack-infra/jenkins-job-builder: * Added support for the Jenkins Slack plugin
pabelanger is looking real good ATM21:22
pabelangeronly my actions for the last hour really21:22
fungiany other infra-core or infra-puppet-core reviewers in favor of temporarily disabling password authentication for wiki accounts (now that new password-authed ones can't be created by non-admins) while we work through cleaning them up?
*** dims has joined #openstack-infra21:23
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: DO NOT MERGE: Print output of brctl show from hosting node
*** aeng has joined #openstack-infra21:23
*** hashar has joined #openstack-infra21:24
fungitripleo, your container jobs take an excessive amount of time to complete21:25
clarkbfungi: looking21:25
fungi"Build has been executing for 2 hr 27 min"21:25
*** kingia has quit IRC21:28
*** lucas-dinner has quit IRC21:28
*** bpokorny has joined #openstack-infra21:29
*** yamahata has joined #openstack-infra21:29
*** davideagnello has quit IRC21:32
pabelangerfungi: are you okay with me blocking helen or do you want to keep using her account to collect data?21:32
*** esker has joined #openstack-infra21:32
fungipabelanger: all the info i could possibly collect is not going to be hapmered by blocking that account21:32
pabelangerfungi: roger21:32
fungiso go for it21:32
anteayamy grandmother's name was Helen21:32
anteayaI keep picturing her at her dressing table with a computer hammering away and cackling to herself21:33
anteayathen closing the laptop and offering tea21:33
anteayaoh Helen21:33
*** flepied has quit IRC21:33
anteayasc68cal: tea?21:34
*** davideagnello has joined #openstack-infra21:34
sc68calanteaya: why thank you :)21:34
smarcetfungi: could u approve i need it to fix openstackid-dev :)21:35
*** aysyd has quit IRC21:35
*** ociuhandu has joined #openstack-infra21:35
*** stevebaker has quit IRC21:35
openstackgerritMerged openstack-infra/puppet-mediawiki: Disable standard password based auth
*** amitgandhinz has joined #openstack-infra21:36
*** lucasagomes has joined #openstack-infra21:36
*** kingia has joined #openstack-infra21:36
*** stevebaker has joined #openstack-infra21:36
*** tpsilva has quit IRC21:37
*** abregman is now known as abregman|afk21:37
anteayasc68cal: I've put the kettle on21:38
fungismarcet: i would have liked to see a second core reviewer on that one, but it's trivial enough that i feel marginally okay single-core approving since you're under a production deadline rush21:38
fungiand it's been sitting there most of the day with just my review21:38
*** dims has quit IRC21:40
openstackgerritMerged openstack-infra/project-config: Skip dsvm jobs on cinder docs/reno/unit test changes
*** arxcruz has joined #openstack-infra21:42
smarcetfungi: thx a lot :)21:44
*** dims has joined #openstack-infra21:44
*** smarcet has quit IRC21:45
openstackgerritMerged openstack-infra/puppet-openstackid: Update App Configuration
fungipabelanger: since 285669 merged a couple minutes ago, do you feel up to writing a message to the -dev ml letting people know we've temporarily broken #success while trying to work through the wiki spam problem, and that we'll announce again once it's back in working order?21:48
*** |-paul-| has quit IRC21:48
fungithough i guess test a #success first to make sure it really is broken21:48
*** baoli_ has quit IRC21:48
fungionce it ends up on the wiki server21:48
fungiokay, jenkins03 finally just went idle, so cleaning it up now21:49
pabelangerfungi: sure21:49
anteayamorning jhesketh21:49
pabelanger#success we are combating the spam on wiki.o.o21:49
openstackstatuspabelanger: Added success to Success page21:49
*** maishsk has quit IRC21:49
pabelangerfungi: still working21:50
pabelangerwill try again in a few minuts21:50
pabelangerbut agree, a ML post would be good21:50
*** fawadkhaliq has joined #openstack-infra21:50
anteayajhesketh: so you restarted jenkins 03 for us last night, thank you21:52
fungiokay, jenkins03 is cleaned up and on its way back up now21:53
jheskethanteaya: yes but it appears to be still having trouble?21:53
*** armax has joined #openstack-infra21:53
anteayajhesketh: so there are some points about restarting a jenkins in order to ensure it removes all its old nodes and uses zmq to communicate with nodepool21:53
*** edmondsw has quit IRC21:53
* jhesketh is still catching up21:53
anteayawell I'm light on details21:53
fungijhesketh: i'm guessing you didn't know to delete the slaves from its config.xml21:53
anteayabut yes both the server you restarted and the server yolanda restarted need restarting again21:53
fungithe zmq issue with with jenkins02 and i have a feeling that one was because of not making sure the java processes were completely dead21:54
jheskethfungi: ah no, I thought removing them from nodepool was enough21:54
jheskethSorry :-(21:54
fungino problem. we'll try to get something written up a little more step-by-step21:54
fungibasically if you nodepool delete while jenkins is offline, it never gets cleaned up in jenkins and so it ends up with numerous dead slaves it thinks should be reachable but aren't21:55
anteayafungi: ah sorry21:55
*** hashar has quit IRC21:55
*** jpr has quit IRC21:56
jheskethfungi: jenkins02 had trouble too?21:56
fungilooks like jenkins03 already has new workers assigned and running jobs, so should be all set21:56
anteayayes after you were asleep yolanda addressed 02 and 0421:56
pabelangerfungi: so, pdf block looks good. Just seen a spammer switch to jpeg upload. But that's been the first spammer in a few hours now21:56
fungipabelanger: fun!21:56
anteayapabelanger: do we allow jpeg upload?21:57
pabelangeranteaya: Ya, there is some default IIRC21:57
jheskethfungi: so why did 02 and 04 need restarting?21:57
fungijhesketh: thread leak sounded like21:57
jheskethRight okay, so a coincidence rather than due to me screwing up 0321:58
anteayapabelanger: can you see how many spammers we have? or is it still a crowd?21:58
anteayajhesketh: yeah, doesn't appear related21:58
sc68calthose spammers are persistent bastards21:58
fungijhesketh: we have a scripted restart of them weekly to keep that under control, but it seems like they are at the point where that's not frequent enough and they go too far out to lunch sich that the restart script can't safely restart them because their apis are too unresponsive already21:58
anteayaI think all the jenkins are independent of each other21:58
anteayasc68cal: I have no opinion on their geneology21:58
*** salv-orl_ has joined #openstack-infra21:59
fungianteaya: reasonably independent, but problems on some can spike load onto others and create new problems for them21:59
anteayaah, thank you21:59
anteayasc68cal: tea is ready21:59
jheskethWhen I left 03 was picking up jobs again. What is the consequence of it trying to connect to dead nodes that weren't removed from config.xml?21:59
fungilike if we were to knock half our jenkins masters offline at peak load, the remaining half with rapidly degrade because they're incapable of handling all the work on their own22:00
anteayafair enough22:00
jheskethfungi: right. I had tried to use those plays but they failed22:00
fungijhesketh: jenkins quickly recognizes they're unreachable and marks them offline on its own, they just never get cleaned up22:00
sc68calanteaya: ah thank you. used the electric kettle at indy hall, going to buy myself one for home22:00
anteayawhere is indy hall?22:01
anteayaI used an electric kettle22:01
sc68cal3rd and market - philly22:01
fungijhesketh: so jenkins03 had some hundreds of "offline" slaves it was periodically checking to see if they were ever coming back22:01
anteayahas a pretty blue light this one22:01
anteayasc68cal: nice22:01
anteayajenkins03 was so hopeful22:01
sc68calthe one at indy hall is pretty advanced - Bonavita variable temperature22:01
fungii've now reset its expectations to mostly match reality22:02
*** salv-orlando has quit IRC22:02
*** annegentle has quit IRC22:02
anteayabest anyone can hope for22:02
jheskethfungi: ah so it just ties 03 up. Did it cause further gate issues  (ie were the other masters unable to keep up? )22:02
sc68calit's got like 10 buttons on it. It's like openstack22:02
sc68calsorry - 622:02
anteayasc68cal: wow22:02
fungijhesketh: it wasn't disruptive to the overall system that i could see, no22:03
anteayacan you convince it to boil water?22:03
fungiso no real harm done22:03
jheskethOkay that's good.22:03
*** kencjohnston has quit IRC22:03
sc68calanteaya: I kid you not, I just pressed all the buttons until something happened22:03
anteayasc68cal: sounds like openstack22:03
anteayaha ha ha22:03
* anteaya sips her tea22:03
anteayaone button, I have full confidence in it22:04
fungithe issue with jenkins02 was more troublesome since it was causing lots of in-use and offline nodes to look like they were satisfying ready node demand, and also sucking up quota because nodepool didn't know to clean them up22:04
jheskethfungi: the real harm from yesterday was done by me though sorry :-(. In fixing why ansible wasn't running I inadvertently caused nodepool to cease working22:04
fungijhesketh: yep, saw that, but you fixed it22:04
fungiso thanks for noticing and correcting22:05
jheskethI had removed Infra cloud from the clouds.yaml which nodepool didn't like.22:05
jheskethYeah I should have noticed it much sooner though :-(22:05
fungianyway, since i've gone all day too busy to eat something, i'm going to disappear and grab a meal, then hopefully back for more fun in an hour or so22:05
*** sdake has joined #openstack-infra22:05
jhesketh(And it was AJaeger that pointed it out )22:05
anteayafungi: enjoy food22:05
anteayajhesketh: he is so helpful that way22:06
*** doug-fish has joined #openstack-infra22:06
jheskethfungi: okay. Thanks for all you do!22:06
anteayaand supportive as you fix it, I'm guessing too22:06
jheskethLet me know if I can help22:06
jheskethanteaya: yeah he's pretty awesome22:06
anteayajhesketh: current issue is wiki spam22:06
anteayathere is an etherpad:
anteayapabelanger: does the etherpad reflect the latest reality?22:07
pabelangerokay, stepping away from desk. All this wiki stuff is getting to me.  But good news, the amount of new spam is almost zero in the last few hours22:07
pabelangeranteaya: no, I will update shortly after walk22:07
anteayaokay thanks, enjoy your walk22:08
*** sarob has quit IRC22:08
ianwfungi: so i think why i got myself confused is that the jobs look all correct, but no bueno ... i bet it has to do with "openstack-dev/"22:08
ianwcause devstack we use custom jobs22:08
anteayaianw: yes I think you are correct22:08
anteayayou can set the path in a specific variable22:09
*** dims has quit IRC22:09
*** baoli has joined #openstack-infra22:10
jheskethanteaya: okay. Sounds like pabelanger has made some great progress. I'll wait for his updates to the pad22:10
anteayamakes sense to me22:11
*** samueldmq has joined #openstack-infra22:13
*** dkranz has quit IRC22:14
*** _ody__ is now known as _ody22:14
*** phschwartz_ is now known as phschwartz22:14
*** sdake has quit IRC22:14
samueldmqhi, there is an approved keystone change in which all gates are presenting 'skipped' status22:15
samueldmqit's patch 23128922:15
*** jsavak has quit IRC22:15
samueldmqwhat does that mean?22:15
*** jsavak has joined #openstack-infra22:15
openstackgerritYuriy Taraday proposed openstack-infra/git-review: Use hash of test ID to pick Gerrit ports in tests
*** sdague has quit IRC22:18
anteayasamueldmq: okay it is in the gate pipeline and has a black dot beside it22:18
anteayasamueldmq: hover over the black dot22:18
*** sdake has joined #openstack-infra22:18
anteayait tells me it has failed because it has a merge conflict22:18
*** yamahata has quit IRC22:18
samueldmqanteaya: perfect! merge conflict22:19
anteayaso somewhere above it since it passed check a merge conflict got in22:19
*** alivigni has quit IRC22:19
anteayazuul saw it and pulled it out of the gate queue since it can't merge22:19
*** yamahata has joined #openstack-infra22:19
anteayabut leave it for now since if the gate resets it will get retested22:19
*** armax has quit IRC22:19
anteayaand it is possible the patch with the merge conflict may not merge22:19
anteayawe don't know yet22:20
samueldmqanteaya: makes sense, we know of other patch that was merging and would cause the merge conflict22:20
anteayathat would be it then22:20
samueldmqanteaya: so that's clear now for us why this is giving the error22:20
*** cloudtrainme has quit IRC22:20
mtreinishclarkb: so was that error in the log repeating itself or was it just a one off?22:21
samueldmqanteaya: thanks for your help; appreciated22:21
mtreinishclarkb: oh, actually I just found the bug in the script22:21
openstackgerritYuriy Taraday proposed openstack-infra/git-review: Use hash of test ID to pick Gerrit ports in tests
anteayasamueldmq: pleasure, thanks for the continued great questions22:21
EmilienMfungi: just fyi
samueldmqanteaya: always trying my best :)22:21
anteayayou are doing a great job22:21
crinklejeblair: clarkb when you have a moment today could you give your feedback on the infracloud network requirements thread?22:22
greghaynescrinkle: that LGTM22:23
*** claudiub has quit IRC22:23
crinklegreghaynes: go away and take care of your child :P22:24
greghaynescrinkle: oh, actually, one detail - the 'management network' needs to be reachable from the nodes we have22:24
greghaynescrinkle: crap, right!22:24
*** sarob has joined #openstack-infra22:25
*** rhallisey has quit IRC22:25
openstackgerritMatthew Treinish proposed openstack-infra/puppet-subunit2sql: Fix bug on missing subunit2sql data
mtreinishclarkb, fungi: ^^^22:26
*** armax has joined #openstack-infra22:26
mtreinishthat should fix the subunit gearman worker22:26
*** eharney has quit IRC22:26
*** abregman|afk has quit IRC22:29
keedyaclarkb is this review need any more modifications ?22:29
*** flepied has joined #openstack-infra22:30
openstackgerritIan Wienand proposed openstack-infra/project-config: Publish bashate docs
ianwanteaya: ^ upon further, further investigation22:30
* anteaya looks at 28630522:31
anteayathe repo has as its publish to site:
prometheanfireianw: can you +w this?
ianwanteaya: yep, but {name}-docs is the job that does the publishing, which you have to pull in via "openstack-*-publish-jobs"22:34
prometheanfireianw: it has two +2 (one from you :P)22:34
mmedvedejhesketh: around? I have a question on zuul swift upload22:34
*** reed_ has joined #openstack-infra22:35
jheskethmmedvede: sure22:35
ianwSpamapS: any reason you didn't workflow 281960 ^ ?22:35
*** doug-fis_ has joined #openstack-infra22:35
anteayaianw: I'd argue for infra-publish-jobs:
anteayaianw: as that gives you check, gate and post22:35
*** thorst has quit IRC22:35
anteayawhereas client just gives you release22:35
jeblaircrinkle: that lgtm22:35
anteayadoes bashate release?22:36
prometheanfirehe did ask that question, but dunno why that would bar a workflow22:36
crinklejeblair: ty22:36
mmedvedejhesketh: so I do not think swift formpost middleware does support setting content-encoding. I was wondering if you thought how to handle that, or maybe there is already solution.22:36
ianwanteaya: that puts things under ""22:37
mmedvedejhesketh: I am actively using zuul swift upload, but have to hack around to set content-encoding for compressed text, so that browser opens it properly.22:37
*** doug-fish has quit IRC22:37
*** Scalefab has joined #openstack-infra22:37
anteayaianw: oh, hmmm22:38
anteayaokay well if bashate releases then I guess client could work22:38
ianwanteaya: yeah, we just tag new releases.  i started down this path because i thought i should probably add release notes :)22:39
*** julim has quit IRC22:39
jheskethmmedvede: hmm not sure if the middleware supports it or not  (haven't looked). We haven't ran into it because we use os_loganalyze and can set our own headers there22:39
jheskethmmedvede: what are you serving the content from22:39
anteayaianw: okay then looks like it should work22:39
anteayayay release notes22:39
anteayathanks for your patience with me22:40
mmedvedejhesketh: we serve it from swift container itself22:40
jheskethmmedvede: so with some kind of cdn?22:40
keedyaanteaya: would you be able to review my pull request :
*** _amrith_ is now known as amrith22:41
*** baoli has quit IRC22:41
*** thorst has joined #openstack-infra22:42
mmedvedejhesketh: I do not think there is cdn involved. But your answer is good enough. I thought maybe I was missing something obvious.22:42
*** rockyg has quit IRC22:42
*** cj has joined #openstack-infra22:43
cjhey folks22:43
cjI'm having a heck of a time with gerritbot22:43
cjmostly because I can't get the test target to run22:44
*** ybathia has quit IRC22:44
cjwhich in turn is because of pbr throwing the «TypeError: dist must be a Distribution instance» error22:44
*** rockyg has joined #openstack-infra22:45
openstackgerritDmitry Borodaenko proposed openstack-infra/project-config: Added new repository for fuel-plugin-murano
*** regXboi has quit IRC22:46
*** thorst has quit IRC22:46
*** sdake has quit IRC22:46
*** doug-fis_ has quit IRC22:48
*** doug-fish has joined #openstack-infra22:48
anteayacj: what are you trying to do with gerritbot?22:49
cjanteaya: get the tests to pass, get one running for the fdio channels, replace openstackgerrit, etc.22:49
cjThey currently crash daily and leave no log messages22:50
*** tiswanso has quit IRC22:50
anteayaI'm sorry I'm still lacking context22:50
anteayayou are trying to run your own instance of gerritbot?22:50
*** jsavak has quit IRC22:50
cj(gerritbot)cjac@build0:/usr/src/git/gerritbot$ pwd22:50
cj(gerritbot)cjac@build0:/usr/src/git/gerritbot$ git describe22:50
cj(gerritbot)cjac@build0:/usr/src/git/gerritbot$ python test -v > /dev/null 2>&1 ; echo $?22:50
anteayacj: please use a paste service22:50
cjanteaya: for some value of "my own"22:51
cjwhere tha t value is "your own"22:51
anteayasorry I'm still lacking context22:51
anteayaand please use a paste service for logs and traces22:51
*** dizquierdo has quit IRC22:52
cjokay.  I'm on the LF release engineering team.  The gerrit bots are hard to maintain, do not print errors to logs and fail silently.22:52
cjbecause of this, we need to re-start the bot daily22:52
*** Scalefab has quit IRC22:52
*** doug-fish has quit IRC22:53
cjI want to patch it up and improve the test suite so that we can identify the problem, fix it and ensure that it stays fixed.22:53
anteayawhat is the LF release?22:53
*** doug-fish has joined #openstack-infra22:53
cjanteaya: I'm not sure.  I am a member of the Release Engineering team at The Linux Foundation22:54
cjI maintain services for projects hosted with The Linux Foundation.22:54
cjIncluding gerritbot instances22:54
*** achanda has quit IRC22:54
cjsuch as openstackgerrit, for instance22:55
*** amitgandhinz has quit IRC22:55
anteayaoh the linux foundation22:55
anteayaokay great22:55
anteayawell we aren't all that happy with our bots at the moment either22:55
anteayafor many of the same reasons you outline22:55
cjGreat, so there is some empitus to get it resolved.22:56
pleia2cj: has the problem been happening since you started running the bots, or just recently?22:56
jeblaircj: i don't think gerritbot has any tests...22:56
jeblaircj: but if it did, "tox -e py27" would be the way to run them22:56
cjpleia2: for quite a while, from what I hear.  I just started 3 weeks ago, and I've been tasked with standing one up.  It won't start, so the ticket is open until I fix that problem, specifically.22:56
pleia2cj: ah, gotcha22:57
pleia2cj: I met one of your colleagues at recently, trying to convince you all to open source more of your ops ;)22:57
*** baoli has joined #openstack-infra22:57
pleia2(helpfully I completely forgot his name)22:58
cjthat won't take much convincing.  Who did you speak with?22:58
cjAndrew Grimberg perhaps?22:58
cjpleia2: did you see ?22:58
pleia2I don't think so22:58
cjKonstantin maybe? :-)22:58
anteayathe gerrit bot tests look like it does pep8 and flake8 at best:
pleia2cj: I did see that <322:58
anteayano unit tests22:59
cjsuper!  I guess that means there's low hanging fruit.22:59
pleia2cj: could be, but I am also bad with faces \o/22:59
anteayavery much so22:59
cjbatting one thousand!22:59
pleia2I think this is why I like irc23:00
cjanteaya: okay, can you tell me how to launch this bot?23:00
anteayabeyond what is in the installation file, no I have not launched this myself23:00
pabelangeranteaya: jhesketh: I have added updates to the current status of wiki.o.o23:00
anteayapabelanger: thank you23:00
anteayapabelanger: did you have a nice walk?23:00
*** thorst has joined #openstack-infra23:00
pabelangerwe haven't had any new spam for the last few hours23:00
anteayapabelanger: yay no new spam23:01
pabelangeranteaya: not long enough23:01
anteayaI feel the same way about my walks23:01
anteayavery windy outside roads are bad23:01
anteayaheard of someone totaling their car today :(23:01
mordredcj: btw -  «TypeError: dist must be a Distribution instance» usually means your setuptools is way too old, iirc23:01
aenghi, wondering is there an outage at the moment?
pleia2cj: there's an init script for it that you should have if you're using our puppet module
anteayaaeng: what are you seeing?23:02
jeblaircj: you may also be interested in
aeng"Whoops, looks like something went wrong."23:02
anteayaaeng: we are tighening things up with the wiki23:02
anteayaaeng: what were you trying to do that you saw that?23:02
aenganteaya, when I click login23:02
aeng"The OpenID authentication failed."23:02
jeblaircj: and then as a first step, i'd recommend adding a "-d" option to it so that you can easily run it in the foreground.23:03
mordredjeblair: do we have the bot spec written up?23:03
jeblairmordred: no23:03
anteayaaeng: so you naviigate to, click login, and get "whoops, looks like something went wrong" is that accurate?23:03
*** sdake has joined #openstack-infra23:03
jeblairmordred: we're still at the impasse wrt interactive and yaml file configuration23:03
aenganteaya, navigate to that page, click login then "The OpenID authentication failed."23:04
*** baoli has quit IRC23:04
anteayaaeng: okay thanks23:04
anteayawho has access to the openid server to take a peek at some logs?23:04
jamesmcarthuraeng: there is a problem with OpenStackID smarcet: should be popping in shortly to request an update23:04
aenganteaya, actually,,, 1) Go to, 2) click login, "The OpenID authentication failed."23:05
openstackgerritAkihiro Motoki proposed openstack-infra/infra-manual: Update translation setup instructions
*** cloudtrainme has joined #openstack-infra23:05
anteayaaeng: ah thank you, that was the part I was missing23:05
aengand if navigate to "", message, Whoops, looks like something went wrong.23:05
*** thorst has quit IRC23:05
anteayajamesmcarthur: thanks for the update23:05
mordredjeblair: oh right23:05
aengjamesmcarthur, thanks for the update. Is someone looking at it now?23:06
pleia2oh good, smarcet is on it23:06
jamesmcarthurI think he might need an assist from someone on Infra, but I’d expect him to pop on in a moment23:07
pleia2I'm around as needed23:07
*** pots has quit IRC23:07
cjoh, hey mordred.  good to see you here.23:08
openstackgerritsebastian marcet proposed openstack-infra/system-config: OpenstackId relase 1.0.12
*** dingyichen has joined #openstack-infra23:08
*** achanda has joined #openstack-infra23:08
cjmordred: huh.  I did a pip install setuptools and removed the system python-setuptools and still get that exception23:08
openstackgerritMerged openstack-infra/project-config: Lower max-servers in rax-iad to 180
mordredcj: I do not get that error :)23:09
cjjeblair: is there a way I can check that review out from gerrit?  I'm still getting used to the gerrit environment.23:09
mordredcj: in your git repo, do "git review -d 253238"23:10
cjmordred: heh.  using virtualenv ?23:10
mordredcj: assuming you have git-review installed of course23:10
mordredcj: yes - I ran "tox -epep8 -r" ... -r says "please recreate the virtualenv23:10
jeblair(and if you don't have git-review installed, you might want to do so, it will make some things much easier)23:10
*** yamamoto_ has joined #openstack-infra23:11
*** sarob has quit IRC23:11
*** smarcet has joined #openstack-infra23:11
jamesmcarthurCan anyone approve this patch to fix OpenStackID?
*** fbo has quit IRC23:11
smarcetfungi: yolanda: mordred: could u please review and approve this one
*** sarob has joined #openstack-infra23:11
jeblairjamesmcarthur, smarcet: how did it break?23:11
smarcetcurrent production version is missing one class that is declared on the app.php.erb template23:12
smarcetso basically is throwing a missing class exception bc manifest23:12
cjpkg_resources.DistributionNotFound: The 'python-daemon<2.1.0,>=2.0.4' distribution was not found and is required by the application23:12
smarcetthat is bc the app.php.erb is dependant on version23:12
mordredwfm ... jeblair easy review above23:12
*** yamahata has quit IRC23:12
cjyeah, installing git-review was one of the first things my boss had me do23:12
*** akscram has quit IRC23:12
mordredcj: good boss!23:12
jeblairmordred: well, that's the change i'm asking questions about :)23:12
*** dimtruck is now known as zz_dimtruck23:12
mordredjeblair: neat23:12
*** sbadia has quit IRC23:13
jeblairmordred, smarcet, jamesmcarthur: i'm wondering how we could be in a position where we need an emergency upgrade to fix something which didn't break because of an upgrade23:13
*** akscram has joined #openstack-infra23:13
smarcet bc the app.php.erb is dependant on version23:13
*** sbadia has joined #openstack-infra23:14
smarceti updated for dev23:14
*** amrith is now known as _amrith_23:14
smarcetbut that classs that i added on the manifest for dev23:14
* aeng head twisting from jeblair comments 23:14
smarcetdoes not exist on current production release23:14
smarcetthat is why is breaking23:14
smarcetproblem is23:14
*** fbo has joined #openstack-infra23:14
anteayasmarcet: but if you added it to -dev how is it breaking in production?23:14
smarcetthe puppet manifest has no tagging23:14
*** dims has joined #openstack-infra23:14
smarcetbc dev and production are not the same23:15
cjmordred: you going to be at LFNW this year?  It occurs to me that the last time I saw you was at one of those.23:15
smarcetproblem is puppet manifest is the same for both23:15
*** Sukhdev has joined #openstack-infra23:15
smarcetdev and production23:15
smarcetand app.php.erb is dependant on release23:15
pleia2smarcet: so a change was made in puppet that broke prod?23:15
anteayaso the puppet manifest ran on production23:15
*** salv-orl_ has quit IRC23:15
anteayaand broke production23:15
*** yamamoto_ has quit IRC23:16
smarcetanteaya: aye23:16
jeblairthis could have been fixed with a revert of the puppet change, right?23:19
*** salv-orlando has joined #openstack-infra23:19
jamesmcarthurjeblair: sure, but it could also be fixed with this latest merge23:19
anteayaor we could be breaking all new things23:19
anteayawe don't know23:19
jamesmcarthurwell, we do know because it’s runnign fine on dev23:20
jeblairtbh, i would have preferred a simple revert of the puppet change that broke this, followed by a correction so that we could roll into production gracefully23:20
jeblairrather than an emergency change to production because something was broken in dev23:20
jamesmcarthurIt’s either an emergency rollback or an emergency change. Again, this is a simple issue of a bad version# in config.23:21
*** smarcet has quit IRC23:23
jeblairi'm happy to help with true emergencies, but not every error made needs to rise to the level of asking folks to drop what they're doing to perform emergency production changes.23:24
jeblairi've approved that change23:24
jeblairhowever, i will not be around if something goes wrong with it23:24
openstackgerritPetr Malik proposed openstack/requirements: Add xmltodict to global-requirements
anteayajeblair: ack23:25
jeblairi will probably ask harder questions of similar changes in the future23:25
jamesmcarthurjeblair: we’re the only people that know how to update and work on that code. We’re a bit hamstrung by this as we rely on you guys to push through changes for an environment that you don’t normally  deal with.23:26
jamesmcarthurjeblair: I get that it’s unusual situation, and I don’t think it’s really ideal for either party.23:26
*** Daisy has joined #openstack-infra23:28
jamesmcarthurNow we’ve had a 30 minute conversation about why this is or isn’t an emergency. Very similar to what’s been happening with the wiki.23:28
jeblairjamesmcarthur: i'm sorry you feel hamstrung.  the entire line of my questions was directed to try to understand why you thought this merited an emergency change like that rather than being able to fix it yourself via a change to puppet.23:28
jamesmcarthurIt’s an emergency because it needed to be fixed. We either needed you to roll it back, or push the fix.23:29
jamesmcarthurIt’s as simple as that.23:29
jeblairjamesmcarthur: right.  rolling back would have been the right thing to do.23:29
jamesmcarthurWe did fix it ourselves. We just needed it to be merged.23:29
*** Qiming has joined #openstack-infra23:30
jamesmcarthurjeblair: I’m having a hard time understanding your stance. You’re not familiar with the environment, the code, or the change.23:30
jamesmcarthurI don’t understand the pushback.23:30
nibalizerjamesmcarthur: I think he doesn't like the interruption23:31
*** Daisy has quit IRC23:31
jamesmcarthurnibalizer: we would have needed to interrupt him in either case. That’s my point.23:31
jeblairjamesmcarthur: no, the puppet rollback would not need to have been escalated to the group that the roll-forward did23:31
*** Daisy has joined #openstack-infra23:32
jeblairthe puppet rollback could have been accomplished by folks who are more familiar with the systems23:32
jeblairby going with the emergency roll-23:32
*** pradk has quit IRC23:32
*** rossella_s has quit IRC23:32
fungiokay, back and catching up23:32
jeblairforward solution, people who are unfamiliar were called upon to make a decision about something that they are not familiar with, in a crisis mode23:32
jeblairwe're happy to help in crisis situations23:32
*** rossella_s has joined #openstack-infra23:33
jeblairbut we should avoid making them when simpler solutions are available23:33
*** sridhar_ram has quit IRC23:33
jamesmcarthurjeblair: nobody could access OpenStackID, including Track Chairs on the last day of review. That was teh emergency. I appreciate your help, but we needed something immediate and it was a quick fix.23:33
*** sc68cal has quit IRC23:34
jeblair(and no information about the situation was provided in the commit message, it was basically just 'merge this or everything stays broken')23:34
cjokay, this is nuts:23:34
cjso first it downloads the version of python-daemon that is specified23:34
cjthen it downloads one that is outside of the specification23:34
cjthen it errors because the one it uses is not the one it specifies23:34
cjand then it throws an exception.23:34
*** baoli has joined #openstack-infra23:35
cjis this maybe because I'm using pbr and it's broken?23:35
*** flepied has quit IRC23:36
*** Daisy has quit IRC23:36
*** andymaier has joined #openstack-infra23:37
anteayacj: that is a distinct possibility23:38
cjjeblair: so what is this -d option that you speak of?  To which command do I pass it?23:38
jeblaircj: it doesn't exist.  i was saying that adding an option to gerritbot to run in the foreground might be useful for debugging23:39
cjoh, I see!23:39
jpmaxmanGerritbot doesn't update the wiki does it (just random shot in the dark question in case it is related to wiki activity earlier).23:39
jeblairjpmaxman: no; only statusbot23:40
EmilienMfungi: can I continue to migrate our last ubuntu jobs?23:40
jamesmcarthurjeblair: “the puppet rollback could have been accomplished by folks who are more familiar with the systems” — Those of us that are familiar with OpenStackID aren’t able to push anything to production or rollback, AFAIK.23:40
jeblairjamesmcarthur: who merged the puppet change that broke it?23:40
pabelangercj: jeblair could be gerritbot is broken on latest python-daemons too. Like zuul and nodepool where23:40
jeblairpabelanger: yes, the change i linked cj to attempts to correct that23:41
pabelangerAh, missed that backscroll23:41
jamesmcarthurjeblair: smarcet asked fungi to merge it, which he did.23:41
jamesmcarthurAgain, this was a minor config error with an easy fix.23:41
jeblairjamesmcarthur: a larger group reviews puppet changes than does system-config changes23:41
jamesmcarthurIt was explained pretty clearly in IRC. I am still not clear on the pushback.23:41
jamesmcarthurjeblair: but none of them are familiar with this environment23:42
jeblairjamesmcarthur: it wasn't explained _at all_ in the commit message for the change you asked me to approve23:42
jeblairjamesmcarthur: i just approved it because you asked me to23:42
jamesmcarthurjeblair: no, but it was explained to you here in IRC and to all the other folks that brought the error to our attention23:42
*** yamamoto_ has joined #openstack-infra23:42
jeblairjamesmcarthur: you do know that i approved the change a long time ago, right?23:43
jamesmcarthurYes. I do.23:43
jeblairok, that's good :)23:43
pabelanger#success #succes still works23:46
openstackstatuspabelanger: Added success to Success page23:46
*** xyang1 has quit IRC23:46
jeblairjamesmcarthur: i am a very conservative system administrator.  i don't like production to be broken, and when it does break, i like to find the safest way to fix it.  for me, that generally means rolling backward, not forward, and identifying how a production break happened and working to avoid it in the future.23:46
pabelangerminus the typo, success bot is still working23:46
nibalizerpabelanger: we learned to spell in the same city I think23:46
aeng+1 jeblair23:46
jeblairjamesmcarthur: so i just want to figure out how to avoid this in the future23:47
jeblairjamesmcarthur: perhaps the puppet module should switch on versions, or perhaps it's a matter of setting defaults in a certain way to make them more future proof.  both things are done elsewhere in our puppet modules, so there may be some patterns to work from there.23:49
*** jtomasek_ has quit IRC23:50
jpmaxmanjeblair: that's understandable but also you do have to trust the people that are touching this code every single day :D  Just an observation.   Also jamesmcarthur seems legitimately confused about how to do a rollback without interrupting the same people that would need to push a fix.  So maybe that's a good place to start.23:50
*** gordc has quit IRC23:50
*** matrohon has quit IRC23:50
openstackgerritMerged openstack-infra/project-config: Switch puppet-lint to ubuntu-trusty
nibalizerthe following machines are disabled in the emergency file: controller00.hpuseast.ic.openstack.org23:51
jamesmcarthurjeblair: right. I’m all for conservative, but there is nobody that deals with this code more than smarcet23:51
nibalizeri'd say we caan put cacti back into rotation23:51
*** hichihara has joined #openstack-infra23:51
SpamapSianw: I didn't +A because I wanted more opinions to land on it.23:51
jamesmcarthurjeblair: This is the first time since we’ve been operating OpenStackID that it went down. So is it a common thing? No. Is it something we’ll be looking at, absolutely.23:52
*** rockyg has quit IRC23:52
mordredjpmaxman: it's not really about trust or not trust - we're all able to keep working on the same system by proactively communicating about what's going on and being diligent in providing each other with the information we all need to understand what's going on23:52
jamesmcarthurmordred: we did provide all the necessary information. That’s why I’m a bit frustrated here.23:52
nibalizerjamesmcarthur: did these changes go to opensatckid-dev first?23:52
jamesmcarthurWe went directly to IRC when there was a problem and we proposed a fix.23:52
*** smarcet has joined #openstack-infra23:53
jamesmcarthurnibalizer: yes23:53
nibalizerand we still broke the production one?23:53
fungijeblair: jamesmcarthur: granted, when i solo-approved 285475 i should have pressed harder asking whether that was going to be safe in production, but it _did_ at least provide a default value for the parameter it was adding23:53
jamesmcarthurThey were already changed there. There was a minor issue with teh versionID in the config.23:53
fungiand i approved it on the grounds that it was urgent and we were stretched thin on puppet reviewers23:53
fungibut i agree that it's unfortunate for an urgent now-now-now change to result in an equally urgent now-now-now fix23:54
jamesmcarthurfungi: smarcet didn’t ask for an “urgent now-now-now” change, he asked if you could merge.23:55
jamesmcarthurWe don’t have approval to merge, thus the request to fungi in IRC to merge.23:55
jamesmcarthurAnd yes, turns out, it broke something. Not ideal, but it’s not as though things don’t get broken. So we proposed an urgent fix.23:55
*** Swami has quit IRC23:55
*** thorst has joined #openstack-infra23:56
mordredjamesmcarthur: totally. so - for next time - putting in a why is really helpful to people. "Change to dev accidentally half-rolled in to production. Attempting to revert the half-change from production is likely more disruptive than just rolling forward since the change in question has been vetter in dev" - in the commit message allows for everyone involved to get a quick bullet understanding of23:56
mordredwhat's going on, why it's important and why this particular course of action was chosen23:56
jamesmcarthurmordred: 100% fair.23:56
*** ybathia has joined #openstack-infra23:56
openstackgerritColleen Murphy proposed openstack/diskimage-builder: Add --version option to disk-image-create
jamesmcarthurmordred: And agreed on the commit message. We will make it a point to be better23:56
mordredjamesmcarthur: cool. and I honestly just mean it as a "it helps us out to understand what's going on" and not as any sort of finger pointing23:57
*** baoli has quit IRC23:57
openstackgerritColleen Murphy proposed openstack/diskimage-builder: Add --version option to disk-image-create
jamesmcarthurmordred: yes sir, it’s perfectly reasonable23:57
*** thorst_ has joined #openstack-infra23:58
smarcetproblem is , we cant change puppet app.php.erb23:58
fungijamesmcarthur: yep, sorry, i misinterpreted smarcet's "this need to be reviewed asap please" to imply it was urgent and blocking a production deployment deadline23:58
smarcetwithout breaking something23:58
smarcetbc puppet is deployed on both servers23:59
doug-fishAre the issues @ under discussion here? (I can tell *something* broken is under discussion, but not quite sure what)23:59
smarcetthere is any way that production server works with a labelled puppet config ?23:59

Generated by 2.14.0 by Marius Gedminas - find it at!