Monday, 2018-10-29

*** bradm has joined #openstack-infra00:01
*** bobh has joined #openstack-infra00:14
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Native zuulv3 tests  https://review.openstack.org/56677900:32
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Remove legacy playbooks  https://review.openstack.org/61381700:32
*** longkb has joined #openstack-infra00:38
*** snapiri has quit IRC00:44
*** snapiri has joined #openstack-infra00:44
*** EvilienM is now known as EmilienM00:50
*** bobh has quit IRC01:02
*** bobh has joined #openstack-infra01:09
*** carl_cai has joined #openstack-infra01:13
*** zhangfei has joined #openstack-infra01:14
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Native zuulv3 tests  https://review.openstack.org/56677901:50
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Remove legacy playbooks  https://review.openstack.org/61381701:50
*** hongbin has joined #openstack-infra02:24
*** dayou_ has joined #openstack-infra02:44
*** dayou_ has quit IRC02:50
*** fried_rice has quit IRC02:53
*** fried_rice has joined #openstack-infra02:54
*** dhellmann has quit IRC02:55
*** mrsoul has joined #openstack-infra02:55
*** dhellmann has joined #openstack-infra03:01
*** apetrich has quit IRC03:03
*** AJaeger has quit IRC03:05
*** AJaeger has joined #openstack-infra03:05
*** bhavikdbavishi has joined #openstack-infra03:06
*** apetrich has joined #openstack-infra03:09
*** d0ugal has quit IRC03:11
*** charz has joined #openstack-infra03:13
*** truongnh has joined #openstack-infra03:18
*** ramishra has joined #openstack-infra03:32
*** d0ugal has joined #openstack-infra03:39
*** bobh has quit IRC03:48
*** udesale has joined #openstack-infra03:54
*** d0ugal has quit IRC04:00
*** d0ugal has joined #openstack-infra04:11
*** carl_cai has quit IRC04:13
*** d0ugal has quit IRC04:17
*** janki has joined #openstack-infra04:34
*** zzzeek_ has quit IRC04:36
*** hemna_ has quit IRC04:37
*** zzzeek has joined #openstack-infra04:41
*** chkumar|off is now known as chandankumar04:44
*** hemna has joined #openstack-infra04:47
*** hongbin has quit IRC04:52
*** truongnh has quit IRC05:19
*** yamamoto has quit IRC05:33
*** yamamoto has joined #openstack-infra05:33
*** markmcclain has quit IRC05:40
*** adam_g has quit IRC05:41
*** adam_g has joined #openstack-infra05:41
*** apetrich has quit IRC05:42
*** markmcclain has joined #openstack-infra05:42
*** jamesmcarthur has quit IRC05:43
*** apetrich has joined #openstack-infra05:43
openstackgerritMerged openstack/diskimage-builder master: Move several packages to bindep.txt  https://review.openstack.org/57050405:45
openstackgerritMerged openstack/diskimage-builder master: Native zuulv3 tests  https://review.openstack.org/56677905:45
*** Bhujay has joined #openstack-infra05:46
openstackgerritMerged openstack/diskimage-builder master: Remove legacy playbooks  https://review.openstack.org/61381705:46
*** truongnh has joined #openstack-infra06:02
*** ykarel has joined #openstack-infra06:03
*** annp has joined #openstack-infra06:06
*** Bhujay has quit IRC06:15
*** kjackal has joined #openstack-infra06:15
*** ramishra has quit IRC06:18
*** florianf|pto is now known as florianf06:28
*** jbadiapa has joined #openstack-infra06:30
*** ramishra has joined #openstack-infra06:32
*** quiquell|off is now known as quiquell06:51
*** diablo_rojo has joined #openstack-infra07:03
*** diablo_rojo has quit IRC07:04
*** ccamacho has joined #openstack-infra07:07
openstackgerritMerged openstack-infra/zuul master: Use doc/requirements.txt  https://review.openstack.org/61379607:11
*** dpawlik has joined #openstack-infra07:16
*** dpawlik has quit IRC07:20
*** dpawlik has joined #openstack-infra07:25
*** dpawlik has quit IRC07:27
*** dpawlik has joined #openstack-infra07:28
*** aojea has joined #openstack-infra07:29
*** ifat_afek has joined #openstack-infra07:30
*** ykarel is now known as ykarel|lunch07:34
*** noama has joined #openstack-infra07:35
*** shardy has joined #openstack-infra07:38
*** pcaruana has joined #openstack-infra07:46
*** lpetrut has joined #openstack-infra07:49
*** d0ugal has joined #openstack-infra07:49
openstackgerritTobias Henkel proposed openstack-infra/nodepool master: Rate limit updateNodeStats  https://review.openstack.org/61368007:51
*** quiquell is now known as quiquell|brb07:51
*** slaweq has joined #openstack-infra07:53
*** kjackal has quit IRC07:53
*** slaweq has quit IRC07:54
*** d0ugal has quit IRC07:56
*** jtomasek has joined #openstack-infra07:58
*** slaweq has joined #openstack-infra07:59
*** slaweq has quit IRC07:59
*** d0ugal has joined #openstack-infra08:00
*** rossella_s has joined #openstack-infra08:01
*** kjackal has joined #openstack-infra08:05
*** oanson has joined #openstack-infra08:05
*** psachin has joined #openstack-infra08:06
*** arxcruz has joined #openstack-infra08:07
*** ginopc has joined #openstack-infra08:08
*** slaweq has joined #openstack-infra08:11
*** yamamoto has quit IRC08:11
*** AJaeger has quit IRC08:11
*** yamamoto has joined #openstack-infra08:11
*** maciejjozefczyk has joined #openstack-infra08:14
*** AJaeger has joined #openstack-infra08:15
*** quiquell|brb is now known as quiquell08:15
*** wznoinsk has quit IRC08:16
*** sshnaidm is now known as sshnaidm|ruck08:18
*** apetrich has quit IRC08:32
*** d0ugal has quit IRC08:32
*** d0ugal has joined #openstack-infra08:39
*** ykarel|lunch is now known as ykarel08:40
*** xek has joined #openstack-infra08:42
*** e0ne has joined #openstack-infra08:45
*** smarcet has joined #openstack-infra08:46
*** smarcet has quit IRC08:47
*** yamamoto has quit IRC08:48
*** apetrich has joined #openstack-infra08:49
*** jpena|off is now known as jpena08:57
*** hashar has joined #openstack-infra09:01
*** tosky has joined #openstack-infra09:02
*** shardy has quit IRC09:07
*** shardy has joined #openstack-infra09:09
*** ifat_afek has quit IRC09:12
*** ifat_afek has joined #openstack-infra09:13
*** panda|off is now known as panda09:19
*** snapiri has quit IRC09:21
*** yamamoto has joined #openstack-infra09:22
*** d0ugal has quit IRC09:27
*** shardy_ has joined #openstack-infra09:29
*** shardy has quit IRC09:31
*** bhavikdbavishi has quit IRC09:45
*** bhavikdbavishi has joined #openstack-infra09:46
*** shardy_ is now known as shardy09:53
*** d0ugal has joined #openstack-infra09:54
*** slaweq has quit IRC09:56
*** slaweq has joined #openstack-infra09:59
*** longkb has quit IRC10:04
*** shardy has quit IRC10:11
*** ifat_afek has quit IRC10:13
*** sshnaidm|ruck is now known as sshnaidm|afk10:18
*** shardy has joined #openstack-infra10:18
*** truongnh has quit IRC10:21
*** sshnaidm|afk has quit IRC10:23
*** sambetts has joined #openstack-infra10:24
*** slaweq has quit IRC10:25
*** slaweq has joined #openstack-infra10:27
*** bhavikdbavishi has quit IRC10:32
*** carl_cai has joined #openstack-infra10:35
*** dtantsur|afk is now known as dtantsur10:35
*** zhangfei has quit IRC10:36
*** shardy has quit IRC10:41
*** pbourke has quit IRC10:49
*** pbourke has joined #openstack-infra10:49
*** admcleod_away is now known as admcleod10:50
*** ansmith_ has quit IRC11:00
*** yamamoto has quit IRC11:00
*** yamamoto has joined #openstack-infra11:00
*** shardy has joined #openstack-infra11:06
*** shardy_ has joined #openstack-infra11:07
*** rcarrillocruz has joined #openstack-infra11:15
*** rpioso has quit IRC11:15
*** eglute has quit IRC11:15
*** spsurya has quit IRC11:16
*** lsell has quit IRC11:16
*** MasterofJOKers has quit IRC11:16
*** cloudnull has quit IRC11:16
*** zigo has quit IRC11:17
*** vkmc has quit IRC11:17
*** lsell has joined #openstack-infra11:18
*** rpioso has joined #openstack-infra11:18
*** spsurya has joined #openstack-infra11:18
*** vkmc has joined #openstack-infra11:22
*** MasterofJOKers has joined #openstack-infra11:22
*** cloudnull has joined #openstack-infra11:22
*** eglute has joined #openstack-infra11:23
*** ramishra has quit IRC11:24
*** ramishra has joined #openstack-infra11:24
*** e0ne has quit IRC11:25
*** udesale has quit IRC11:26
*** ifat_afek has joined #openstack-infra11:26
*** ifat_afek has quit IRC11:28
*** ifat_afek has joined #openstack-infra11:31
*** jamesdenton has joined #openstack-infra11:32
*** janki has quit IRC11:32
*** e0ne has joined #openstack-infra11:34
dhellmanno/11:39
*** ramishra_ has joined #openstack-infra11:40
AJaegerwelcome back, dhellmann ! Hope you had a great vacation11:42
dhellmannhi, AJaeger! yes, it was very relaxing, thanks11:42
dhellmanntoday's goal is to remember how to use a computer11:43
*** ramishra has quit IRC11:43
AJaegerdhellmann: you mastered steps 1 and 2 with showing up here ;)11:43
*** yamamoto has quit IRC11:44
dhellmannhaha11:44
*** ccamacho has quit IRC11:44
*** ccamacho has joined #openstack-infra11:45
*** sshnaidm has joined #openstack-infra11:51
*** jamesmcarthur has joined #openstack-infra11:54
*** jamesmcarthur has quit IRC11:59
*** kjackal has quit IRC11:59
*** rh-jelabarre has joined #openstack-infra11:59
*** ifat_afek has quit IRC12:00
*** rlandy has joined #openstack-infra12:02
*** ifat_afek has joined #openstack-infra12:05
fungii think they're called "business machines"12:09
AJaegerfungi: you mean dhellmann received his blue hat already? ;)12:10
fungiheh12:10
dhellmannAJaeger : when you take a red hat and blue wash it you get purple, right? my niece will love that.12:10
fungiquite some news to greet you on your return from vacation12:11
dhellmannyeah, that'll teach me to take another vacation12:11
AJaegerdhellmann: Looking forward to the summit to see the new coloured hats ;)12:11
dhellmannheh12:11
fungiyou might want to double-check you didn't accidentally travel to a parallel dimension12:11
dhellmannI was driving in France, so that's entirely possible12:11
AJaeger(I know it will take far longer...)12:11
dhellmannit would explain the wind farm12:12
AJaegerinfra-root, could you review https://review.openstack.org/613509 and https://review.openstack.org/602252 , please?12:12
*** bobh has joined #openstack-infra12:13
*** kjackal has joined #openstack-infra12:17
*** trown|outtypewww is now known as trown12:18
*** boden has joined #openstack-infra12:19
openstackgerritAndreas Jaeger proposed openstack-infra/jeepyb master: Use https for links  https://review.openstack.org/61350912:19
AJaegerfungi, thanks for reviews - I rebased my change on the one you approved due to conflicts ^12:19
*** yamamoto has joined #openstack-infra12:22
dmelladoI just came back from holidays and well and shocked, tbh12:23
dmelladoxD12:23
*** bhavikdbavishi has joined #openstack-infra12:24
openstackgerritMerged openstack-infra/zuul-sphinx master: Use tox-docs directly  https://review.openstack.org/60225212:25
*** _ari_ has joined #openstack-infra12:26
*** gfidente has joined #openstack-infra12:27
openstackgerritMerged openstack-infra/jeepyb master: Remove leading whitespaces from the Gerrit welcome message  https://review.openstack.org/31092612:28
*** edmondsw has quit IRC12:29
*** yamamoto has quit IRC12:29
*** yamamoto has joined #openstack-infra12:29
*** StevenK has quit IRC12:32
*** StevenK has joined #openstack-infra12:32
*** edmondsw has joined #openstack-infra12:36
*** roman_g has joined #openstack-infra12:43
*** carl_cai has quit IRC12:44
*** bhavikdbavishi has quit IRC12:44
*** udesale has joined #openstack-infra12:47
*** zul has joined #openstack-infra12:48
*** ansmith_ has joined #openstack-infra12:48
*** niedbalski has quit IRC12:50
*** bobh has quit IRC12:51
*** rh-jelabarre has quit IRC12:51
*** rh-jelabarre has joined #openstack-infra12:52
*** kjackal has quit IRC12:53
*** kjackal has joined #openstack-infra12:53
*** agopi has quit IRC12:57
*** kgiusti has joined #openstack-infra12:58
*** e0ne has quit IRC12:58
*** yamamoto has quit IRC12:59
*** yamamoto has joined #openstack-infra13:00
*** cburgess has joined #openstack-infra13:01
*** quiquell is now known as quiquell|lunch13:01
*** quiquell|lunch is now known as quiquell13:02
*** tpsilva has joined #openstack-infra13:02
*** jpena is now known as jpena|lunch13:03
*** jamesmcarthur has joined #openstack-infra13:07
*** e0ne has joined #openstack-infra13:07
*** d0ugal has quit IRC13:09
*** shrasool has joined #openstack-infra13:12
*** dpawlik has quit IRC13:12
*** zigo has joined #openstack-infra13:14
*** ifat_afek has quit IRC13:14
*** e0ne has quit IRC13:17
*** e0ne has joined #openstack-infra13:19
*** mriedem has joined #openstack-infra13:19
*** aojea has quit IRC13:22
*** elbragstad is now known as lbragstad13:22
*** e0ne has quit IRC13:24
*** dpawlik has joined #openstack-infra13:26
*** dpawlik has quit IRC13:26
*** bobh has joined #openstack-infra13:27
*** dpawlik has joined #openstack-infra13:27
*** shrasool has quit IRC13:29
*** jaosorior has quit IRC13:30
*** e0ne has joined #openstack-infra13:30
*** agopi has joined #openstack-infra13:30
mordreddmellado: that should teach you the dangers of taking holiday13:32
*** shardy_ has quit IRC13:33
*** shardy has quit IRC13:37
rcarrillocruzinorite13:37
rcarrillocruzjust back from paternity leave13:37
rcarrillocruzlike, WTF13:37
*** sthussey has joined #openstack-infra13:39
*** shardy has joined #openstack-infra13:42
*** shardy_ has joined #openstack-infra13:44
mordredinfra-root: due to circumstances that arose over the weekend, I find my calendar today a bit more full of phone calls than normal, so I might not be around as much as I might usually be13:44
*** niedbalski has joined #openstack-infra13:44
Shrewsshocking13:45
*** smarcet has joined #openstack-infra13:46
*** apetrich has quit IRC13:46
*** xek has quit IRC13:50
*** xek has joined #openstack-infra13:50
fungimordred: thanks for the heads up13:52
*** MasterofJOKers has quit IRC13:52
rcarrillocruzlulz13:54
*** jamesmcarthur has quit IRC13:55
*** jpena|lunch is now known as jpena|mtg13:55
dmelladomordred: totally13:56
dmelladolol, you too rcarrillocruz?13:56
rcarrillocruzyeah dude13:57
rcarrillocruzjust back13:57
rcarrillocruzi've been off 2 weeks13:57
rcarrillocruzi got very surprised when i got your whatsapp at 8am, like...something is happening, is too early to get a message from Dani haha13:57
*** jpich has joined #openstack-infra13:58
*** smarcet has quit IRC13:59
*** fried_rice is now known as efried13:59
*** apetrich has joined #openstack-infra14:00
openstackgerritMerged openstack-infra/jeepyb master: Use https for links  https://review.openstack.org/61350914:02
*** smarcet has joined #openstack-infra14:03
*** jaosorior has joined #openstack-infra14:05
*** aojeagarcia has joined #openstack-infra14:05
*** shrasool has joined #openstack-infra14:09
*** jamesmcarthur has joined #openstack-infra14:11
*** jamesmcarthur has quit IRC14:11
*** jamesmcarthur has joined #openstack-infra14:11
fungigreat day for a cpu fan failure14:15
*** SteelyDan is now known as dansmith14:16
*** sshnaidm is now known as sshnaidm|ruck14:18
*** shrasool has quit IRC14:18
*** ifat_afek has joined #openstack-infra14:20
*** quiquell has quit IRC14:21
*** quiquell has joined #openstack-infra14:21
*** smarcet has quit IRC14:39
*** psachin has quit IRC14:43
mordredfungi: who needs a cpu fan14:47
mordredfungi: if you put a blanket on your legs between you and the laptop, the heat will burn you less14:47
*** dklyle has joined #openstack-infra14:47
clarkbmordred with the protips14:52
*** psachin has joined #openstack-infra14:55
* mordred really needs to fix the cpu fan in his laptop14:56
*** shrasool has joined #openstack-infra14:57
*** jpena|mtg is now known as jpena14:59
*** beekneemech is now known as bnemec15:00
*** MasterofJOKers has joined #openstack-infra15:00
*** cfriesen has joined #openstack-infra15:01
fungii have a drawer full of replacement cpu fans (as well as other spare parts) for my workstation, just a matter of taking everything apart. nearly done now15:01
mordredoh - workstation. you probably don't want to put that on your lap anyway15:02
*** irclogbot_1 has joined #openstack-infra15:03
fungiit's tiny. 4.5"sq x 1" deep, vesa-mounted to the back of my workstand15:04
fungiwould be fine on a lap if not for all the cables15:04
*** psachin has quit IRC15:07
*** jpich has quit IRC15:09
mordredfungi: some might find all the cables to be fashionable15:10
*** dtantsur is now known as dtantsur|brb15:10
*** smarcet has joined #openstack-infra15:15
*** jd_ has quit IRC15:16
*** jd_ has joined #openstack-infra15:16
*** rkukura has joined #openstack-infra15:16
*** ykarel is now known as ykarel|away15:20
*** d0ugal has joined #openstack-infra15:24
*** armax has joined #openstack-infra15:27
quiquellmordred, fungi: don't we have br-ex at fedora nodesets at zuul ?15:29
*** smarcet has quit IRC15:31
*** smarcet has joined #openstack-infra15:33
clarkbquiquell: that isnt a nodeset functionality it is a job functionality. I beliece devstack jobs set that up for neutron15:34
*** dpawlik has quit IRC15:35
clarkbbut by default the multinode role doesnt because it is neutron specific15:35
*** dpawlik has joined #openstack-infra15:35
quiquellclarkb: ok will look in the pre roles15:35
*** dpawlik has quit IRC15:37
*** d0ugal has quit IRC15:37
*** dpawlik has joined #openstack-infra15:38
*** jaosorior has quit IRC15:38
*** bhavikdbavishi has joined #openstack-infra15:38
*** d0ugal has joined #openstack-infra15:38
*** lpetrut has quit IRC15:40
*** gyee has joined #openstack-infra15:40
*** dpawlik has quit IRC15:42
*** ykarel|away has quit IRC15:43
*** haleyb has joined #openstack-infra15:44
clarkbI'm going to be juggling a three year old this morning for a bit so other family members can go visit the dentist. I expect I'll poke at improving my zuul usage number script to break down which jobs use the most time (where we could put effrot to speed things up with the biggest impact), and make percentage of usage autoamtically calculated. I'm open to other thoughts on how to make that data useful15:52
clarkbAJaeger: also I'll review those changes now15:52
clarkband they both merged the best way to start a morning :)15:52
*** dpawlik has joined #openstack-infra15:54
*** dpawlik has quit IRC15:58
clarkboh I should have it automatically break down usage using the governance repo proejcts list15:58
clarkbthat makes it more oepnstack specific but meh15:58
openstackgerritMathieu GagnĆ© proposed openstack-infra/project-config master: Revert "Disable inap-mtl01 provider"  https://review.openstack.org/61397015:59
*** d0ugal has quit IRC16:02
*** yamamoto has quit IRC16:04
*** yamamoto has joined #openstack-infra16:05
*** yamamoto has quit IRC16:05
*** yamamoto has joined #openstack-infra16:10
*** sshnaidm|ruck is now known as sshnaidm|bbl16:13
*** zul has quit IRC16:14
*** yamamoto has quit IRC16:14
*** shrasool has quit IRC16:15
*** quiquell is now known as quiquell|off16:15
*** smarcet has quit IRC16:15
*** dtantsur|brb is now known as dtantsur16:16
*** finucannot is now known as stephenfin16:17
*** smarcet has joined #openstack-infra16:17
*** tosky has quit IRC16:18
*** eharney has joined #openstack-infra16:20
*** ykarel|away has joined #openstack-infra16:23
*** shrasool has joined #openstack-infra16:24
*** beagles is now known as beagles_brb16:29
*** ansmith_ has quit IRC16:29
*** ansmith has joined #openstack-infra16:30
openstackgerritAlex Schultz proposed openstack-infra/project-config master: Add noop to instack  https://review.openstack.org/61397916:33
*** smarcet has quit IRC16:33
*** smarcet has joined #openstack-infra16:34
*** bhavikdbavishi has quit IRC16:34
*** bhavikdbavishi has joined #openstack-infra16:35
*** e0ne has quit IRC16:36
*** shrasool has quit IRC16:46
*** sambetts is now known as sambetts|afk16:46
*** trown is now known as trown|lunch16:48
*** shrasool has joined #openstack-infra16:50
*** yamamoto has joined #openstack-infra16:52
*** jcoufal has joined #openstack-infra16:55
*** kopecmartin is now known as kopecmartin|off16:55
*** udesale has quit IRC17:00
*** kjackal has quit IRC17:00
*** kjackal_v2 has joined #openstack-infra17:01
AJaegermgagne: is inap-mtl01 ready ? That's great, thanks!17:01
*** zul has joined #openstack-infra17:01
AJaegerinfra-root, want to +2A https://review.openstack.org/#/c/613970/ and move to gate so that we get 190 nodes in addition from inap-mtl01, please?17:01
*** d0ugal has joined #openstack-infra17:04
*** eernst has joined #openstack-infra17:08
*** aojeagarcia has quit IRC17:09
*** d0ugal has quit IRC17:12
*** shrasool has quit IRC17:12
*** markvoelker has joined #openstack-infra17:12
*** eernst has quit IRC17:16
*** harlowja has quit IRC17:17
*** eernst has joined #openstack-infra17:17
mgagneAJaeger: yes17:19
*** harlowja has joined #openstack-infra17:21
clarkbAJaeger: looking17:23
clarkbalso thank you mgagne !17:23
prometheanfirefungi: did you say that the listserv stuff was updated to allow for dkim? (a couple weeks ago)17:23
fungimordred: the starlingx crowd are looking at using netlify cms for their main starlingx.io website... i remember you mentioning possibly using it for zuul-ci.org as well. how were you expecting to have it interoperate with gerrit? or had you gotten that far?17:23
mgagneclarkb: lets see how it goes17:23
fungiprometheanfire: the new openstack-discuss list will be, by way of turning off any message body and subject mangling17:24
prometheanfirefungi: righ, that's the one I was wondering about (about to sub to it with my non-gentoo (main) email)17:25
openstackgerritBenoĆ®t Bayszczak proposed openstack-infra/zuul master: upgrade minimum Ansible version to 2.5.2  https://review.openstack.org/61399917:27
clarkbI've refined the usage data scraping a bit http://paste.openstack.org/show/733572/17:28
clarkbI'll update my change to zuul too in a moment17:28
clarkbthis now scrapes all the available zuul.log files too17:29
clarkbso a month from last week we'll be able to do monthly snapshots17:29
fungiexcellent!17:33
fungithanks for working on that, it's going to come in really handy17:33
clarkbas soon as pep8 passes I'll push the latest script version too :)17:34
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool master: Cleanup down ports  https://review.openstack.org/60982917:35
mordredfungi: yes - we spoke about doing work to integrate it with gerrit - we so far haven't gotten that work done17:35
mordredfungi: we spoke with the netlify-cms team and theyre' open/receptive to it17:35
*** shardy_ has quit IRC17:36
mordredfungi: I'd love to collaborate on the work needed to gerrit-enable netlify-cms17:36
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool master: Cleanup down ports  https://review.openstack.org/60982917:36
AJaegerfungi, clarkb, do you want to enqueue https://review.openstack.org/#/c/613970/ directly into gate pipeline? Otherwise we have to wait at least another hour...17:38
clarkbAJaeger: ++ I can do that as soon as I push up this change17:39
openstackgerritClark Boylan proposed openstack-infra/zuul master: Small script to scrape Zuul job node usage  https://review.openstack.org/61367417:39
clarkbthat is what I used to generate the paste above17:40
*** shardy has quit IRC17:40
clarkbok enqueuing 613970 now17:40
*** trown|lunch is now known as trown17:41
*** jamesmcarthur has quit IRC17:41
*** ifat_afek has quit IRC17:41
mordredfungi: basically, we need to write a gerrit driver for netlify cms - which both corvus and I have indicated interest in actually working on - but of course having time is tricky. we could also just sit down with someone who does have time to spec out what is needed17:41
*** bnemec has quit IRC17:42
clarkbenqueue is slow, I think that means zuul is rebuilding configs right now17:42
* clarkb tries to be patient17:42
clarkbthere it goes17:42
clarkbAJaeger: done17:42
*** jamesmcarthur has joined #openstack-infra17:42
AJaegerclarkb: I think there are some more tripleo jobs - they have added jobs to other repos e.g. barbican, see http://git.openstack.org/cgit/openstack/barbican/tree/.zuul.yaml#n183 (same is true for other projects as well)17:43
AJaegerclarkb: thanks17:44
*** beagles_brb is now known as beagles17:44
*** jamesmcarthur has quit IRC17:44
clarkbAJaeger: ya that is why I break it down by project, repo, and job because jobs can apply to many repos and projects17:44
*** jamesmcarthur has joined #openstack-infra17:44
AJaegerok17:44
clarkbif say 50% of the job cost was running unittetss we'd know improving that job to be faster would help globally17:44
clarkbbut it isn't, they are tripleo specific jobs which may run outside of tripleo, but the cost is high to tripleo directly17:45
*** noama has quit IRC17:45
fungimordred: thanks, yeah it sounds like they're going to set up a github repo to act as their netlify backend for now17:45
AJaegerclarkb: I see17:45
mordredfungi: that's "awesome"17:47
fungiindeed :/17:47
AJaegerclarkb: want to give out the time range you used for scraping in your script? meaning from 28th to 29th or whatever it is17:47
clarkbAJaeger: its from when we restarted Zuul on thursday. That is when we started logging this data17:48
clarkbAJaeger: we have 30 day log retention so once we get 30 days past that restart it will be a 30 day snapshot17:48
clarkband then output was generated about half an hour ago17:49
AJaegerclarkb: the output of the script could contain: Oldest data, newest data in it so that we can easily compare that.17:49
clarkbAJaeger: thats a good idea17:49
*** bnemec has joined #openstack-infra17:50
*** shrasool has joined #openstack-infra17:50
clarkbAJaeger: I'll work on that after some breakfast/lunch17:50
*** diablo_rojo has joined #openstack-infra17:51
*** yamamoto has quit IRC17:51
*** yamamoto has joined #openstack-infra17:51
*** lpetrut has joined #openstack-infra17:54
*** jamesmcarthur has quit IRC17:55
openstackgerritMerged openstack-infra/project-config master: Revert "Disable inap-mtl01 provider"  https://review.openstack.org/61397017:55
*** eharney has quit IRC17:55
clarkbmgagne: ^ fyi17:55
mgagne+117:55
*** shrasool has quit IRC17:59
*** eharney has joined #openstack-infra18:05
*** jpena is now known as jpena|off18:07
*** jamesmcarthur has joined #openstack-infra18:08
*** yamamoto has quit IRC18:15
*** yamamoto has joined #openstack-infra18:15
*** yamamoto has quit IRC18:16
*** ykarel|away has quit IRC18:17
*** smarcet has quit IRC18:27
*** smarcet has joined #openstack-infra18:28
*** rossella_s has quit IRC18:32
openstackgerritClark Boylan proposed openstack-infra/zuul master: Small script to scrape Zuul job node usage  https://review.openstack.org/61367418:33
clarkbAJaeger: ^ done http://paste.openstack.org/show/733577/ is output18:33
AJaegerclarkb: nice, thanks!18:34
mordredclarkb: woot18:37
AJaegerclarkb: how long does it take until 613970 is live? It merged over 40 mins ago and http://grafana.openstack.org/d/ykvSNcImk/nodepool-inap?orgId=1 shows no images booting yet - is that ok?18:39
clarkbAJaeger: let me check the ansible puppet cron18:39
clarkboh it may still be in the emergency list :/ I'll fix18:39
clarkbya that was it18:40
AJaegerthanks, clarkb !18:40
*** jamesmcarthur has quit IRC18:41
clarkbAJaeger: mgagne I removed nl03 from the emergency file and changed max-servers to 190 to speed things up18:41
*** jamesmcarthur has joined #openstack-infra18:41
AJaegerclarkb: first 0.3 images are building ;)18:41
*** jamesmcarthur_ has joined #openstack-infra18:42
roman_g0.3 :)18:42
*** jamesmcarthur has quit IRC18:46
*** graphene has joined #openstack-infra18:51
*** robcresswell has joined #openstack-infra18:51
*** yamamoto has joined #openstack-infra18:51
*** jamesmcarthur_ has quit IRC18:51
*** jamesmcarthur has joined #openstack-infra18:52
*** jaosorior has joined #openstack-infra18:52
mordredAJaeger: when you made the zuul patch to use r'' - did you use a script, or just by hand?18:52
clarkbif anyone is wondering gzip.open() and open() have different default open modes on python318:53
clarkbthis seems like it should be a bug18:53
clarkbgzip.open is 'rb' and open is 'rt'18:53
AJaegermordred: by hand - why?18:54
AJaegermordred: I updated to flake 3.6.0 and fixed all problems reported...18:55
fungiclarkb: yeah, i honestly don't know why they did it that way. i've found it quite annoying18:55
AJaeger(and sometimes went through by hand quickly looking for similar pattern)18:55
mordredAJaeger: I was doing the same in the sdk repo - and was wondering if you'd written a magical script to do it I could also use18:55
mordredAJaeger: :)18:55
AJaegermordred: sorry, no ;(18:55
*** bhavikdbavishi has quit IRC18:55
mordredAJaeger: no worries - I often just assume you have magic somewhere18:56
AJaeger;)18:57
AJaegerI wonder why we build and delete nodes in inap - but not run them - see http://grafana.openstack.org/d/ykvSNcImk/nodepool-inap?orgId=1 .18:57
clarkbAJaeger: mgagne it could be the new image timeouts again. I'll look and see if logs say anything18:58
clarkbyup timeouts waiting for node to come up18:58
clarkbopenstack.exceptions.ResourceTimeout: Timeout waiting for the server to come up.18:59
*** kjackal_v2 has quit IRC19:04
*** kjackal has joined #openstack-infra19:04
*** smarcet has quit IRC19:11
*** dtantsur is now known as dtantsur|afk19:11
*** scas has quit IRC19:11
*** graphene has quit IRC19:13
*** graphene has joined #openstack-infra19:15
*** diablo_rojo has quit IRC19:16
*** lpetrut has quit IRC19:18
*** diablo_rojo has joined #openstack-infra19:20
openstackgerritTobias Henkel proposed openstack-infra/zuul-jobs master: Add prepare-workspace-git role  https://review.openstack.org/61303619:22
*** diablo_rojo has quit IRC19:22
*** shardy has joined #openstack-infra19:22
*** diablo_rojo has joined #openstack-infra19:23
*** eernst has quit IRC19:23
openstackgerritChris Dent proposed openstack-infra/project-config master: Add publish to pypi jobs for gabbi-tempest  https://review.openstack.org/61401419:26
*** rossella_s has joined #openstack-infra19:27
*** Tengu has quit IRC19:28
*** shardy has quit IRC19:28
*** Tengu has joined #openstack-infra19:29
*** eharney has quit IRC19:32
*** yamamoto has quit IRC19:32
*** yamamoto has joined #openstack-infra19:32
openstackgerritChris Dent proposed openstack-infra/infra-manual master: Add info on needing publish-to-pypi template  https://review.openstack.org/61401719:42
clarkbmgagne: AJaeger the inap grafana grpah doesn't look happy. Should we do anything with it?19:44
AJaegerclarkb: at least we now started launching nodes - first 4 ones are in use. But the 258 deleted look odd19:45
*** graphene has quit IRC19:46
AJaegerand nothing new building - yeah, looks unhappy19:47
fungithundering herd populating the image cache on all the nova hosts?19:48
clarkbfungi: ya that is possible19:48
clarkbquestion is whether it can recover19:48
AJaegervery slowly deleting nodes - now 25619:49
AJaegerso, do we need to wait another hour or two?19:49
clarkbAJaeger: I'm willing to wait if mgagne is19:49
AJaegerclarkb: now at 239 - let's wait a bit19:50
*** mrhillsman is now known as mrhillsman_lunch19:53
*** ramishra_ has quit IRC20:01
*** lbragstad has quit IRC20:03
clarkbin use keeps going up as deleting falling20:11
*** lbragstad has joined #openstack-infra20:18
*** kgiusti has left #openstack-infra20:21
fungisounds more and more likely it was indeed an image cache thundering herd in that case20:23
clarkbit continues to trend the right direction. I'll keep an eye on it20:24
fungii wonder if a nodepool slow-start feature would make sense... some parameter that says how rapidly to phase in max-servers increases20:25
fungiso that we don't slam providers all in one go20:25
*** jamesmcarthur has quit IRC20:25
anteayahow practical20:26
mgagneclarkb: still stuck in deleting?20:27
clarkbmgagne: no things are deleting just a little slowly, also instances are booting and in use20:28
clarkbmgagne: I think it was a thundering herd image problem that we seem to be getting over now20:28
*** dpawlik has joined #openstack-infra20:28
mgagneok, at least it's not affecting other systems, it should now be isolated from a network bw perspective20:28
*** jamesmcarthur has joined #openstack-infra20:30
clarkbmgagne: ok, feel free to ping me if you need anything from your side I should be around the rest of the day20:30
mgagnealright, I'll let nodepool do its thing unless our netadmin pings me.20:31
*** imacdonn has quit IRC20:33
*** imacdonn has joined #openstack-infra20:33
*** eharney has joined #openstack-infra20:34
*** jcoufal has quit IRC20:37
*** ansmith has quit IRC20:39
*** smarcet has joined #openstack-infra20:43
*** rossella_s has quit IRC20:43
*** eharney has quit IRC20:48
*** eernst has joined #openstack-infra20:52
*** jamesmcarthur has quit IRC20:55
*** jamesmcarthur has joined #openstack-infra20:56
*** trown is now known as trown|outtypewww20:58
*** robcresswell has quit IRC21:00
*** eernst has quit IRC21:04
*** mrhillsman_lunch is now known as mrhillsman21:04
*** eernst has joined #openstack-infra21:04
*** bobh has quit IRC21:08
*** bobh has joined #openstack-infra21:10
*** hashar has quit IRC21:12
*** bobh has quit IRC21:15
*** gfidente has quit IRC21:16
*** shrasool has joined #openstack-infra21:17
*** dpawlik has quit IRC21:22
*** dpawlik has joined #openstack-infra21:23
clarkbinap graph looks normal the last 10 minutes21:23
mgagne:D21:23
*** dpawlik_ has joined #openstack-infra21:26
*** dpawlik_ has quit IRC21:26
*** dpawlik has quit IRC21:27
*** e0ne has joined #openstack-infra21:29
*** jamesmcarthur has quit IRC21:32
*** e0ne has quit IRC21:36
*** dpawlik has joined #openstack-infra21:42
*** mriedem is now known as mriedem_away21:42
*** xek has quit IRC21:46
*** dpawlik has quit IRC21:46
*** bobh has joined #openstack-infra21:49
*** smarcet has quit IRC21:51
clarkbinfra-root I'd like to delete nodepool.openstack.org tomorrow. If you have any files/scripts/data on that server or otherwise know of a reason I should not do this please speak up21:52
clarkbthe zk cluster looks happy from where I am looking though21:52
*** bobh has quit IRC21:55
*** pcaruana has quit IRC21:57
*** bobh has joined #openstack-infra21:57
*** shrasool has quit IRC22:01
*** rh-jelabarre has quit IRC22:07
*** bgmccollum has quit IRC22:09
*** boden has quit IRC22:13
*** bobh has quit IRC22:16
openstackgerritAlex Schultz proposed openstack-infra/project-config master: Add noop to instack-undercloud  https://review.openstack.org/61363022:17
*** shrasool has joined #openstack-infra22:17
openstackgerritAlex Schultz proposed openstack-infra/project-config master: Add noop to instack  https://review.openstack.org/61397922:17
clarkbmwhahaha: re ^ as a note the master branch will stick around because gerrit has a sad when the repo HEAD changes22:18
clarkbmwhahaha: but we can totally stop treating it as a development point22:18
mwhahahaclarkb: yea that's what i'm expecting22:19
*** smarcet has joined #openstack-infra22:30
*** eernst has quit IRC22:44
*** smarcet has quit IRC22:56
*** tpsilva has quit IRC22:58
*** shrasool has quit IRC23:03
anticwfollowing up on a line of conversation from last week ... the only thing than can cause a merge for most openstack projects is zuul?23:14
*** kjackal has quit IRC23:14
anticw3rd party CICD don't get permission to the gerrit merge action?23:15
clarkbanticw: correct23:15
clarkbzuul is the only account that has the submit permission in gerrit23:15
anticwthe zuul queue (from my last read of somehting fungi pointed me at i think) is global ... there is no way to 'fast track' something that you think zuul can process in seconds/minutes?23:19
clarkbanticw: sort of. Small "safe" tasks that can be run on the zuul executors can execute there directly. They do no wait for nodes in the queue, they just run23:20
clarkbanticw: anything that isn't safe to run on the executor needs to run on provisioned nodes as part of the queuing23:20
clarkbanticw: http://lists.zuul-ci.org/pipermail/zuul-discuss/2018-October/000575.html is a proposal to change how prioritization of that queuing is done23:21
clarkbanticw: to attempt to be more fair. If you are interested your input on that thread would probably be useful23:21
anticwclarkb: yeah, that was the thread23:22
anticwclarkb: what defines a 'Small "safe"' task?23:22
clarkbanticw: basically anything that can run in a confined directory on the executor (no access to external files) that doesn't run arbitrary code on the executor. So making http requests off host are safe as they can be done with the ansible module that does that.23:23
clarkbyou can do other manipulations as long as they are encoded in ansible iirc and don't have a way to access external data23:24
clarkber external on host data (external off host is ok)23:24
clarkbanticw: I believe you can also add host then ssh into another machine using the per project ssh keys23:25
clarkbthen run remote ansible that way (it is something theoretically doable that the infra team is looking at for CD deployments of ifnra itself via zuul, we just haven't gotten it working yet due to things external to zuul)23:25
anticwclarkb: oh, so ansible somehow knows what's safe and what isn't?23:26
clarkbanticw: ya, zuul has patched out the unsafe bits of ansible and will prevent them from running on the executor23:27
anticwclarkb: i'm asking going back to the idea of running some (not all) jobs elsewhere ... could this 'safe executor' block for 20 mins waiting to see if a remote job was processed successfully?23:27
clarkbliek shell or command are both not allowed iirc23:27
clarkbanticw: my first impression of that is we should avoid doing that because that prevents zuul from doing other work and running other jobs which will contribute to the existing problem23:28
anticwyeah, prob means 20 minute delay (loop) waiting on http isn't gonna work either ...23:28
clarkb(zuul executors self throttle)23:28
anticwclarkb: right, wasn't sure of the concurrency model, i can imagine some cases where you just create 1000s or millions of such jobs but that sounds like it isn't the case23:28
*** smarcet has joined #openstack-infra23:33
*** sthussey has quit IRC23:34
*** eernst has joined #openstack-infra23:53
*** smarcet has quit IRC23:56
*** rlandy has quit IRC23:56

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!