| clarkb | and it is doing stuff | 00:00 |
|---|---|---|
| dims | thanks mordred | 00:00 |
| clarkb | dims: you ready for it to merge? | 00:00 |
| dims | yep clarkb | 00:00 |
| dims | all the depends-on stuff is merged | 00:00 |
| *** thorst has joined #openstack-infra | 00:01 | |
| clarkb | there it goes | 00:01 |
| ianw | fungi: did you update the citycloud mirrors? not all regions have a 4C-8GB-200GB flavor | 00:02 |
| dims | clarkb : cross my fingers :) | 00:02 |
| clarkb | pabelanger: mordred ya doesn't look like the puppet agent thing is fatal. Will have to look and see what the puppet state is once I can get in | 00:02 |
| dims | will check back in a bit | 00:02 |
| *** thorst has quit IRC | 00:02 | |
| *** aeng has joined #openstack-infra | 00:02 | |
| pabelanger | clarkb: ya, that matches what I remember | 00:02 |
| pabelanger | I cannot find my python-minimal patch, means I didn't push it up | 00:03 |
| clarkb | pabelanger: for the python thing I think we want to have just before setup_pip somethign that says "if is_ubuntu and which python is empty then install python" | 00:03 |
| pabelanger | clarkb: sure | 00:03 |
| clarkb | ianw: also don't forget to accept the new hosts' ssh keys on the puppet master | 00:03 |
| *** rlandy has quit IRC | 00:04 | |
| clarkb | there is no puppet agent running, that may be due to default install of puppet on xenial then we install and configure it the way we want and reboot undoing that olld agent project | 00:04 |
| clarkb | s/project/process/ | 00:04 |
| clarkb | ansible is slow to detect the reboot happened | 00:05 |
| clarkb | once that is done I will update dns including dns for the main mirror name | 00:06 |
| *** dingyichen has joined #openstack-infra | 00:06 | |
| clarkb | then I will likely start chocolate and go make dinner | 00:06 |
| pabelanger | kk | 00:07 |
| clarkb | oh its not ansible doing the reboot, its just launch-node and that is being slow to return? weird | 00:08 |
| clarkb | I guess ssh has to timeout? | 00:08 |
| ianw | ok, should i try swapping mirror.iad.rax.openstack.org -> mirror01 and see how it picks up? | 00:09 |
| clarkb | ianw: I think so | 00:09 |
| clarkb | I wonder if hpcloud firewalls are killing some bit of icmp I need for this to happen quickly | 00:10 |
| clarkb | and if that will cause an eventual timeout and failure | 00:10 |
| clarkb | s/hpcloud/hpe/ | 00:10 |
| *** gildub has joined #openstack-infra | 00:11 | |
| clarkb | maybe this will never return :/ | 00:17 |
| clarkb | strace says it is polling a fd in a loop it must be that one | 00:21 |
| *** sree has joined #openstack-infra | 00:22 | |
| ianw | ok, mirror.iad.rax.openstack.org now a cname to mirror01, a couple of hits coming through | 00:22 |
| ianw | i've put the old host at mirror-old.iad.rax just in case for now | 00:22 |
| clarkb | I think we may want `shutdown -r +10s` so that ssh can cleanly exit | 00:23 |
| clarkb | I'm going to ^C which I expect to cause my node to be deleted then rebuild using ^ and will push that in a patch if it fixes | 00:23 |
| clarkb | ^C not changing anything so this may need a bit more firepower | 00:25 |
| *** sree has quit IRC | 00:26 | |
| clarkb | ok infracloud vanilla building again | 00:27 |
| ianw | ok, mirror01.iad.rax getting busy on both proxy & afs serving, looking good | 00:36 |
| *** gildub has quit IRC | 00:39 | |
| *** iyamahat has quit IRC | 00:40 | |
| clarkb | | Aug 17 00:36:53 mirror01 openafs-client-precheck[20722]: Failed to load openafs.ko. Does it need to be built? | 00:41 |
| clarkb | then later | Aug 17 00:36:53 mirror01 systemd[1]: Started OpenAFS client. | 00:41 |
| clarkb | so maybe that is a non issue | 00:41 |
| ianw | yeah, it builds during install afaik | 00:41 |
| *** bobh has joined #openstack-infra | 00:47 | |
| *** iyamahat has joined #openstack-infra | 00:48 | |
| clarkb | arg shtudown only takes incrememnts in minutes | 00:48 |
| clarkb | so +10s is a syntax error | 00:48 |
| *** LindaWang has joined #openstack-infra | 00:52 | |
| *** iyamahat has quit IRC | 00:52 | |
| *** cshastri has joined #openstack-infra | 00:52 | |
| *** EricGonczer_ has joined #openstack-infra | 00:56 | |
| *** EricGonczer_ has quit IRC | 01:02 | |
| *** thorst has joined #openstack-infra | 01:03 | |
| *** shu-mutou-AWAY has quit IRC | 01:03 | |
| *** thorst has quit IRC | 01:08 | |
| *** sflanigan has quit IRC | 01:11 | |
| *** gildub has joined #openstack-infra | 01:14 | |
| ianw | ok i think that's the raw details of everything in https://etherpad.openstack.org/p/mirror-xenial | 01:16 |
| ianw | i'll start on the other two rax regions since that's going well | 01:16 |
| clarkb | ianw: ok mirror01.regionone.infracloud-vanilla.openstack.org is up at 15.184.65.64 | 01:16 |
| clarkb | I don't think I will update dns yet as I need to cook dinner | 01:16 |
| clarkb | feel free to update dns if you like or I'll get it in the morning | 01:17 |
| *** sflanigan has joined #openstack-infra | 01:17 | |
| *** sflanigan has joined #openstack-infra | 01:17 | |
| *** Sukhdev has quit IRC | 01:18 | |
| ianw | ok, see how i go after my lunch :) | 01:19 |
| *** esberglu has joined #openstack-infra | 01:20 | |
| *** iyamahat has joined #openstack-infra | 01:20 | |
| *** Apoorva_ has joined #openstack-infra | 01:23 | |
| *** liujiong has joined #openstack-infra | 01:26 | |
| *** Apoorva has quit IRC | 01:26 | |
| *** Apoorva_ has quit IRC | 01:27 | |
| *** spzala has joined #openstack-infra | 01:28 | |
| *** calebb has quit IRC | 01:29 | |
| *** liujiong has quit IRC | 01:32 | |
| ianw | clarkb: yeah i'm seeing that puppet agent thing now too. i think when i brought it up first, it wasn't matching in sites.pp so not doing a full puppeting of openstack::server or whatever it is | 01:35 |
| *** thorst has joined #openstack-infra | 01:38 | |
| *** calebb has joined #openstack-infra | 01:38 | |
| *** thorst has quit IRC | 01:39 | |
| openstackgerrit | Eric Kao proposed openstack-infra/project-config master: enable qos in congress dsvm job https://review.openstack.org/494350 | 01:40 |
| *** jamesden_ is now known as jamesdenton | 01:41 | |
| *** claudiub has quit IRC | 01:42 | |
| *** thorst has joined #openstack-infra | 01:42 | |
| *** felipemonteiro has joined #openstack-infra | 01:47 | |
| *** esberglu has quit IRC | 01:52 | |
| *** liujiong has joined #openstack-infra | 01:53 | |
| clarkb | ya I think it self corrects once puppet runs | 01:53 |
| *** r-daneel_ has joined #openstack-infra | 01:53 | |
| *** mat128 has joined #openstack-infra | 01:53 | |
| *** r-daneel has quit IRC | 01:53 | |
| *** r-daneel_ is now known as r-daneel | 01:53 | |
| *** ramishra has joined #openstack-infra | 01:59 | |
| *** iyamahat has quit IRC | 01:59 | |
| *** gcb has joined #openstack-infra | 02:07 | |
| *** bobh has quit IRC | 02:14 | |
| *** cuongnv has joined #openstack-infra | 02:15 | |
| *** r-daneel has quit IRC | 02:17 | |
| *** r-daneel has joined #openstack-infra | 02:17 | |
| *** zhurong has joined #openstack-infra | 02:19 | |
| *** thorst has quit IRC | 02:20 | |
| *** liujiong has quit IRC | 02:24 | |
| *** felipemonteiro has quit IRC | 02:28 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: Add configure-logserver role https://review.openstack.org/489113 | 02:31 |
| *** bobh has joined #openstack-infra | 02:32 | |
| *** EricGonczer_ has joined #openstack-infra | 02:33 | |
| EmilienM | clarkb: hey, any chance to get https://review.openstack.org/#/c/494334/ please? | 02:51 |
| openstackgerrit | Merged openstack-infra/devstack-gate master: Update grenade settings for stable/pike https://review.openstack.org/493057 | 02:54 |
| *** spzala has quit IRC | 02:56 | |
| *** spzala has joined #openstack-infra | 02:56 | |
| *** EricGonczer_ has quit IRC | 02:59 | |
| *** felipemonteiro has joined #openstack-infra | 03:01 | |
| *** mat128 has quit IRC | 03:03 | |
| *** spzala has quit IRC | 03:06 | |
| clarkb | looks like fungi just got it | 03:07 |
| openstackgerrit | Merged openstack-infra/project-config master: TEMP - Disable voting on tripleo upgrade jobs https://review.openstack.org/494334 | 03:13 |
| *** Sukhdev has joined #openstack-infra | 03:19 | |
| *** thorst has joined #openstack-infra | 03:20 | |
| *** gouthamr has quit IRC | 03:24 | |
| *** thorst has quit IRC | 03:25 | |
| *** iyamahat has joined #openstack-infra | 03:28 | |
| *** gildub has quit IRC | 03:32 | |
| *** rtjure has quit IRC | 03:34 | |
| *** liujiong has joined #openstack-infra | 03:37 | |
| *** rtjure has joined #openstack-infra | 03:39 | |
| *** yamahata has quit IRC | 03:39 | |
| *** iyamahat has quit IRC | 03:40 | |
| *** nicolasbock has joined #openstack-infra | 03:46 | |
| *** ykarel|afk has joined #openstack-infra | 03:51 | |
| *** ykarel|afk is now known as ykarel | 03:53 | |
| *** ramishra has quit IRC | 03:54 | |
| *** ramishra has joined #openstack-infra | 03:56 | |
| *** Sukhdev has quit IRC | 03:57 | |
| *** bobh has quit IRC | 04:00 | |
| *** links has joined #openstack-infra | 04:01 | |
| *** spzala has joined #openstack-infra | 04:01 | |
| *** dave-mcc_ has quit IRC | 04:02 | |
| *** gildub has joined #openstack-infra | 04:04 | |
| *** spzala has quit IRC | 04:06 | |
| *** felipemonteiro has quit IRC | 04:08 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Move RAX mirrors to numbered hosts https://review.openstack.org/494370 | 04:15 |
| *** udesale has joined #openstack-infra | 04:15 | |
| *** masber has joined #openstack-infra | 04:20 | |
| ianw | #status log all RAX mirror hosts (iad, ord and dfw) migrated to new Xenial based hosts | 04:21 |
| openstackstatus | ianw: finished logging | 04:21 |
| *** thorst has joined #openstack-infra | 04:22 | |
| *** dhajare has joined #openstack-infra | 04:25 | |
| *** thorst has quit IRC | 04:26 | |
| *** yamahata has joined #openstack-infra | 04:27 | |
| *** aeng has quit IRC | 04:35 | |
| ianw | ok, not going to take on more upgrades today, i think we should monitor the rax ones for a bit anyway and see where we're at for a few hours anyway. see https://etherpad.openstack.org/p/mirror-xenial | 04:40 |
| *** openstack has quit IRC | 04:42 | |
| *** openstack has joined #openstack-infra | 04:44 | |
| *** johnthetubaguy has joined #openstack-infra | 04:44 | |
| *** vponomaryov has joined #openstack-infra | 04:44 | |
| *** peterlisak has joined #openstack-infra | 04:44 | |
| *** Diabelko has joined #openstack-infra | 04:44 | |
| *** krtaylor has joined #openstack-infra | 04:44 | |
| *** MasterofJOKers_ is now known as MasterofJOKers | 04:44 | |
| *** SergeyLukjanov has joined #openstack-infra | 04:44 | |
| *** EmilienM has joined #openstack-infra | 04:44 | |
| *** zz_ja has joined #openstack-infra | 04:44 | |
| *** kjackal has joined #openstack-infra | 04:44 | |
| *** _ari_ has joined #openstack-infra | 04:44 | |
| *** jlvillal has joined #openstack-infra | 04:44 | |
| *** sflanigan has quit IRC | 04:44 | |
| *** sflanigan has joined #openstack-infra | 04:44 | |
| *** niedbalski has joined #openstack-infra | 04:44 | |
| *** lennyb has joined #openstack-infra | 04:44 | |
| *** lastmikoi has joined #openstack-infra | 04:44 | |
| *** jrist has quit IRC | 04:44 | |
| *** jrist has joined #openstack-infra | 04:44 | |
| *** markvoelker has joined #openstack-infra | 04:44 | |
| *** dimak has joined #openstack-infra | 04:44 | |
| *** mudpuppy has joined #openstack-infra | 04:44 | |
| *** rvba has quit IRC | 04:45 | |
| *** rvba has joined #openstack-infra | 04:45 | |
| *** HeOS has joined #openstack-infra | 04:45 | |
| *** ekhugen has joined #openstack-infra | 04:45 | |
| *** quite has joined #openstack-infra | 04:45 | |
| *** rook is now known as Guest88899 | 04:45 | |
| *** tojuvone has joined #openstack-infra | 04:45 | |
| *** mrodden has joined #openstack-infra | 04:45 | |
| *** openstackstatus has joined #openstack-infra | 04:45 | |
| *** gaelL has joined #openstack-infra | 04:45 | |
| *** jklare has joined #openstack-infra | 04:45 | |
| *** oanson has joined #openstack-infra | 04:45 | |
| *** kaisers has joined #openstack-infra | 04:45 | |
| *** zeus is now known as Guest180 | 04:45 | |
| *** dansmith has joined #openstack-infra | 04:45 | |
| *** ilpianista_ has quit IRC | 04:45 | |
| *** cburgess has joined #openstack-infra | 04:45 | |
| *** honza is now known as Guest28838 | 04:45 | |
| *** fbouliane has joined #openstack-infra | 04:45 | |
| *** mugsie has joined #openstack-infra | 04:45 | |
| *** abelur has joined #openstack-infra | 04:45 | |
| *** loquacities has joined #openstack-infra | 04:45 | |
| *** katyafervent_awa has joined #openstack-infra | 04:45 | |
| *** Swimingly has joined #openstack-infra | 04:45 | |
| *** ChanServ sets mode: +v openstackstatus | 04:45 | |
| *** amotoki has joined #openstack-infra | 04:45 | |
| *** mtreinish has joined #openstack-infra | 04:45 | |
| *** bradjones has joined #openstack-infra | 04:45 | |
| *** sai has joined #openstack-infra | 04:45 | |
| *** Dinesh_Bhor has joined #openstack-infra | 04:45 | |
| *** jianghuaw has joined #openstack-infra | 04:45 | |
| *** antonym has joined #openstack-infra | 04:45 | |
| *** masayukig[m] has quit IRC | 04:45 | |
| *** dmsimard|off has joined #openstack-infra | 04:45 | |
| *** Gorian has joined #openstack-infra | 04:46 | |
| *** mandre has joined #openstack-infra | 04:46 | |
| *** patriciadomin has joined #openstack-infra | 04:46 | |
| *** mugsie has quit IRC | 04:46 | |
| *** mugsie has joined #openstack-infra | 04:46 | |
| *** masayukig has joined #openstack-infra | 04:46 | |
| *** Swimingly has quit IRC | 04:46 | |
| *** Swimingly has joined #openstack-infra | 04:46 | |
| *** timrc has joined #openstack-infra | 04:46 | |
| *** dansmith is now known as Guest53762 | 04:46 | |
| *** EmilienM has quit IRC | 04:46 | |
| *** EmilienM has joined #openstack-infra | 04:46 | |
| *** zigo has joined #openstack-infra | 04:46 | |
| *** antonym has quit IRC | 04:46 | |
| *** antonym has joined #openstack-infra | 04:46 | |
| *** toabctl has joined #openstack-infra | 04:46 | |
| *** mmedvede has joined #openstack-infra | 04:46 | |
| *** andreykurilin has joined #openstack-infra | 04:46 | |
| *** mattoliverau has joined #openstack-infra | 04:46 | |
| *** jogo has joined #openstack-infra | 04:46 | |
| *** wznoinsk has joined #openstack-infra | 04:46 | |
| *** timburke has joined #openstack-infra | 04:46 | |
| *** SpamapS has joined #openstack-infra | 04:46 | |
| *** ruhe has joined #openstack-infra | 04:46 | |
| *** bkero has joined #openstack-infra | 04:46 | |
| *** wfoster has joined #openstack-infra | 04:46 | |
| *** leifmadsen has joined #openstack-infra | 04:46 | |
| *** weshay has joined #openstack-infra | 04:46 | |
| *** amoralej has joined #openstack-infra | 04:46 | |
| *** bradjones is now known as Guest45420 | 04:46 | |
| *** arxcruz has joined #openstack-infra | 04:46 | |
| *** devananda has joined #openstack-infra | 04:46 | |
| *** filler has joined #openstack-infra | 04:46 | |
| *** quakers has joined #openstack-infra | 04:46 | |
| *** toabctl has quit IRC | 04:46 | |
| *** toabctl has joined #openstack-infra | 04:46 | |
| *** bkero has quit IRC | 04:46 | |
| *** bkero has joined #openstack-infra | 04:46 | |
| *** jogo has quit IRC | 04:46 | |
| *** jogo has joined #openstack-infra | 04:46 | |
| *** lihi has joined #openstack-infra | 04:46 | |
| *** njohnston has joined #openstack-infra | 04:46 | |
| *** jascott1 has joined #openstack-infra | 04:47 | |
| *** dgonzalez has joined #openstack-infra | 04:47 | |
| *** dtantsur has joined #openstack-infra | 04:47 | |
| *** puiterwijk has joined #openstack-infra | 04:47 | |
| *** yuanying has joined #openstack-infra | 04:47 | |
| *** uberjay has joined #openstack-infra | 04:47 | |
| *** eventingmonkey has joined #openstack-infra | 04:47 | |
| *** leifmadsen has quit IRC | 04:47 | |
| *** leifmadsen has joined #openstack-infra | 04:47 | |
| *** pbourke has joined #openstack-infra | 04:47 | |
| *** hemna has joined #openstack-infra | 04:47 | |
| *** zzzeek has joined #openstack-infra | 04:47 | |
| *** rybridges1 has joined #openstack-infra | 04:47 | |
| *** mdbooth has joined #openstack-infra | 04:47 | |
| *** niska has joined #openstack-infra | 04:47 | |
| *** rhagarty has joined #openstack-infra | 04:47 | |
| *** kambiz has joined #openstack-infra | 04:47 | |
| *** daniel has joined #openstack-infra | 04:47 | |
| *** rfolco has joined #openstack-infra | 04:47 | |
| *** stephenfin has joined #openstack-infra | 04:47 | |
| *** kevinbenton has joined #openstack-infra | 04:47 | |
| *** inc01 has joined #openstack-infra | 04:47 | |
| *** jtomasek has joined #openstack-infra | 04:47 | |
| *** yuval has joined #openstack-infra | 04:48 | |
| *** hwoarang has joined #openstack-infra | 04:48 | |
| *** ilpianista has joined #openstack-infra | 04:48 | |
| *** daniel is now known as Guest48406 | 04:48 | |
| *** dpawlik has joined #openstack-infra | 04:48 | |
| *** hwoarang has quit IRC | 04:48 | |
| *** hwoarang has joined #openstack-infra | 04:48 | |
| *** gtmanfred has joined #openstack-infra | 04:48 | |
| *** rtjure has joined #openstack-infra | 04:48 | |
| *** xarses has joined #openstack-infra | 04:48 | |
| *** gtmanfred has quit IRC | 04:48 | |
| *** gtmanfred has joined #openstack-infra | 04:48 | |
| *** numans has joined #openstack-infra | 04:48 | |
| *** aviau has joined #openstack-infra | 04:48 | |
| *** luzC has joined #openstack-infra | 04:48 | |
| *** Sukhdev has joined #openstack-infra | 04:48 | |
| *** gagehugo has joined #openstack-infra | 04:48 | |
| *** dteselkin has joined #openstack-infra | 04:48 | |
| *** xarses has quit IRC | 04:48 | |
| *** jgriffith has joined #openstack-infra | 04:48 | |
| *** fabo has joined #openstack-infra | 04:48 | |
| *** charz has joined #openstack-infra | 04:48 | |
| *** seanhandley has joined #openstack-infra | 04:48 | |
| *** zhurong has joined #openstack-infra | 04:48 | |
| *** DinaBelova has joined #openstack-infra | 04:49 | |
| *** lucasagomes has joined #openstack-infra | 04:49 | |
| *** fabo has quit IRC | 04:49 | |
| *** fabo has joined #openstack-infra | 04:49 | |
| *** jistr has joined #openstack-infra | 04:49 | |
| *** otherwiseguy has joined #openstack-infra | 04:49 | |
| *** jlk has joined #openstack-infra | 04:49 | |
| *** castulo has joined #openstack-infra | 04:49 | |
| *** clayton has joined #openstack-infra | 04:49 | |
| *** dims has joined #openstack-infra | 04:49 | |
| *** xarses has joined #openstack-infra | 04:49 | |
| *** spiffxp has joined #openstack-infra | 04:49 | |
| *** ianw has joined #openstack-infra | 04:49 | |
| *** zerick has joined #openstack-infra | 04:49 | |
| *** stevemar has joined #openstack-infra | 04:49 | |
| *** jhesketh has joined #openstack-infra | 04:49 | |
| *** jpena|off has joined #openstack-infra | 04:49 | |
| *** hughsaunders has joined #openstack-infra | 04:49 | |
| *** igormarnat has joined #openstack-infra | 04:49 | |
| *** jaosorior has joined #openstack-infra | 04:49 | |
| *** jlk has quit IRC | 04:49 | |
| *** jlk has joined #openstack-infra | 04:49 | |
| *** dims is now known as Guest58397 | 04:49 | |
| *** akscram1 has joined #openstack-infra | 04:49 | |
| *** thorre has joined #openstack-infra | 04:49 | |
| *** xarses has quit IRC | 04:49 | |
| *** adreznec has joined #openstack-infra | 04:49 | |
| *** Jeffrey4l has joined #openstack-infra | 04:49 | |
| *** smcginnis has joined #openstack-infra | 04:49 | |
| *** larsks has joined #openstack-infra | 04:50 | |
| *** bexelbie has joined #openstack-infra | 04:50 | |
| *** mnaser has joined #openstack-infra | 04:50 | |
| *** tobiash has joined #openstack-infra | 04:50 | |
| *** andreaf has joined #openstack-infra | 04:50 | |
| *** notmyname has joined #openstack-infra | 04:50 | |
| *** ericyoung has joined #openstack-infra | 04:50 | |
| *** purp has joined #openstack-infra | 04:50 | |
| *** xarses has joined #openstack-infra | 04:50 | |
| *** panda has joined #openstack-infra | 04:50 | |
| *** mrunge has joined #openstack-infra | 04:50 | |
| *** funzo has joined #openstack-infra | 04:50 | |
| *** trown has joined #openstack-infra | 04:50 | |
| *** jcook has joined #openstack-infra | 04:50 | |
| *** vipul has joined #openstack-infra | 04:50 | |
| *** dmanchad has joined #openstack-infra | 04:50 | |
| *** kgiusti has joined #openstack-infra | 04:50 | |
| *** Krenair has joined #openstack-infra | 04:51 | |
| *** masayukig[m] has joined #openstack-infra | 04:51 | |
| *** markmcd has joined #openstack-infra | 04:51 | |
| *** jesusaurum has joined #openstack-infra | 04:51 | |
| *** ramishra has joined #openstack-infra | 04:51 | |
| *** bnemec has joined #openstack-infra | 04:51 | |
| *** kazsh has joined #openstack-infra | 04:51 | |
| *** cshastri has joined #openstack-infra | 04:52 | |
| *** migi has joined #openstack-infra | 04:52 | |
| *** med_ has joined #openstack-infra | 04:52 | |
| *** asilenkov has joined #openstack-infra | 04:52 | |
| *** med_ is now known as Guest75213 | 04:52 | |
| *** rcarrill1 has joined #openstack-infra | 04:53 | |
| *** aeng has joined #openstack-infra | 04:53 | |
| *** Qiming has joined #openstack-infra | 04:53 | |
| *** rodrigods has joined #openstack-infra | 04:54 | |
| *** adisky__ has joined #openstack-infra | 04:54 | |
| *** dangers_away has joined #openstack-infra | 04:54 | |
| *** cinerama has joined #openstack-infra | 04:54 | |
| *** Keitaro has joined #openstack-infra | 04:54 | |
| *** raginbajin has joined #openstack-infra | 04:55 | |
| *** tnovacik has joined #openstack-infra | 04:56 | |
| *** tristanC has joined #openstack-infra | 04:56 | |
| *** zhurong has quit IRC | 04:59 | |
| *** tojuvone has left #openstack-infra | 04:59 | |
| *** spzala has joined #openstack-infra | 05:01 | |
| *** szaher has joined #openstack-infra | 05:04 | |
| *** spzala has quit IRC | 05:05 | |
| *** ilpianista_ has joined #openstack-infra | 05:06 | |
| *** markvoelker has quit IRC | 05:07 | |
| *** GregHouse has quit IRC | 05:09 | |
| *** LindaWang has joined #openstack-infra | 05:09 | |
| *** michaelxin has joined #openstack-infra | 05:10 | |
| *** lindsayk has joined #openstack-infra | 05:11 | |
| clarkb | ianw: sounds good thanks | 05:11 |
| clarkb | also woo freenode weirdness | 05:11 |
| *** cuongnv has quit IRC | 05:14 | |
| *** GregHouse has joined #openstack-infra | 05:15 | |
| *** jascott1 has quit IRC | 05:16 | |
| *** jascott1 has joined #openstack-infra | 05:17 | |
| *** masuberu has joined #openstack-infra | 05:18 | |
| *** isaacb has joined #openstack-infra | 05:18 | |
| *** masber has quit IRC | 05:21 | |
| *** jascott1 has quit IRC | 05:21 | |
| *** thorst has joined #openstack-infra | 05:22 | |
| *** e0ne has joined #openstack-infra | 05:23 | |
| *** masber has joined #openstack-infra | 05:25 | |
| *** sree has joined #openstack-infra | 05:25 | |
| *** thorst has quit IRC | 05:27 | |
| *** cuongnv has joined #openstack-infra | 05:27 | |
| *** masuberu has quit IRC | 05:28 | |
| *** zhurong has joined #openstack-infra | 05:29 | |
| *** jaosorior has quit IRC | 05:32 | |
| *** jaosorior has joined #openstack-infra | 05:32 | |
| *** udesale has joined #openstack-infra | 05:34 | |
| *** eeiden has joined #openstack-infra | 05:40 | |
| *** e0ne has quit IRC | 05:41 | |
| *** udesale__ has joined #openstack-infra | 05:43 | |
| *** gcb has joined #openstack-infra | 05:45 | |
| *** udesale has quit IRC | 05:45 | |
| *** aeng_ has joined #openstack-infra | 05:46 | |
| *** aeng__ has joined #openstack-infra | 05:46 | |
| *** aeng__ has quit IRC | 05:46 | |
| *** aeng has quit IRC | 05:48 | |
| *** dmsimard|off has quit IRC | 05:48 | |
| *** dmsimard has joined #openstack-infra | 05:49 | |
| *** quite has quit IRC | 05:49 | |
| *** dmsimard is now known as dmsimard|off | 05:49 | |
| *** quite has joined #openstack-infra | 05:50 | |
| *** Guest28838 is now known as honza | 05:50 | |
| *** Sukhdev has quit IRC | 05:53 | |
| *** rcernin has joined #openstack-infra | 05:57 | |
| *** ramishra has quit IRC | 06:01 | |
| *** spzala has joined #openstack-infra | 06:02 | |
| *** iyamahat has joined #openstack-infra | 06:02 | |
| *** iyamahat has quit IRC | 06:03 | |
| *** iyamahat has joined #openstack-infra | 06:03 | |
| *** ramishra has joined #openstack-infra | 06:03 | |
| *** aeng_ has quit IRC | 06:05 | |
| *** kjackal has quit IRC | 06:05 | |
| *** e0ne has joined #openstack-infra | 06:06 | |
| *** spzala has quit IRC | 06:06 | |
| *** florianf has joined #openstack-infra | 06:07 | |
| *** markvoelker has joined #openstack-infra | 06:08 | |
| *** tumbarka has joined #openstack-infra | 06:09 | |
| *** tnovacik has quit IRC | 06:09 | |
| *** gildub has quit IRC | 06:13 | |
| *** sree has quit IRC | 06:16 | |
| *** pgadiya has joined #openstack-infra | 06:16 | |
| *** isaacb has quit IRC | 06:17 | |
| *** openstackgerrit has joined #openstack-infra | 06:18 | |
| openstackgerrit | Omer Anson proposed openstack-infra/project-config master: Dragonflow check: Add dragonflow-kuryr and OSA-dragonflow jobs https://review.openstack.org/494388 | 06:18 |
| *** sree has joined #openstack-infra | 06:18 | |
| *** binbincong has quit IRC | 06:19 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Update translation reviewing URL https://review.openstack.org/494389 | 06:19 |
| *** pgadiya has quit IRC | 06:22 | |
| *** GregHouse has quit IRC | 06:22 | |
| *** ianychoi has quit IRC | 06:23 | |
| *** thorst has joined #openstack-infra | 06:23 | |
| *** liusheng has joined #openstack-infra | 06:25 | |
| *** GregHouse has joined #openstack-infra | 06:25 | |
| *** ianychoi has joined #openstack-infra | 06:26 | |
| *** gildub has joined #openstack-infra | 06:27 | |
| *** thorst has quit IRC | 06:28 | |
| *** isaacb has joined #openstack-infra | 06:28 | |
| *** eranrom has joined #openstack-infra | 06:35 | |
| *** udesale has joined #openstack-infra | 06:40 | |
| *** udesale__ has quit IRC | 06:42 | |
| *** markvoelker has quit IRC | 06:42 | |
| *** pcaruana has joined #openstack-infra | 06:42 | |
| *** slaweq has joined #openstack-infra | 06:44 | |
| openstackgerrit | Jamie Lennox proposed openstack-infra/glean master: Define CONFIG_DRIVE_LABEL in default case https://review.openstack.org/494396 | 06:44 |
| jamielennox | mnaser: can you have a look at ^, it appears it was broken by your last patch and so glean 1.9.2 is broken | 06:46 |
| *** martinkopec has joined #openstack-infra | 06:47 | |
| *** tesseract has joined #openstack-infra | 06:47 | |
| *** mika has joined #openstack-infra | 06:47 | |
| *** mika has joined #openstack-infra | 06:47 | |
| *** slaweq has quit IRC | 06:48 | |
| *** ttx has quit IRC | 06:49 | |
| *** ttx has joined #openstack-infra | 06:49 | |
| *** tuanluong has joined #openstack-infra | 06:53 | |
| jamielennox | does anyone know if it's possible to have nodepool-builder/dib specify a version number for glean? | 06:53 |
| *** Jeffrey4l has quit IRC | 06:56 | |
| *** liujiong_lj has joined #openstack-infra | 06:56 | |
| *** Jeffrey4l has joined #openstack-infra | 06:57 | |
| *** liujiong has quit IRC | 06:57 | |
| *** kjackal has joined #openstack-infra | 06:57 | |
| *** mika has quit IRC | 06:58 | |
| *** mika has joined #openstack-infra | 06:58 | |
| *** mika has joined #openstack-infra | 06:58 | |
| *** gildub has quit IRC | 06:58 | |
| *** shardy has joined #openstack-infra | 07:03 | |
| *** markus_z has joined #openstack-infra | 07:04 | |
| *** eranrom has quit IRC | 07:08 | |
| *** nicolasbock has joined #openstack-infra | 07:09 | |
| openstackgerrit | Flavio Percoco proposed openstack-infra/project-config master: Split COE scenarios: 1 for each COE https://review.openstack.org/494272 | 07:09 |
| *** ykarel is now known as ykarel|afk | 07:10 | |
| *** Dinesh_Bhor has quit IRC | 07:11 | |
| *** eranrom has joined #openstack-infra | 07:12 | |
| *** ccamacho has joined #openstack-infra | 07:12 | |
| *** jpich has joined #openstack-infra | 07:13 | |
| *** coolsvap has joined #openstack-infra | 07:16 | |
| *** jbadiapa_ has joined #openstack-infra | 07:17 | |
| *** apetrich has quit IRC | 07:18 | |
| *** Dinesh_Bhor has joined #openstack-infra | 07:18 | |
| *** dizquierdo has joined #openstack-infra | 07:19 | |
| *** tuanluong has quit IRC | 07:20 | |
| *** lin_yang has quit IRC | 07:21 | |
| *** thorst has joined #openstack-infra | 07:24 | |
| *** jpena|off is now known as jpena | 07:24 | |
| *** frickler has joined #openstack-infra | 07:29 | |
| *** thorst has quit IRC | 07:29 | |
| *** liujiong_lj is now known as liujiong | 07:32 | |
| *** lbragstad has quit IRC | 07:33 | |
| *** lbragstad_ has joined #openstack-infra | 07:33 | |
| *** shardy is now known as shardy_afk | 07:34 | |
| *** rtjure has quit IRC | 07:35 | |
| *** arturb has joined #openstack-infra | 07:35 | |
| *** cuongnv has quit IRC | 07:35 | |
| *** cuongnv has joined #openstack-infra | 07:36 | |
| *** rtjure has joined #openstack-infra | 07:39 | |
| *** markvoelker has joined #openstack-infra | 07:40 | |
| *** sshnaidm|afk has joined #openstack-infra | 07:41 | |
| *** sshnaidm|afk is now known as sshnaidm | 07:41 | |
| *** alexchadin has joined #openstack-infra | 07:42 | |
| *** gibi has joined #openstack-infra | 07:42 | |
| *** gildub has joined #openstack-infra | 07:43 | |
| *** SotK has joined #openstack-infra | 07:44 | |
| *** dmellado has joined #openstack-infra | 07:47 | |
| AJaeger | yolanda, could you put these on your review list, please? https://review.openstack.org/494389 https://review.openstack.org/493936 | 07:48 |
| yolanda | hi AJaeger , will do | 07:48 |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Update Horizon meeting to reflect current PTLs https://review.openstack.org/494257 | 07:48 |
| AJaeger | thanks, yolanda | 07:48 |
| yolanda | np | 07:51 |
| *** mikal has joined #openstack-infra | 07:51 | |
| dmellado | yolanda: o/ | 07:51 |
| yolanda | hi dmellado | 07:52 |
| *** markus_z has quit IRC | 07:53 | |
| *** wendar has quit IRC | 07:54 | |
| *** honza has quit IRC | 07:54 | |
| *** honza has joined #openstack-infra | 07:54 | |
| *** honza is now known as Guest11829 | 07:55 | |
| *** wendar has joined #openstack-infra | 07:55 | |
| openstackgerrit | Duong Ha-Quang proposed openstack-infra/project-config master: Add Kolla Ansible Plugins project https://review.openstack.org/491412 | 07:55 |
| *** jogo has quit IRC | 07:56 | |
| *** rhagarty has quit IRC | 07:56 | |
| *** zzzeek has quit IRC | 07:56 | |
| *** castulo has quit IRC | 07:57 | |
| *** castulo has joined #openstack-infra | 07:58 | |
| *** tesseract has quit IRC | 07:58 | |
| *** zzzeek has joined #openstack-infra | 07:59 | |
| *** rhagarty has joined #openstack-infra | 08:01 | |
| *** spzala has joined #openstack-infra | 08:03 | |
| openstackgerrit | Merged openstack-infra/project-config master: zuul: layout: AIO: Add openSUSE Leap 42.3 as non-voting CI job https://review.openstack.org/494116 | 08:03 |
| *** claudiub has joined #openstack-infra | 08:03 | |
| *** derekh has joined #openstack-infra | 08:04 | |
| *** sflanigan has quit IRC | 08:06 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add comment to zuul.yaml https://review.openstack.org/493936 | 08:07 |
| *** markus_z has joined #openstack-infra | 08:09 | |
| *** spzala has quit IRC | 08:10 | |
| *** markvoelker has quit IRC | 08:12 | |
| *** priteau has joined #openstack-infra | 08:12 | |
| *** wendar has quit IRC | 08:14 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add IRC notifications for #openstack-publiccloud. https://review.openstack.org/493934 | 08:14 |
| *** odyssey4me_ is now known as odyssey4me | 08:15 | |
| *** wendar has joined #openstack-infra | 08:15 | |
| openstackgerrit | Merged openstack-infra/project-config master: OpenStack-Ansible .txt files should not be skipped https://review.openstack.org/493759 | 08:16 |
| *** egonzalez has joined #openstack-infra | 08:16 | |
| *** openstackgerrit has quit IRC | 08:17 | |
| *** priteau has quit IRC | 08:18 | |
| *** Anticimex has joined #openstack-infra | 08:19 | |
| *** iyamahat has quit IRC | 08:19 | |
| *** priteau has joined #openstack-infra | 08:19 | |
| *** shardy_afk is now known as shardy | 08:20 | |
| *** gaurangt has joined #openstack-infra | 08:21 | |
| *** priteau has quit IRC | 08:23 | |
| *** openstackgerrit has joined #openstack-infra | 08:24 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Move creating fake image to oooq extras https://review.openstack.org/494233 | 08:24 |
| *** thorst has joined #openstack-infra | 08:25 | |
| *** dulek has quit IRC | 08:27 | |
| *** markus_z has quit IRC | 08:28 | |
| *** e0ne has quit IRC | 08:28 | |
| *** thorst has quit IRC | 08:30 | |
| *** apetrich has joined #openstack-infra | 08:31 | |
| *** priteau has joined #openstack-infra | 08:35 | |
| *** logan- has quit IRC | 08:37 | |
| *** eranrom_ has joined #openstack-infra | 08:37 | |
| *** eranrom has quit IRC | 08:41 | |
| *** priteau has quit IRC | 08:41 | |
| *** eranrom_ has quit IRC | 08:41 | |
| *** dingyichen has quit IRC | 08:43 | |
| frickler | AJaeger: yolanda: there is a series of p-c gate failures due to a change in service types, do you know if there is a fix for that already proposed? e.g. http://logs.openstack.org/55/494155/1/gate/gate-project-config-jenkins-project/b697b31/console.html#_2017-08-17_07_54_57_447933 | 08:43 |
| *** markus_z has joined #openstack-infra | 08:44 | |
| yolanda | frickler, i'm not aware of that | 08:47 |
| frickler | seems related to https://review.openstack.org/493325, but looks like an unintentional bug to me, mordred or efried might know more | 08:51 |
| *** jogo has joined #openstack-infra | 08:53 | |
| *** camunoz has joined #openstack-infra | 08:55 | |
| *** alexchadin has quit IRC | 08:56 | |
| *** alexchadin has joined #openstack-infra | 08:57 | |
| *** aspiers has joined #openstack-infra | 08:59 | |
| vdrok | morning all, https://review.openstack.org/494166 hitting the same issue with service types | 09:01 |
| *** apetrich has quit IRC | 09:01 | |
| *** apetrich has joined #openstack-infra | 09:02 | |
| *** alexchadin has quit IRC | 09:05 | |
| *** alexchadin has joined #openstack-infra | 09:05 | |
| *** Guest11829 is now known as honza | 09:09 | |
| *** priteau has joined #openstack-infra | 09:09 | |
| *** markvoelker has joined #openstack-infra | 09:10 | |
| *** jklare has quit IRC | 09:12 | |
| frickler | yeah, my current assumption is that project-config/tools/jenkins-projects-checks.py is still missing code to handle the new secondary service types properly. might need a revert of 493325 for now | 09:16 |
| *** jklare has joined #openstack-infra | 09:17 | |
| *** jascott1 has joined #openstack-infra | 09:18 | |
| *** afazekas is now known as afazekas|sick | 09:19 | |
| *** jklare has quit IRC | 09:21 | |
| *** jklare has joined #openstack-infra | 09:22 | |
| *** jascott1 has quit IRC | 09:22 | |
| *** tosky has joined #openstack-infra | 09:24 | |
| *** thorst has joined #openstack-infra | 09:26 | |
| *** thorst has quit IRC | 09:30 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/project-config master: Skip secondary service types https://review.openstack.org/494456 | 09:33 |
| frickler | ^^ that should fix it, hopefully, just copied the skip code over | 09:33 |
| *** markvoelker has quit IRC | 09:43 | |
| *** sambetts_ is now known as sambetts | 09:47 | |
| *** afazekas|sick is now known as afazekas | 09:47 | |
| *** kjackal has quit IRC | 09:51 | |
| *** kjackal has joined #openstack-infra | 09:51 | |
| *** electrofelix has joined #openstack-infra | 09:53 | |
| frickler | hmm, zuul seems a bit stuck, too, isn't it? | 09:54 |
| *** cuongnv has quit IRC | 09:54 | |
| *** alexchadin has quit IRC | 09:55 | |
| *** alexchadin has joined #openstack-infra | 09:56 | |
| *** isviridoz has quit IRC | 10:04 | |
| *** felipemonteiro has joined #openstack-infra | 10:05 | |
| *** spzala has joined #openstack-infra | 10:07 | |
| *** ralonsoh has joined #openstack-infra | 10:08 | |
| *** claudiub has quit IRC | 10:09 | |
| *** kong has quit IRC | 10:10 | |
| *** beisner has quit IRC | 10:11 | |
| *** coolsvap has quit IRC | 10:11 | |
| *** felipemonteiro has quit IRC | 10:11 | |
| *** florianf has quit IRC | 10:12 | |
| *** robled has quit IRC | 10:12 | |
| *** spzala has quit IRC | 10:12 | |
| *** kong has joined #openstack-infra | 10:13 | |
| *** beisner has joined #openstack-infra | 10:14 | |
| *** dgonzalez has quit IRC | 10:14 | |
| *** bogdando has joined #openstack-infra | 10:14 | |
| *** ykarel|afk is now known as ykarel | 10:15 | |
| *** coolsvap has joined #openstack-infra | 10:15 | |
| *** dgonzalez has joined #openstack-infra | 10:16 | |
| *** robled has joined #openstack-infra | 10:17 | |
| *** robled has joined #openstack-infra | 10:17 | |
| *** udesale has quit IRC | 10:18 | |
| openstackgerrit | Flavio Percoco proposed openstack-infra/project-config master: Split COE scenarios: 1 for each COE https://review.openstack.org/494272 | 10:24 |
| *** florianf has joined #openstack-infra | 10:25 | |
| *** thorst has joined #openstack-infra | 10:27 | |
| *** zhurong has quit IRC | 10:27 | |
| *** e0ne has joined #openstack-infra | 10:27 | |
| AJaeger | frickler: thanks | 10:30 |
| *** thorst has quit IRC | 10:31 | |
| *** zhurong has joined #openstack-infra | 10:32 | |
| AJaeger | frickler: unfortunately fails test suite ;( - you need to handle this differently | 10:36 |
| *** liujiong has quit IRC | 10:38 | |
| *** alexchadin has quit IRC | 10:39 | |
| *** alexchadin has joined #openstack-infra | 10:40 | |
| *** alexchadin has quit IRC | 10:40 | |
| *** markvoelker has joined #openstack-infra | 10:40 | |
| *** alexchadin has joined #openstack-infra | 10:40 | |
| *** alexchadin has quit IRC | 10:41 | |
| frickler | AJaeger: yeah, just saw that, but at least down from 3 to 1 errors ;), the senlin change in 493325 needs more action | 10:41 |
| *** alexchadin has joined #openstack-infra | 10:41 | |
| *** alexchadin has quit IRC | 10:41 | |
| *** alexchadin has joined #openstack-infra | 10:42 | |
| *** bogdando has quit IRC | 10:43 | |
| *** jkilpatr has quit IRC | 10:44 | |
| *** alexchadin has quit IRC | 10:46 | |
| *** apetrich_ has joined #openstack-infra | 10:47 | |
| *** apetrich has quit IRC | 10:47 | |
| *** bogdando has joined #openstack-infra | 10:48 | |
| *** ramishra has quit IRC | 10:48 | |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/project-config master: Update service type validation https://review.openstack.org/494456 | 10:49 |
| *** ramishra has joined #openstack-infra | 10:50 | |
| *** bogdando has left #openstack-infra | 10:52 | |
| *** sree has quit IRC | 10:55 | |
| *** sree has joined #openstack-infra | 10:56 | |
| *** kjackal has quit IRC | 10:57 | |
| *** makowals has quit IRC | 10:58 | |
| *** markus_z has quit IRC | 10:58 | |
| AJaeger | mordred: efried, please review ^ | 10:58 |
| *** cshastri has quit IRC | 10:59 | |
| AJaeger | Qiming: ^ | 10:59 |
| *** sree has quit IRC | 11:00 | |
| *** shardy is now known as shardy_lunch | 11:01 | |
| *** katkapilatova has joined #openstack-infra | 11:02 | |
| *** sree has joined #openstack-infra | 11:02 | |
| *** vaidy has quit IRC | 11:03 | |
| *** Zara has joined #openstack-infra | 11:05 | |
| *** sree has quit IRC | 11:06 | |
| *** aviau has quit IRC | 11:08 | |
| *** aviau has joined #openstack-infra | 11:08 | |
| *** claudiub has joined #openstack-infra | 11:08 | |
| *** spzala has joined #openstack-infra | 11:09 | |
| openstackgerrit | Sam Betts proposed openstack-infra/project-config master: Ensure Ironic grenade tests correct branches https://review.openstack.org/494482 | 11:09 |
| *** apetrich_ has quit IRC | 11:12 | |
| *** markus_z has joined #openstack-infra | 11:13 | |
| *** spzala has quit IRC | 11:13 | |
| *** markvoelker has quit IRC | 11:13 | |
| *** ramishra has quit IRC | 11:14 | |
| *** wolverineav has joined #openstack-infra | 11:15 | |
| *** ramishra has joined #openstack-infra | 11:15 | |
| *** ramishra has quit IRC | 11:17 | |
| *** nicolasbock_ has joined #openstack-infra | 11:18 | |
| *** ramishra has joined #openstack-infra | 11:18 | |
| *** danieli has joined #openstack-infra | 11:19 | |
| *** danieli has quit IRC | 11:19 | |
| *** danieli has joined #openstack-infra | 11:19 | |
| *** markvoelker has joined #openstack-infra | 11:21 | |
| *** ramishra has quit IRC | 11:21 | |
| *** nicolasbock has quit IRC | 11:21 | |
| *** ramishra has joined #openstack-infra | 11:21 | |
| *** jkilpatr has joined #openstack-infra | 11:24 | |
| *** thorst has joined #openstack-infra | 11:27 | |
| *** lucasagomes is now known as lucas-hungry | 11:30 | |
| *** thorst has quit IRC | 11:32 | |
| *** alexchadin has joined #openstack-infra | 11:34 | |
| *** ldnunes has joined #openstack-infra | 11:35 | |
| *** makowals has joined #openstack-infra | 11:37 | |
| *** jkilpatr has quit IRC | 11:37 | |
| *** jkilpatr has joined #openstack-infra | 11:37 | |
| *** Guest58397 is now known as dims | 11:37 | |
| *** Guest88899 is now known as rook | 11:39 | |
| openstackgerrit | Sam Betts proposed openstack-infra/project-config master: Ensure Ironic grenade tests correct branches https://review.openstack.org/494482 | 11:39 |
| *** kgiusti has left #openstack-infra | 11:39 | |
| frickler | who is wiki admin? I'm trying to change my email address at https://wiki.openstack.org/wiki/Special:ChangeEmail as linked from Special:Preferences, but that gives me "No such special page" | 11:41 |
| AJaeger | frickler: fungi should be able to help... | 11:42 |
| AJaeger | frickler: you married? | 11:42 |
| frickler | AJaeger: yep, though there are other occations where one could change ones last name ;) | 11:44 |
| *** thorst has joined #openstack-infra | 11:45 | |
| fungi | witness protection program ;) | 11:45 |
| *** Julien-zte has joined #openstack-infra | 11:46 | |
| *** kjackal has joined #openstack-infra | 11:46 | |
| AJaeger | frickler: congrats! | 11:46 |
| fungi | frickler: lemme see if i can figure something out, though i'm about to head out of town for a week | 11:46 |
| openstackgerrit | Tomasz Trębski proposed openstack-infra/project-config master: Provide unified gate configuration https://review.openstack.org/490790 | 11:47 |
| frickler | fungi: not urgent, the old address is still working as an alias for now | 11:47 |
| AJaeger | frickler, mordred, efried : I suggest to merge https://review.openstack.org/#/c/494486/ and then merge frickler's change https://review.openstack.org/494456 without the senlin rename to unblock project-config - and then enhance project-config check to handle aliases properly. | 11:49 |
| AJaeger | infra-root, frickler, mordred, efried: We cannot merge any project config changes due to the changes done in https://review.openstack.org/#/c/484011/ and 493325 for services-types-authority. We first need to update project-config/tools/jenkins-projects-checks.py to handle the aliases correctly. mordred, efried, can you look into this today? | 11:51 |
| *** arturb has quit IRC | 11:51 | |
| *** rhallisey has joined #openstack-infra | 11:51 | |
| *** sbezverk has joined #openstack-infra | 11:53 | |
| frickler | AJaeger: to be precise, only changes touching jenkins/jobs/projects.yaml are blocked, but well ;), I'm fine with reverting my change back to PS1, feel free to do so if I'm away later | 11:53 |
| *** alexchadin has quit IRC | 11:55 | |
| *** alexchadin has joined #openstack-infra | 11:56 | |
| *** shardy_lunch is now known as shardy | 11:57 | |
| AJaeger | frickler: that makes it half of the changes ;). Let's see what I hear. Another option is to disable senlin publishing for now. | 11:58 |
| *** alexchadin has quit IRC | 11:58 | |
| *** alexchadin has joined #openstack-infra | 11:58 | |
| fungi | frickler: did you change your primary e-mail address at login.ubuntu.com? looking through our mediawiki config, i get the impression it may take e-mail address updates when logging in via openid (by authorizing the openid to share your e-mail address with the wiki account system) | 11:59 |
| AJaeger | frickler: any change to jenkins/jobs is blocked | 11:59 |
| mnaser | jamielennox ouch. sorry about introducing that bug | 12:00 |
| *** mat128 has joined #openstack-infra | 12:00 | |
| frickler | fungi: launchpad did share the new address (just did a re-login to confirm), but the email options in preferences still show the old one | 12:03 |
| fungi | frickler: weird. i'm still digging for other possible causes | 12:04 |
| frickler | fungi: ah, wait, there are some options on the OpenID tab | 12:05 |
| frickler | ticking "User profile information fields which will be automatically updated from OpenID persona every time when you log in: Email address" worked | 12:06 |
| *** markus_z has quit IRC | 12:07 | |
| *** ff is now known as flaper87 | 12:07 | |
| *** flaper87 is now known as Guest38917 | 12:08 | |
| *** spzala has joined #openstack-infra | 12:09 | |
| *** mattmceuen has joined #openstack-infra | 12:11 | |
| *** Guest38917 has quit IRC | 12:11 | |
| *** ff has joined #openstack-infra | 12:11 | |
| *** spzala has quit IRC | 12:14 | |
| *** jtomasek_ has joined #openstack-infra | 12:14 | |
| *** jtomasek has quit IRC | 12:16 | |
| *** alexchadin has quit IRC | 12:18 | |
| *** alexchadin has joined #openstack-infra | 12:19 | |
| *** dprince has joined #openstack-infra | 12:20 | |
| *** markus_z has joined #openstack-infra | 12:21 | |
| *** gaurangt has quit IRC | 12:22 | |
| *** mattmceuen has quit IRC | 12:24 | |
| *** sshnaidm is now known as sshnaidm|afk | 12:26 | |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: Use new keystoneauth version discovery https://review.openstack.org/493582 | 12:27 |
| *** ff is now known as flaper87282809 | 12:29 | |
| *** flaper87282809 is now known as fl7134a92per87 | 12:29 | |
| efried | AJaeger Still around? | 12:29 |
| *** fl7134a92per87 is now known as ff | 12:29 | |
| *** jcoufal has joined #openstack-infra | 12:31 | |
| *** mat128 has quit IRC | 12:34 | |
| *** kgiusti has joined #openstack-infra | 12:34 | |
| fungi | frickler: awesome, so it works the way i interpreted, you just have to allow it in preferences--good to know--thanks for finding that! | 12:34 |
| *** rlandy has joined #openstack-infra | 12:36 | |
| *** jpena is now known as jpena|lunch | 12:36 | |
| *** mat128 has joined #openstack-infra | 12:36 | |
| *** dave-mccowan has joined #openstack-infra | 12:38 | |
| efried | fungi I need to make project-config use os-service-types. Do I add that guy to jenkins/scripts/release-tools/requirements.txt? Or...? | 12:41 |
| *** pleia2 has joined #openstack-infra | 12:42 | |
| *** lucas-hungry is now known as lucasagomes | 12:44 | |
| openstackgerrit | Samuel de Medeiros Queiroz proposed openstack-infra/shade master: Use new keystoneauth version discovery https://review.openstack.org/493582 | 12:44 |
| *** LindaWang has quit IRC | 12:44 | |
| efried | fungi Looks like tox.ini | 12:45 |
| *** heroux has joined #openstack-infra | 12:46 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Use new keystoneauth version discovery https://review.openstack.org/493582 | 12:47 |
| *** sree has joined #openstack-infra | 12:49 | |
| openstackgerrit | Merged openstack-infra/glean master: Define CONFIG_DRIVE_LABEL in default case https://review.openstack.org/494396 | 12:50 |
| *** sshnaidm|afk is now known as sshnaidm | 12:51 | |
| kklimonda | is there a reason other than lack of time for not moving to newer puppet releases? | 12:53 |
| jamielennox | mnaser: no worries, it happens to everyone. in this case it had your name right there so i thought i'd make sure it wasn't something i was doing | 12:53 |
| *** isaacb has quit IRC | 12:53 | |
| cmurphy | kklimonda: lack of time and people resources | 12:53 |
| pabelanger | mnaser: need seeing a few jobs fail on networking: http://logs.openstack.org/39/494039/2/gate/gate-tripleo-ci-centos-7-containers-multinode/1e2715b/logs/devstack-gate-setup-host.txt | 12:54 |
| cmurphy | kklimonda: we're partway through with http://specs.openstack.org/openstack-infra/infra-specs/specs/puppet_4_prelim_testing.html if you want to help pick up some of the work :) | 12:54 |
| mnaser | pabelanger that's so weird. not 2 seconds before the ping check passed in there and it can make http requests | 12:55 |
| mnaser | and that's over public so its just sitting on a vlan so no overlay network | 12:56 |
| mordred | efried, AJaeger: can I just suggest changing the senlin api-ref-jobs job definition to clustering instead of adding alias support? | 12:56 |
| efried | mordred Glad you're here, was gonna ask about that. | 12:56 |
| mordred | efried, AJaeger: the point of that check to is ensure people publish docs to their correct official location | 12:56 |
| pabelanger | mnaser: ya, I know we recently changed to a single ping | 12:57 |
| efried | mordred I'm about to propose a switchover to os-service-types in project-config. | 12:57 |
| mnaser | pabelanger this might sound really silly but because the mirror was left for a really long time, is it possible that there might be some config changes that it's missing? perhaps surrounding iptables and icmp rate limiting (dont want to point fingers yet but im unsure) | 12:57 |
| mordred | in senlin's case, the recent change is because we were too eager in assigning them a new official name a little while ago | 12:57 |
| *** zhurong has quit IRC | 12:57 | |
| mordred | efried: yes - I think we should definitely use os-service-types - but I think we should also set up a re-revert and then a project-config patch that switches senlin's project type and land them one after the other | 12:58 |
| efried | mordred Re-revert is done: https://review.openstack.org/#/c/494529/ | 12:58 |
| mordred | cool - patch coming | 12:58 |
| efried | mordred You doing the senlin project type one? | 12:58 |
| efried | I'm doing the os-service-types one. | 12:58 |
| mordred | yah | 12:59 |
| kklimonda | cmurphy: mmm, can't promise anything but we'll see what I can sell later, after we do the initial deployment with puppet3. I was just checking if you are not planning on dropping puppet altogether. | 13:00 |
| cmurphy | kklimonda: the idea gets tossed around a lot but i don't think that's where we're headed, at least not anytime soon | 13:01 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Update senlin's api-ref publication location https://review.openstack.org/494533 | 13:01 |
| mordred | AJaeger, pabelanger: ^^ | 13:01 |
| *** liujiong has joined #openstack-infra | 13:02 | |
| *** gildub has quit IRC | 13:02 | |
| *** dhill_ has joined #openstack-infra | 13:02 | |
| frickler | mordred: instead of that one you could just take https://review.openstack.org/494456 as is, that includes your patch | 13:02 |
| openstackgerrit | Eric Fried proposed openstack-infra/project-config master: Switch to os-service-types https://review.openstack.org/494456 | 13:04 |
| efried | mordred frickler fungi I did that via ^^. | 13:05 |
| openstackgerrit | Rosario Di Somma proposed openstack-infra/shade master: WIP: Use direct calls to get_<resource>_by_id https://review.openstack.org/494535 | 13:05 |
| efried | But that guy needs to *not* include the senlin update, because then it'd be gridlocked with the re-revert in sta. | 13:05 |
| efried | So I think mordred is proposing the senlin update separately. | 13:06 |
| frickler | efried: ah, o.k., I still think you need to merge with mordreds patch in order for jenkins to pass, but lets see what the checks say | 13:06 |
| efried | frickler The sta revert has merged at this point. | 13:06 |
| efried | In order for jenkins to pass now, we just need to get the secondary service stuff ironed out, which ^^ should do. | 13:06 |
| *** gothicmindfood has joined #openstack-infra | 13:06 | |
| *** esberglu has joined #openstack-infra | 13:07 | |
| openstackgerrit | Attila Darazs proposed openstack-infra/tripleo-ci master: Use local mirrors for multinode during DLRN build https://review.openstack.org/494537 | 13:08 |
| pabelanger | mnaser: fungi: clarkb: if we are looking into glean this morning, it would be nice to also get: https://review.openstack.org/#/q/topic:skip-dns reviews | 13:08 |
| *** isviridov_away has joined #openstack-infra | 13:09 | |
| *** spzala has joined #openstack-infra | 13:10 | |
| *** spzala has quit IRC | 13:10 | |
| *** spzala has joined #openstack-infra | 13:11 | |
| *** baoli has joined #openstack-infra | 13:11 | |
| frickler | efried: ah, right. does tox install deps from master or from pypi, though? in the latter case you might need a new release first | 13:15 |
| efried | frickler Oh, sh*t, you're right. | 13:15 |
| efried | Has lib thaw happened yet? | 13:15 |
| *** udesale has joined #openstack-infra | 13:16 | |
| *** hongbin has joined #openstack-infra | 13:16 | |
| mordred | os-service-types shouldn't be affected by freeze - it's not a per-release lib | 13:19 |
| efried | mordred Yet releases in the releases project still need to go under deliverables/<release>/ ? | 13:19 |
| *** jascott1 has joined #openstack-infra | 13:20 | |
| mordred | indeed. I think o-s-t is an edge-case we should likely discuss at some point | 13:20 |
| efried | mordred I'm gonna propose a new release. Do you suggest 1.1.0? | 13:20 |
| mordred | sounds good to me | 13:20 |
| mordred | efried: the re-revert hasn't landed? | 13:21 |
| efried | mordred uh, but we should probably get https://review.openstack.org/#/c/493366/ in first, nah? | 13:21 |
| efried | mordred I -Wed it, so I hope not. | 13:21 |
| efried | mordred Did you propose the projects.yaml update? | 13:21 |
| mordred | I did - https://review.openstack.org/#/c/494533/ | 13:21 |
| efried | oh, missed it. | 13:21 |
| mordred | also: remote: https://review.openstack.org/494543 Add redirect from resource-cluster to clustering | 13:22 |
| mordred | which adds a redirect to the docs repo | 13:22 |
| efried | mordred https://review.openstack.org/#/c/494529/1 | 13:22 |
| mordred | and once we're done with this we should add os-service-types support to www-generator in openstack-manuals so that we can just add redirects from all aliases to all official types | 13:22 |
| mnaser | mordred when you have a minute, a piece of code of glean from 2015 .. https://review.openstack.org/#/c/493330/ .. i think it's fine but a pair of eyes from you as you committed it would be better | 13:22 |
| mordred | mnaser: +A | 13:23 |
| mnaser | mordred merci! | 13:23 |
| efried | mordred You working on getting https://review.openstack.org/#/c/493366/ ready? You agree we should get that into os-service-types 1.1.0? | 13:24 |
| *** jascott1 has quit IRC | 13:24 | |
| mordred | efried, frickler, pabelanger: I think let's land the re-revert and land https://review.openstack.org/#/c/494533/ | 13:25 |
| efried | mordred I don't think we can do that until we accomodate secondary service types. | 13:25 |
| mordred | oh - right - because we also landed placement | 13:25 |
| mordred | efried: you know what ... | 13:26 |
| frickler | mordred: efried: you could take PS1 from https://review.openstack.org/494456 for that | 13:26 |
| efried | Could have other problems: http://logs.openstack.org/56/494456/3/check/gate-project-config-jenkins-project/be6d5d0/console.html | 13:26 |
| mordred | efried: oh - we're dumb | 13:27 |
| mordred | we don't need a new release | 13:27 |
| *** sc` has joined #openstack-infra | 13:27 | |
| efried | Is tox.ini pulling from master? | 13:27 |
| mordred | efried: we nede for our use of os-service-types to be passed a requests Session | 13:27 |
| mordred | turns out we have support for this use case in the lib :) | 13:27 |
| efried | eh? | 13:28 |
| mordred | oh. nope. sorry. still coffeeing | 13:28 |
| mordred | there are two different issues - one of them is solved by a requests.Sesison - but we still need to release secondary project support | 13:28 |
| mordred | I'm all caught up now | 13:28 |
| * mordred goes to fix ost patch | 13:28 | |
| *** bswartz has joined #openstack-infra | 13:29 | |
| efried | mordred By taking out that ValueError and checking the get_service_data_for_project() return for None, right? | 13:29 |
| pabelanger | mnaser: clarkb: fungi: Wondering if we should consifer rolling vexxhost back down to 10nodes for the moment. http://status.openstack.org/elastic-recheck/#1686542 is showing a fair number of job timeouts on the cloud. Maybe we can see if there is some pattern to them | 13:30 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Set undercloud nameserver to localhost 127.0.0.1 https://review.openstack.org/494545 | 13:30 |
| efried | Though I don't understand why that should have given None at the moment, since all the types should be valid. Unless sta hasn't finished publishing yet. | 13:30 |
| efried | or is that the requests.Session thing? | 13:30 |
| AJaeger | efried: still around - but busy right now. Thanks for diving into this! | 13:30 |
| efried | AJaeger cool, I think we've got it covered. Too many cooks, if anything :) | 13:31 |
| fungi | infra-root: reminder, i'm disappearing here in a few minutes and travelling through a week from tomorrow, so will have intermittent internet access at odd hours (for me anyway) but will try to check in when i can in case there's anything urgent | 13:31 |
| mnaser | pabelanger eek i had a search in logstash for Job timed out, result: FAILURE | 13:32 |
| mnaser | i guess that's not as accurate | 13:32 |
| pabelanger | fungi: enjoy! | 13:32 |
| AJaeger | frickler, efried, mordred : To get a change in, the post job of service-type needs to publish content first! | 13:32 |
| AJaeger | and we have a long queue of post-jobs ;( | 13:32 |
| AJaeger | mordred: https://review.openstack.org/#/c/494543 needs to be done for api-site | 13:33 |
| mnaser | pabelanger i'm in favour i guess, i see around ~500 jobs completed in the past day with 30 timeouts | 13:33 |
| mnaser | if that helps things | 13:33 |
| frickler | AJaeger: post queue is down to one after the morning rush of periodic has been handled | 13:34 |
| mordred | AJaeger: oh - wrong repo? | 13:34 |
| AJaeger | efried, mordred : I'll be mainly offline the next hours. glad to see you moving forward here. I just gave all information I noticed... | 13:34 |
| *** LindaWang has joined #openstack-infra | 13:34 | |
| AJaeger | mordred: yes, wrong repo. developer.o.o .htaccess lives in api-site repo, not openstack-manual | 13:35 |
| pabelanger | mnaser: clarkb: http://logs.openstack.org/59/493259/10/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/e8e97e5/ would be a good job to audit. that is gate pipeline non tripleo | 13:35 |
| mordred | AJaeger: ok. cool | 13:35 |
| pabelanger | http://logs.openstack.org/59/493259/10/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/e8e97e5/console.html#_2017-08-15_21_06_26_704991 | 13:35 |
| mordred | AJaeger: the project-info files in openstack-manual - does api-site have similar? I want to use os-service-types to set up redirects from all aliases to all official types | 13:35 |
| pabelanger | setup workspace was slow for some reason | 13:35 |
| mordred | efried: your suggestion for the uniitest change won't work - you can't have a set of dicts | 13:36 |
| *** Guest53762 is now known as dansmith | 13:36 | |
| efried | mordred Which change? | 13:36 |
| efried | mordred oh, okay. | 13:37 |
| efried | Yeah, that makes sense. | 13:37 |
| efried | But in principle, I still don't like relying on ordering. | 13:37 |
| efried | Though I'm not gonna hold it up for that. | 13:37 |
| efried | mordred Just resolve the merge conflict and I'm +2 | 13:38 |
| mordred | efried: although now that makes me want to return hashable dict-like objects and make that method return a set of them | 13:38 |
| mordred | but yeah - let's not do that for now :) | 13:38 |
| AJaeger | mordred: api-site uses tools/www-generator.py with Jinja - but openstack-manuals is much more sophisticated. | 13:38 |
| efried | mordred You can add a TODO | 13:38 |
| AJaeger | mordred: So, should be doable, just was not done yet. | 13:38 |
| pabelanger | mnaser: http://logs.openstack.org/59/493259/10/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/e8e97e5/logs/dstat-csv_log.txt.gz if you want to graph dstat | 13:38 |
| mordred | AJaeger: ok - so they both have www-generator support. I'll make patches for them | 13:38 |
| efried | mordred I'll fix up https://review.openstack.org/#/c/494456/ | 13:38 |
| AJaeger | thanks, mordred | 13:38 |
| mordred | AJaeger: thanks! we'll get this all rolled out real quick | 13:39 |
| AJaeger | mordred, efried : Thanks! | 13:39 |
| pabelanger | mnaser: load seems to be climbing through out the job | 13:39 |
| * AJaeger will be back later... | 13:39 | |
| *** EricGonczer_ has joined #openstack-infra | 13:39 | |
| mnaser | pabelanger i wouldnt be surprised by a noisy neighbour thing | 13:39 |
| *** felipemonteiro has joined #openstack-infra | 13:39 | |
| mnaser | it's really hard to schedule VMs properly without being able to predict workloads | 13:40 |
| *** amoralej is now known as amoralej|lunch | 13:40 | |
| *** felipemonteiro_ has joined #openstack-infra | 13:40 | |
| dtroyer | Howdy Infra-land! According to the Project Driver's Guide (https://docs.openstack.org/infra/manual/drivers.html#merge-commits) I should have a python-openstackclient-release group to allow to merge master into the feature/osc4 branch, one does not exist. I would be fine with using python-openstackclient-core for this if that's an option. | 13:40 |
| mordred | efried: also - I'm totally going to make an ost change to use ordereddicts - because the sync patches are unreadable | 13:41 |
| *** cdent has joined #openstack-infra | 13:41 | |
| * cdent waves | 13:41 | |
| openstackgerrit | Eric Fried proposed openstack-infra/project-config master: Switch to os-service-types https://review.openstack.org/494456 | 13:41 |
| *** alexchadin has quit IRC | 13:41 | |
| efried | mordred ^ fixes that NoneType error; but the cause thereof is still going to be sta not having published yet. | 13:42 |
| *** jpena|lunch is now known as jpena | 13:42 | |
| *** lbragstad_ is now known as lbragstad | 13:42 | |
| efried | cdent Howdy. We're unraveling the spagetti. You may want to check eavesdrop since about an hour ago. | 13:42 |
| cdent | thanks efried | 13:43 |
| efried | cdent First thing that needs to land is https://review.openstack.org/#/c/494456/ | 13:43 |
| cdent | reading that now | 13:43 |
| efried | cdent Then https://review.openstack.org/#/c/494529/ | 13:43 |
| mordred | efried: -1 left- add a Session so that we're consuming latest data | 13:44 |
| efried | cdent Then https://review.openstack.org/#/c/494533/1 | 13:44 |
| efried | mordred rgr | 13:44 |
| *** felipemonteiro has quit IRC | 13:44 | |
| efried | mordred Just any ol requests.Session() ? | 13:44 |
| mordred | yup | 13:44 |
| *** dhajare has quit IRC | 13:45 | |
| mordred | once we get unwedged here - we should to our todo-list a gate job that validates changes to service-types-authority will not break project-config | 13:45 |
| efried | mordred prolly no need to bother caching that guy | 13:45 |
| efried | mordred ++ | 13:45 |
| mordred | efried: agree | 13:45 |
| efried | mordred only_remote=True? | 13:45 |
| mordred | efried: yah - if we can't fetch that file, we should fail the gate job - stale data for the gate could be bad | 13:46 |
| openstackgerrit | Eric Fried proposed openstack-infra/project-config master: Switch to os-service-types https://review.openstack.org/494456 | 13:46 |
| * cdent mumbles something vague about coupling | 13:47 | |
| *** vaidy has joined #openstack-infra | 13:49 | |
| *** b3nt_pin is now known as beagles | 13:49 | |
| *** eharney has joined #openstack-infra | 13:49 | |
| cdent | efried: the first change will allow https://review.openstack.org/#/c/494533/ to pass tests, yes? that’s failing on secondaries | 13:50 |
| efried | cdent That's the idea, yes. | 13:50 |
| cdent | ✔ | 13:50 |
| efried | uh, actually cdent we need both the first and second changes. | 13:50 |
| efried | cdent See my comment on https://review.openstack.org/#/c/494529/ | 13:51 |
| cdent | efried: i was referring to the test failures, specifically | 13:51 |
| efried | cdent That guy should have three test failures. Two for secondaries, one for clustering. | 13:51 |
| efried | hm | 13:51 |
| efried | it doesn't. | 13:51 |
| efried | oh. Because Depends-On :) | 13:52 |
| efried | So yeah, what I said holds, just that the Depends-On is covering #2 | 13:52 |
| numans | EmilienM, Hi, can you please have a look at this when you get some time - https://review.openstack.org/#/c/490622/ | 13:53 |
| efried | mordred https://review.openstack.org/#/c/494456/ good now? | 13:53 |
| *** alexchadin has joined #openstack-infra | 13:56 | |
| *** ramishra has quit IRC | 13:56 | |
| *** ykarel has quit IRC | 13:56 | |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci master: WIP: containers periodic test https://review.openstack.org/475747 | 13:56 |
| *** ramishra has joined #openstack-infra | 13:57 | |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci master: WIP: containers periodic test https://review.openstack.org/475747 | 13:59 |
| *** bobh has joined #openstack-infra | 13:59 | |
| *** EricGonc_ has joined #openstack-infra | 14:00 | |
| *** ihrachys has joined #openstack-infra | 14:00 | |
| EmilienM | numans: ok | 14:01 |
| *** EricGonczer_ has quit IRC | 14:01 | |
| mordred | efried: yes. looks good | 14:01 |
| *** edmondsw has joined #openstack-infra | 14:02 | |
| efried | mordred Except we do still need a new os-service-types release. | 14:02 |
| mordred | yes | 14:02 |
| mordred | did you submit the thing? | 14:02 |
| mordred | oh - you can't until we land those patches | 14:02 |
| *** armax has joined #openstack-infra | 14:02 | |
| efried | mordred Yeah, we need another core review on https://review.openstack.org/#/c/493366/ I guess. cdent ? | 14:03 |
| * cdent looks | 14:03 | |
| mordred | efried: so - we're ACTUALLY going to need two releases today to do this properly - and then we'll definitely get gate jobs in place to make sure we don't do this to ourselves again | 14:04 |
| EmilienM | numans: =1 | 14:04 |
| EmilienM | +1 even | 14:04 |
| efried | mordred Two releases of what? | 14:04 |
| mordred | oh- no - we can go it with one | 14:04 |
| *** slaweq has joined #openstack-infra | 14:05 | |
| mordred | the ost release will include the clustering rename before we land the re-revert | 14:05 |
| mordred | so what we ACTUALLY want to do is squash the use-ost and clustering rename patches | 14:05 |
| mordred | into one patch | 14:05 |
| mordred | that patch should then turn green when we make an ost release | 14:05 |
| efried | umno, because now we're using latest sta data from the use-ost patch. | 14:05 |
| mordred | oh. hahaha | 14:06 |
| cdent | efried, mordred: is efried’s concern on test ordering irrelevant? | 14:06 |
| mordred | cdent: it is for now, yes | 14:06 |
| * cdent approves | 14:06 | |
| mordred | efried: did youmake the release patch already or want me to? | 14:07 |
| efried | mordred I've got it open, but will need the final commit hash before I propose it. | 14:07 |
| mordred | k. cool. | 14:07 |
| *** dhajare has joined #openstack-infra | 14:08 | |
| efried | Which might as well include https://review.openstack.org/#/c/494550/ as well. <== cdent | 14:08 |
| efried | (Thanks for jumping in to help here, cdent) | 14:08 |
| mordred | cdent: I have also added to my list a) getting the sync job fixed b) updating sta publish to use ordereddict from yaml so that the json files are emitted in the source order from yaml so that patches proposed to update ost are reviewable | 14:09 |
| cdent | so in this change, we’re still int he state where senlin == resource-cluster, right? | 14:10 |
| efried | ++ | 14:10 |
| mordred | cdent: c) adding gate job to validate sta changes against project-config repo d) updating openstack-manuals and api-site to use ost in their validation and redirect generation | 14:10 |
| mordred | cdent: in the second ost change senlin is clustering (that's a manual sync from before the revert) | 14:10 |
| *** rbrndt has joined #openstack-infra | 14:10 | |
| mordred | so ost will have the data in it we intend to be correct | 14:11 |
| mordred | efried: actually - projects.yaml patches are broken no matter what because of placement too right? | 14:11 |
| efried | cdent mordred Actually, it looks like the second ost change still has resource-cluster as the official service type. | 14:11 |
| cdent | yeah, it does | 14:12 |
| mordred | ok. let's go ahead and land the sta change so that that is correct | 14:12 |
| mordred | then do a proper sync from the actual published data | 14:12 |
| *** apetrich_ has joined #openstack-infra | 14:12 | |
| * cdent holds fire | 14:12 | |
| mordred | if projects.yaml changes continue to be a little borked for the next 30 minutes that's just gonna be life | 14:12 |
| *** trown is now known as trown|brb | 14:12 | |
| efried | mordred So we *are* going t odo two releases. | 14:12 |
| efried | to do* | 14:13 |
| mordred | nope. we're just going to accept that projects.yaml chagnes are busted at the moment | 14:13 |
| mordred | and we're going to do the fix correctly rather than quickly | 14:13 |
| efried | mordred In that case we are indeed going to have to squash the two project-config changes | 14:13 |
| mordred | efried: that's fine | 14:13 |
| mordred | cdent: if you can +A https://review.openstack.org/#/c/494529 - that will make service-types-authority correct - and when it publishes we can sync from the published file | 14:14 |
| mordred | efried: (can you un-wip that?) | 14:14 |
| mordred | also - I'm glad we hit this issue in this particular context - it's a good lesson and shows us some holes before we get ost added to things as dependencies | 14:15 |
| cdent | better now than later | 14:15 |
| mordred | since this is, although annoying, not as fatal as other places | 14:15 |
| mordred | cdent: exactly | 14:15 |
| efried | mordred Un -Wed | 14:17 |
| *** trown|brb is now known as trown | 14:17 | |
| cdent | okay, +Wd | 14:17 |
| mordred | woot | 14:17 |
| efried | mordred cdent With a new comment helpfully describing the new order of events. | 14:17 |
| cdent | efried: thanks for making the notes with the plan. is good. | 14:17 |
| cdent | jinxish | 14:17 |
| efried | :) | 14:17 |
| *** ykarel has joined #openstack-infra | 14:20 | |
| mordred | k. sta patch is queued for a post job node | 14:23 |
| efried | mordred You can cheat the sync, though, nah? | 14:24 |
| mordred | yah | 14:24 |
| efried | Cause the ost sync thing is still busted? | 14:25 |
| mordred | yah. one sec | 14:25 |
| *** amoralej|lunch is now known as amoralej | 14:25 | |
| mordred | efried, cdent: k. https://review.openstack.org/494550 is updated | 14:26 |
| numans | EmilienM, thanks | 14:27 |
| efried | mordred sha looks good, but the date stamp is gonna be off. That means once the bot is working... oh, it still won't try to republish, cause it ignores both the sha and the date stamp. Cool. | 14:28 |
| efried | mordred +2 | 14:28 |
| cdent | mordred: I’m +2 as well, shall I +W it as well, we ready for that? | 14:30 |
| *** alexchadin has quit IRC | 14:32 | |
| *** hongbin has quit IRC | 14:32 | |
| mordred | yup | 14:33 |
| mordred | also - fwiw - I went ahead and did https://review.openstack.org/494581 Use OrderedDict in yaml to json transformation | 14:33 |
| *** martinkopec has quit IRC | 14:33 | |
| mordred | the ordereddict/yaml stuff in there is actually lifted from project-config - we do the same thing to do normalization of the giant projects.yaml file :) | 14:33 |
| *** gmann has joined #openstack-infra | 14:34 | |
| cdent | as ya do | 14:34 |
| efried | holy crap, is all that really necessary? I thought json.dumps had an option to sort keys or something. | 14:34 |
| efried | Yeah, sort_keys | 14:35 |
| efried | mordred ^ why doesn't dumps(sort_keys=True) work? | 14:36 |
| mordred | efried: heh. I suppose we could do that too :) | 14:37 |
| *** isaacb has joined #openstack-infra | 14:37 | |
| efried | mordred I suppose the difference is whether the keys are ordered alphabetically versus for human consumption. | 14:38 |
| mordred | efried: I honestly think the only time it'll ever matter is in eyeballing ost sync patches - so as long as the result is that we can see only what has changed and not random dict reordering, I think we're good | 14:39 |
| mordred | efried: updated- great suggestion | 14:39 |
| efried | mordred Agree. +2 | 14:39 |
| *** ralonsoh has quit IRC | 14:39 | |
| efried | mordred cdent https://review.openstack.org/494586 <== I think that'll wind up being the right hash; there should be no merge commit necessary for https://review.openstack.org/#/c/494550/ | 14:40 |
| *** alexchadin has joined #openstack-infra | 14:40 | |
| cdent | we ready for trigger on 494581 | 14:42 |
| cdent | ? | 14:42 |
| cdent | mordred, efried ^ | 14:42 |
| cdent | because it looks good to me | 14:42 |
| *** isaacb has quit IRC | 14:43 | |
| mordred | cdent: yah. | 14:43 |
| cdent | done | 14:43 |
| cdent | i can do this all day | 14:44 |
| *** baoli has quit IRC | 14:45 | |
| *** jbadiapa_ has quit IRC | 14:45 | |
| *** baoli has joined #openstack-infra | 14:48 | |
| *** alexchadin has quit IRC | 14:51 | |
| *** alexchadin has joined #openstack-infra | 14:52 | |
| *** ramishra has quit IRC | 14:52 | |
| *** alexchadin has quit IRC | 14:52 | |
| *** ramishra has joined #openstack-infra | 14:52 | |
| *** alexchadin has joined #openstack-infra | 14:53 | |
| *** alexchadin has quit IRC | 14:53 | |
| *** ykarel has quit IRC | 14:53 | |
| *** alexchadin has joined #openstack-infra | 14:53 | |
| *** jeblair has joined #openstack-infra | 14:55 | |
| *** alexchadin has quit IRC | 14:56 | |
| *** loquacities has quit IRC | 14:56 | |
| *** Swami has joined #openstack-infra | 14:57 | |
| *** xyang1 has joined #openstack-infra | 14:57 | |
| *** tjones has joined #openstack-infra | 14:59 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: Revert "TEMP - Disable voting on tripleo upgrade jobs" https://review.openstack.org/494596 | 14:59 |
| *** Guest48406 is now known as Odd_Bloke | 14:59 | |
| *** Odd_Bloke has quit IRC | 15:00 | |
| *** Odd_Bloke has joined #openstack-infra | 15:00 | |
| *** udesale has quit IRC | 15:00 | |
| *** raissa_ is now known as raissa | 15:00 | |
| *** raissa is now known as Guest2133 | 15:00 | |
| *** Guest2133 is now known as raissa_ | 15:01 | |
| *** liujiong has quit IRC | 15:02 | |
| *** LindaWang has quit IRC | 15:02 | |
| *** ff is now known as flaper87 | 15:04 | |
| *** flaper87 has quit IRC | 15:04 | |
| *** flaper87 has joined #openstack-infra | 15:04 | |
| *** dougwig has joined #openstack-infra | 15:06 | |
| pabelanger | ianw: clarkb: is there a list of mirrors that were rotated out last night to xenial? | 15:09 |
| pabelanger | mordred: mind looking at https://review.openstack.org/493329/ (glean) context is glean@service.d/override.conf patch: https://review.openstack.org/493665/ | 15:10 |
| *** srobert has joined #openstack-infra | 15:11 | |
| *** ccamacho has quit IRC | 15:16 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Remove duplicate parameter_defaults section https://review.openstack.org/494603 | 15:17 |
| *** zoli has joined #openstack-infra | 15:18 | |
| pabelanger | zoli: we should be able to continue here | 15:18 |
| zoli | pabelanger, thanks | 15:19 |
| zoli | summary: I am trying to build reverse proxy to access repos via HTTPS | 15:19 |
| pabelanger | zoli: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/templates/mirror.vhost.erb today is our reverse proxy cache for regional mirrors. Right now we are under a freeze since we are trying to upgrade mirrors to xenial. | 15:19 |
| pabelanger | zoli: maybe you can explain the issue you are trying to solve | 15:20 |
| zoli | so each job does not have to get the RPMs by itself | 15:20 |
| mordred | pabelanger: done | 15:20 |
| pabelanger | zoli: sure, which RPMs in this case | 15:20 |
| zoli | pabelanger, e.g. https://trunk.rdoproject.org/centos7/6d/4f/6d4fc217095f07e0c13cf2db99b18add461f8bb1_9077dc93/ | 15:21 |
| *** rkukura has joined #openstack-infra | 15:21 | |
| zoli | just wanted to ask whether there sis something similar in place | 15:22 |
| zoli | not to reinvent wheel | 15:22 |
| pabelanger | zoli: okay, so we already reverse proxy rdoproject, so you need to source /etc/ci/mirror_info.sh on our nodes and use the NODEPOOL_RDO_PROXY variable: http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/scripts/configure_mirror.sh#n100 | 15:22 |
| *** lin_yang has joined #openstack-infra | 15:23 | |
| pabelanger | zoli: you'll need to append /centos7/6d/4f/6d4fc217095f07e0c13cf2db99b18add461f8bb1_9077dc93/ to the variable, to create http://mirror.dfw.rax.openstack.org:8080/rdo/centos7/6d/4f/6d4fc217095f07e0c13cf2db99b18add461f8bb1_9077dc93/ | 15:23 |
| pabelanger | zoli: currently puppet-openstack-infra and some tripleo jobs do this, which EmilienM also knows about | 15:24 |
| zoli | ok, thanks | 15:25 |
| pabelanger | mordred: danke | 15:25 |
| clarkb | pabelanger: ya there is an etherpad | 15:25 |
| clarkb | give me a minute to get a link | 15:25 |
| * zoli will read the stuff linked above and eventually come back with questions :) | 15:25 | |
| *** katkapilatova has left #openstack-infra | 15:26 | |
| *** jpena is now known as jpena|off | 15:26 | |
| pabelanger | clarkb: np, I see it now. Thanks | 15:27 |
| clarkb | https://etherpad.openstack.org/p/mirror-xenial | 15:27 |
| pabelanger | k, citycloud is still pending | 15:27 |
| *** dangers_away is now known as danger | 15:27 | |
| clarkb | I finally got vanilla booted and happy yesterday evening and haven't switched dns, I will switch dns on that shortly then boot chocolate | 15:27 |
| clarkb | I did want to check e-r and logstash first though based on the rax data | 15:28 |
| pabelanger | ack | 15:28 |
| openstackgerrit | Sam Yaple proposed openstack-infra/project-config master: Add additional LOCI repos https://review.openstack.org/492637 | 15:28 |
| clarkb | the pip hash mismatch error has dropped off almost completely so that looks good | 15:28 |
| clarkb | I guess check the apache log on the server really quick too | 15:29 |
| clarkb | ord logs look clean after a quick scan | 15:31 |
| *** Swami has quit IRC | 15:31 | |
| *** logan- has joined #openstack-infra | 15:31 | |
| *** rcernin has quit IRC | 15:32 | |
| clarkb | pabelanger: and the yum no more mirrors error has dropped off in rax almost completely so I think we are good to keep replacing them | 15:32 |
| clarkb | I'll work on finishing up infracloud this morning | 15:32 |
| pabelanger | clarkb: agree! Some of those failures are DNS related, but working to fix that in tripleo-ci and hopefully our new glean release | 15:33 |
| clarkb | pabelanger: did you have a python patch you want me to test when I boot chocolate? | 15:34 |
| *** e0ne has quit IRC | 15:34 | |
| clarkb | pabelanger: maybe we should get that done really quickly so we don't have to all have different hacked up launch node scripts :) | 15:34 |
| *** HeOS has quit IRC | 15:35 | |
| pabelanger | clarkb: ya, will defer to your patch now | 15:36 |
| *** camunoz has quit IRC | 15:37 | |
| *** inc01 has quit IRC | 15:37 | |
| *** inc0 has joined #openstack-infra | 15:38 | |
| *** krotscheck has joined #openstack-infra | 15:39 | |
| *** logan- has quit IRC | 15:40 | |
| *** krotscheck has left #openstack-infra | 15:40 | |
| *** logan- has joined #openstack-infra | 15:40 | |
| clarkb | mine is super hacky I can work on something reusable | 15:41 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Switch to os-service-types https://review.openstack.org/494533 | 15:41 |
| mordred | efried, pabelanger, frickler: ^^ updated that to include the senlin clustering change - also ost 1.1.0 is now released so that should be green | 15:42 |
| efried | mordred How/where did ost 1.1.0 get released? I missed it. | 15:43 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: External Calendar Sync https://review.openstack.org/487683 | 15:44 |
| jeblair | infra-root: i'm approving the internap removal (i verified all images/nodes are gone) | 15:44 |
| *** tongli has joined #openstack-infra | 15:44 | |
| *** ccamacho has joined #openstack-infra | 15:44 | |
| pabelanger | jeblair: ++ | 15:44 |
| *** ccamacho has left #openstack-infra | 15:44 | |
| *** rkukura has quit IRC | 15:44 | |
| pabelanger | mordred: +2 | 15:44 |
| *** tongli has quit IRC | 15:44 | |
| *** coolsvap has quit IRC | 15:44 | |
| numans | AJaeger, pabelanger Hi, can you please add this to your review queue - https://review.openstack.org/#/c/490622/ | 15:45 |
| *** alexchadin has joined #openstack-infra | 15:45 | |
| mordred | efried: we switched to indendependnt releases, so I pushed a tag and zuul published it to pypi: https://pypi.python.org/pypi/os-service-types | 15:48 |
| *** gmann has quit IRC | 15:48 | |
| mordred | efried: incidentally, I spun up https://etherpad.openstack.org/p/os-service-types-checks to track the various things we learned need to be checked/updated | 15:49 |
| efried | mordred Okay, that doesn't happen anywhere in openstack-land? | 15:49 |
| efried | nice | 15:49 |
| mordred | efried: for independent releases we have them set up such that people can push signed tags and that triggers the jobs that build and publish (this is the way all releases happened before the openstack/releases repo) - in fact, the automation around opentsack/releases also just winds up pushing a signed tag and triggering the same jobs - so it's really all about whether those tags come from the | 15:50 |
| mordred | release team automation or from a project-specific release team | 15:51 |
| efried | mordred Okay; "pushing a tag" is not a thing I'm familiar with. | 15:51 |
| efried | I vaguely know what git tags are; but I've never done anything with git push that doesn't happen via git-review. Didn't know that was possible for gerrit-governed projects. | 15:52 |
| *** apetrich_ has quit IRC | 15:52 | |
| *** udesale has joined #openstack-infra | 15:55 | |
| mordred | efried: it is for tags (I've LONG wanted gerrit to have teh ability to propose a tag like we propose a commit - but alas that's not a thing) | 15:55 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Use cached branch list in dynamic config https://review.openstack.org/494618 | 15:55 |
| efried | got it. | 15:56 |
| mordred | efried: I mean - it's actually possible to directly push to gerrit projects - we just don't allow that in openstack and instead enforce code review and CI jobs | 15:56 |
| mordred | efried: I haven't wanted gerrit to have the ability to propose a tag enough that I've been willing to write the java to implement it ;) | 15:56 |
| efried | mordred Oh, I feel you there. Loathe java. | 15:57 |
| *** dklyle has joined #openstack-infra | 15:57 | |
| *** udesale has quit IRC | 15:58 | |
| *** udesale has joined #openstack-infra | 15:58 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove internap in favor of inap https://review.openstack.org/493074 | 15:58 |
| clarkb | ERROR: No domain with a name or ID of 'openstack.org' exists. | 15:59 |
| clarkb | oh did I not source the third file... | 15:59 |
| clarkb | that was it ignore the noise | 15:59 |
| *** markus_z has quit IRC | 15:59 | |
| *** trown is now known as trown|lunch | 16:00 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Create a whitelist for /etc configs https://review.openstack.org/493973 | 16:00 |
| mgagne | clarkb: we (inap) will perform a very short maintenance to switch a database server, API operations will be affected. The maintenance should take about ~5m. Do you have a preferred time for this maintenance? | 16:01 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: External Calendar Sync https://review.openstack.org/487683 | 16:01 |
| clarkb | mgagne: that should be fairly transparent to us since only api operations are affected | 16:01 |
| mgagne | clarkb: so any time is good for you? | 16:02 |
| clarkb | mgagne: so I don't think it matters too much. If you'd like us to be around to test things then pacific time business hours are good for me :) | 16:02 |
| *** shardy has quit IRC | 16:02 | |
| mgagne | ok, how about in ~30m ? | 16:02 |
| clarkb | mgagne: that should be fine. Let's double check with zuul though since they are using inap now for testing v3 I think | 16:03 |
| clarkb | jeblair: pabelanger ^ does that work for you? | 16:03 |
| pabelanger | no issue for me | 16:03 |
| pabelanger | we still have infracloud for zuulv3 | 16:03 |
| *** egonzalez has quit IRC | 16:04 | |
| clarkb | while I'm editing dns is there any reason to keep our old osic dns records around? | 16:05 |
| clarkb | infra-root good with me deleting the osic mirror records? | 16:05 |
| *** srobert has quit IRC | 16:05 | |
| cloudnull | RIP osic.xyz :'( | 16:06 |
| jeblair | clarkb, mgagne: sounds fine | 16:06 |
| cloudnull | hahaha | 16:06 |
| jeblair | clarkb: want to delete internap dns record too? | 16:06 |
| jeblair | i'll go actually delete the old internap mirror now | 16:06 |
| clarkb | jeblair: yup can do | 16:08 |
| *** rcernin has joined #openstack-infra | 16:08 | |
| mordred | clarkb: yah | 16:08 |
| jeblair | #status log deleted mirror.mtl01.internap.openstack.org (internap -> inap rename) | 16:10 |
| openstackstatus | jeblair: finished logging | 16:10 |
| clarkb | jeblair: mgagne there are two, one for nyj and one for mtl, the nyj one is long gone right? | 16:10 |
| jeblair | clarkb: afaik yes | 16:10 |
| jeblair | i haven't seen any active nyj stuff anywhere in my work to rename internap | 16:10 |
| clarkb | ya I think we migrated regions. I will clean that record too | 16:11 |
| jeblair | thx | 16:11 |
| clarkb | and maybe after doing infracloud mirror upgrades go through our dns record list and see if we can clean anything else out | 16:11 |
| jeblair | probably a good idea | 16:11 |
| jeblair | clarkb: oh, can probably drop zl07, zl08, zl09 | 16:11 |
| jeblair | sorry, i should have done that yesterday | 16:12 |
| pabelanger | jeblair: clarkb: Ya, we should likely land https://review.openstack.org/492301/ too. I'll have to rebase first | 16:12 |
| pabelanger | removes internap-nyj01 from clouds.yaml | 16:12 |
| clarkb | jeblair: ok cleaning those now | 16:12 |
| jeblair | pabelanger: that probably needs a rebase on master to get the inap change | 16:12 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Remove internap-nyj01 https://review.openstack.org/492301 | 16:13 |
| jeblair | ah yeah you just said that :) | 16:13 |
| pabelanger | :) | 16:13 |
| clarkb | jeblair: done | 16:14 |
| *** jpich has quit IRC | 16:14 | |
| *** srobert has joined #openstack-infra | 16:16 | |
| clarkb | new infracloud mirror is slowly seeing use. Should be able to delete the old server in about an hour | 16:16 |
| clarkb | *vanilla | 16:16 |
| pabelanger | cool | 16:16 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Remove internap clouds https://review.openstack.org/494629 | 16:17 |
| *** Apoorva has joined #openstack-infra | 16:18 | |
| jeblair | pabelanger: +2d your change and put that on top of it ^ | 16:18 |
| *** Apoorva has quit IRC | 16:19 | |
| *** Apoorva has joined #openstack-infra | 16:19 | |
| *** ramishra has quit IRC | 16:20 | |
| *** tumbarka__ has joined #openstack-infra | 16:21 | |
| openstackgerrit | Merged openstack-infra/glean master: Simplify glean.sh options https://review.openstack.org/493329 | 16:21 |
| *** ramishra has joined #openstack-infra | 16:22 | |
| *** alexchadin has quit IRC | 16:23 | |
| *** sree has quit IRC | 16:24 | |
| *** spzala has quit IRC | 16:25 | |
| *** sree has joined #openstack-infra | 16:25 | |
| *** iyamahat has joined #openstack-infra | 16:25 | |
| *** spzala has joined #openstack-infra | 16:25 | |
| *** dizquierdo has quit IRC | 16:26 | |
| *** sree has quit IRC | 16:29 | |
| *** udesale__ has joined #openstack-infra | 16:29 | |
| *** udesale has quit IRC | 16:29 | |
| *** udesale__ has quit IRC | 16:30 | |
| *** spzala has quit IRC | 16:30 | |
| *** udesale has joined #openstack-infra | 16:30 | |
| *** r-daneel has joined #openstack-infra | 16:32 | |
| *** dhajare has quit IRC | 16:32 | |
| *** sambetts is now known as sambetts|afk | 16:33 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Install python2 on newer Ubuntus https://review.openstack.org/494638 | 16:34 |
| clarkb | pabelanger: I'm going to test ^ on chocolate | 16:34 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: External Calendar Sync https://review.openstack.org/487683 | 16:36 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Install python2 on newer Ubuntus https://review.openstack.org/494638 | 16:40 |
| *** jtomasek_ has quit IRC | 16:41 | |
| *** sshnaidm is now known as sshnaidm|off | 16:44 | |
| pabelanger | clarkb: ack | 16:45 |
| *** vponomaryov has quit IRC | 16:47 | |
| *** udesale has quit IRC | 16:50 | |
| *** derekh has quit IRC | 16:52 | |
| *** spzala has joined #openstack-infra | 16:54 | |
| *** lucasagomes is now known as lucas-afk | 16:54 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Return 404 on unknown tenants https://review.openstack.org/494642 | 16:54 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: External Calendar Sync https://review.openstack.org/487683 | 16:58 |
| pabelanger | clarkb: when you have time: http://logs.openstack.org/53/493953/3/gate/gate-tripleo-ci-centos-7-nonha-multinode-oooq/27ca6d9/console.html setup workspace hung for almost 3 hours on that job. Failed to git clone from zm07.o.o, but not sure why devstack-gate didn't see the return code and fail the job | 16:59 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: External Calendar Sync https://review.openstack.org/487683 | 17:00 |
| clarkb | pabelanger: d-g doesn't set -x | 17:00 |
| clarkb | er | 17:00 |
| clarkb | -e | 17:00 |
| pabelanger | clarkb: Ah, that explains it. Why reason why? | 17:00 |
| clarkb | I think it was just never done and now at this point would be a fairly large undertaking to correct | 17:00 |
| clarkb | in part because we expect the zuul ref fetches to fail in many cases | 17:00 |
| clarkb | (so you have to special case a bunch of things) | 17:01 |
| pabelanger | ya | 17:01 |
| pabelanger | that was in infracloud, likely networking issues still. | 17:01 |
| pabelanger | Did we ever contacts somebody at HPE? | 17:01 |
| clarkb | fungi sent email but I haven't seen a response | 17:02 |
| clarkb | I meant to go and run tcpdump again at some point just to see if they secretly fixed the switch problem | 17:02 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: External Calendar Sync https://review.openstack.org/487683 | 17:02 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Use sample environments for OVB configuration https://review.openstack.org/494643 | 17:04 |
| jeblair | pabelanger, clarkb: i feel like infracloud got better when we stopped image uploads and worse when we restarted them. it may be that their bulk is taking a substantial portion of our bandwidth. | 17:04 |
| clarkb | jeblair: ya | 17:04 |
| clarkb | jeblair: one way around that may be to stick a builder in infracloud just for those two clouds | 17:04 |
| Diabelko | hi \o | 17:04 |
| clarkb | since that way dib maintains a local cache and the upload is all local l2 | 17:04 |
| pabelanger | jeblair: clarkb: can we limit upload workers per region now? Maybe just decrease to 1 image at a time | 17:05 |
| clarkb | Diabelko: hello | 17:05 |
| clarkb | pabelanger: I'm not aware of that feature, I think we can limit the total number of uploads but not per region | 17:06 |
| pabelanger | clarkb: ya, regional builder could fix that | 17:06 |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: Set the undercloud_docker_registry_mirror for upstream jobs https://review.openstack.org/494644 | 17:07 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci master: Use sample environments for OVB configuration https://review.openstack.org/494643 | 17:08 |
| Diabelko | could you guys point to me to part of the code (and config) which posts comments to Gerrit? | 17:09 |
| *** slagle has joined #openstack-infra | 17:09 | |
| Diabelko | I'm trying to have copy of status table in my CI and this part is clearly missing | 17:09 |
| *** xarses_ has joined #openstack-infra | 17:11 | |
| clarkb | Diabelko: the table is built by some javascript we inject into gerrit's html | 17:11 |
| clarkb | Diabelko: is your CI running against our gerrit or a different Gerrit server? if different server you'll need to get that js first | 17:12 |
| Diabelko | different gerrit and I've got that js already ;) | 17:12 |
| Diabelko | only commenting part missing | 17:12 |
| Diabelko | currently it displays current job status from zuul, but nothing from the comments - I assume they need to be formatted in a proper way | 17:13 |
| jeblair | Diabelko: do you have job_name_in_report=True in zuul.conf? | 17:15 |
| Diabelko | yes | 17:15 |
| Diabelko | but my Zuul comments are plaintext - no color in them and no formatting | 17:16 |
| Diabelko | your comments are preformatted | 17:16 |
| jeblair | Diabelko: can you show me what the comments look like? for example, "ssh review gerrit query 494642 --comments" produces this output: http://paste.openstack.org/show/618706/ | 17:17 |
| jeblair | (a gerrit query over ssh is the easiest way i know to see the plain text form of the comments) | 17:18 |
| *** russellb has quit IRC | 17:19 | |
| *** russellb has joined #openstack-infra | 17:19 | |
| Diabelko | mmm, that's not it then | 17:19 |
| Diabelko | I have the same comment format | 17:19 |
| Diabelko | back to investigating then :D | 17:19 |
| jeblair | Diabelko: i think the javascript *also* requires some special commentlinks in the gerrit config | 17:19 |
| jeblair | Diabelko: http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/review.pp#n168 | 17:19 |
| *** trown|lunch is now known as trown | 17:20 | |
| jeblair | Diabelko: basically, the commentlink there will translate that text format into HTML, and then the javascript will pull that html into the box | 17:20 |
| jeblair | Diabelko: so if you're seeing plaintext in the comments section of the webpage, that may be the missing piece | 17:20 |
| Diabelko | I have that in my Gerrit config, but apparently it never kicked in | 17:21 |
| Diabelko | but I see what's wrong now | 17:23 |
| Diabelko | thank you | 17:23 |
| jeblair | Diabelko: np! | 17:23 |
| *** alexchadin has joined #openstack-infra | 17:24 | |
| clarkb | pabelanger: ok chocolate is up so my change https://review.openstack.org/494638 seems to work if you want to get that in. I'm going to update dns records in chocolate now then swing around and clean up old mirror in vanilla | 17:24 |
| clarkb | 15.184.68.140 is new chocolate mirror if you want to poke at it | 17:24 |
| *** SamYaple has joined #openstack-infra | 17:25 | |
| *** tosky has quit IRC | 17:27 | |
| pabelanger | clarkb: left comment on 49438 | 17:30 |
| pabelanger | 494638* | 17:30 |
| clarkb | pabelanger: thanks will clean that up in a moment. Chocolate should start using new mirror now as I just updated the main dns record | 17:31 |
| pabelanger | clarkb: confirmed, I see the new mirror via HTTP | 17:32 |
| *** EricGonc_ has quit IRC | 17:35 | |
| clarkb | pabelanger: is there a reason to use python-minimal over python? I think for us at least since we want pip and all that regular is fine? | 17:35 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Install python2 on newer Ubuntus https://review.openstack.org/494638 | 17:36 |
| clarkb | there it is updated | 17:36 |
| *** EricGonczer_ has joined #openstack-infra | 17:37 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Use cached branch list in dynamic config https://review.openstack.org/494618 | 17:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Allow multiple semaphore definitions within a project https://review.openstack.org/494650 | 17:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Reload configuration when branches are created or deleted https://review.openstack.org/494651 | 17:38 |
| pabelanger | clarkb: I think we can get away with python-minimal at this stage, but ianw added that logic | 17:39 |
| AJaeger | could I get a second +2 for a URL change in the commit message for the translation jobs, please? https://review.openstack.org/494389 | 17:39 |
| clarkb | pabelanger: do you think I should update my patch to use minimal? | 17:40 |
| pabelanger | clarkb: if you want, I know I used python-minimal without issues | 17:40 |
| jeblair | AJaeger: done | 17:40 |
| clarkb | ok let me do that just beacuse it is consistent with the old stuff | 17:41 |
| AJaeger | thanks, jeblair | 17:41 |
| *** srobert has quit IRC | 17:41 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Install python2 on newer Ubuntus https://review.openstack.org/494638 | 17:42 |
| clarkb | I'm tailing old vanilla mirror apache access logs to confirm it is no longer in use and should have it deleted shortly | 17:43 |
| *** dizquierdo has joined #openstack-infra | 17:43 | |
| pabelanger | clarkb: +2 | 17:44 |
| AJaeger | andreaf: are you fine with https://review.openstack.org/#/c/492797/ ? | 17:44 |
| *** vhosakot has joined #openstack-infra | 17:45 | |
| *** cdent has left #openstack-infra | 17:45 | |
| AJaeger | mnaser, numans , your rechecks are too early - we first need to merge https://review.openstack.org/494533 | 17:46 |
| AJaeger | that should happen any minute now... | 17:46 |
| mnaser | AJaeger sorry about that, i thought that was just a weird transient thing that was resolved by now, thank for letting me know | 17:46 |
| numans | AJaeger, oops.sorry. i was about to sign off | 17:47 |
| andreaf | AJaeger: yeah sorry about the late reply on that | 17:47 |
| *** slaweq_ has joined #openstack-infra | 17:47 | |
| AJaeger | numans, mnaser - wait for the failure and then recheck it again - by then we should be green again... | 17:47 |
| andreaf | AJaeger: +1 | 17:47 |
| mnaser | AJaeger ack will do | 17:47 |
| AJaeger | thanks, andreaf | 17:48 |
| numans | AJaeger, thanks. | 17:48 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: WIP Replace paste/webob with aiohttp https://review.openstack.org/494655 | 17:51 |
| AJaeger | numans, mnaser due to the high load, the change 494533 is still in change queue - so, best recheck tomorrow or after verifying that it's merged. | 17:51 |
| numans | AJaeger, ack | 17:51 |
| mnaser | AJaeger added myself to the ps so ill keep a close eye, thanks | 17:51 |
| openstackgerrit | Merged openstack-infra/project-config master: Update translation reviewing URL https://review.openstack.org/494389 | 17:52 |
| *** apetrich has joined #openstack-infra | 17:52 | |
| *** slaweq_ has quit IRC | 17:53 | |
| *** dizquierdo has quit IRC | 17:54 | |
| Diabelko | yes, so the part missing is proper link formatting and adding that in layout.yaml for a job | 17:55 |
| Diabelko | thanks again ;) | 17:55 |
| *** iyamahat has quit IRC | 17:56 | |
| *** iyamahat has joined #openstack-infra | 17:56 | |
| *** srobert has joined #openstack-infra | 17:56 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-zuul master: Remove zuul-executor pid file on stop https://review.openstack.org/494660 | 17:59 |
| *** SumitNaiksatam has joined #openstack-infra | 17:59 | |
| *** kjackal has quit IRC | 17:59 | |
| pabelanger | jeblair: clarkb: ^ should be our fix for leaked pid | 17:59 |
| pabelanger | that is how other init scripts handle it | 17:59 |
| jeblair | pabelanger: thanks | 18:00 |
| pabelanger | clarkb: did you want help with citycloud mirrors? | 18:00 |
| *** dprince has quit IRC | 18:00 | |
| clarkb | pabelanger: yes please | 18:01 |
| jeblair | pabelanger: ultimately, we should try to fix it in zuul, but it's probably not worth doing until we finish finger multiplexing | 18:01 |
| *** rkukura has joined #openstack-infra | 18:01 | |
| pabelanger | jeblair: sure | 18:01 |
| pabelanger | clarkb: okay, I'll look at sto2 now | 18:01 |
| *** jkilpatr has quit IRC | 18:02 | |
| *** jkilpatr has joined #openstack-infra | 18:03 | |
| *** jcoufal has quit IRC | 18:05 | |
| *** ekcs has joined #openstack-infra | 18:08 | |
| clarkb | old vanilla server is now gone | 18:09 |
| ekcs | Hi all. Any hints on what the problem might be here? http://logs.openstack.org/50/494350/1/check/gate-project-config-jenkins-project/834f96b/console.html#_2017-08-17_01_42_27_251859 | 18:12 |
| ekcs | https://review.openstack.org/#/c/494350/1/jenkins/jobs/congress.yaml | 18:12 |
| *** esberglu has quit IRC | 18:13 | |
| *** esberglu has joined #openstack-infra | 18:14 | |
| AJaeger | ekcs: wait until https://review.openstack.org/#/c/494533/ is merged - that one will fix it... | 18:14 |
| AJaeger | ekcs: so, nothing wrong on your side - we broke project-config and our currently recovering - please recheck once the change is in - or rebase on top of it. | 18:15 |
| ekcs | ah got it thanks a lot AJaeger ! | 18:15 |
| *** dprince has joined #openstack-infra | 18:16 | |
| openstackgerrit | Merged openstack-infra/project-config master: Create testpypi_secret secret for zuulv3 https://review.openstack.org/494276 | 18:16 |
| jeblair | pabelanger: ^ | 18:17 |
| *** jcoufal has joined #openstack-infra | 18:17 | |
| *** esberglu has quit IRC | 18:18 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Allow requesting secrets by a different name https://review.openstack.org/494343 | 18:19 |
| *** esberglu has joined #openstack-infra | 18:24 | |
| AJaeger | infra-root, http://grafana.openstack.org/dashboard/db/nodepool-internap http://grafana.openstack.org/dashboard/db/centos http://grafana.openstack.org/dashboard/db/devstack-failure-rate http://grafana.openstack.org/dashboard/db/fedora all exist - but we removed the files in project-config. Is there an easy way to remove those from grafana? | 18:27 |
| clarkb | pabelanger: looking at logstash I don't think the xenial upgrades are a silver bullet for the yum issues, but for pip's problems this looks really good | 18:27 |
| pabelanger | clarkb: that yum is query also include DNS failures, and other jobs not hitting our mirrors | 18:28 |
| clarkb | ah | 18:28 |
| pabelanger | clarkb: looking at tripleo jobs this morning, things are merging in gate much better | 18:29 |
| clarkb | good to know, the pip hash mismatch may be a better indicator then | 18:29 |
| pabelanger | I haven't seen any rax failures | 18:29 |
| openstackgerrit | Merged openstack-infra/puppet-zuul master: Install ara on the executors https://review.openstack.org/494260 | 18:29 |
| *** electrofelix has quit IRC | 18:30 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Remove publish-openstack-python-branch-tarball job https://review.openstack.org/494670 | 18:30 |
| *** Swami has joined #openstack-infra | 18:30 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Add publish-openstack-python-branch-tarball job https://review.openstack.org/494672 | 18:32 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Add publish-openstack-python-branch-tarball job https://review.openstack.org/494672 | 18:33 |
| pabelanger | jeblair: mordred: As I move publish-openstack-python-branch-tarball into project-config as a trusted job, I cannot remember what that offers us^ | 18:34 |
| jeblair | pabelanger: job needs to be in a trusted project in order to add tarball host to inventory | 18:35 |
| jeblair | pabelanger: ah, but it inherits from the upload job | 18:36 |
| pabelanger | jeblair: ya, do we still want that? | 18:36 |
| *** florianf has quit IRC | 18:37 | |
| pabelanger | I think if we parent to publish-openstack-artifacts it could be untrusted. But think I am missing something | 18:37 |
| jeblair | pabelanger: honestly, i think we may have just wanted to put it there so it's next to all its friends. | 18:37 |
| pabelanger | k | 18:38 |
| jeblair | pabelanger: however, i think it could theoretically go into openstack-zuul-jobs. | 18:39 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Allow requesting secrets by a different name https://review.openstack.org/494343 | 18:39 |
| *** signed8bit has joined #openstack-infra | 18:39 | |
| jeblair | mordred: thoughts ^? | 18:39 |
| *** signed8bit has quit IRC | 18:39 | |
| pabelanger | I like openstack-zuul-jobs for dynamic reconfiguration of those playbooks, less chance to wedge things. However, these jobs shouldn't change often | 18:42 |
| *** mattmceuen has joined #openstack-infra | 18:43 | |
| *** nicolasbock_ has quit IRC | 18:43 | |
| *** alexchadin has quit IRC | 18:46 | |
| *** alexchadin has joined #openstack-infra | 18:46 | |
| *** iyamahat has quit IRC | 18:47 | |
| clarkb | pabelanger: I'm gonna grab lunch then delete chocolate server and work on citycloud kna1 next (it appeas to have show up in e-r queries often) | 18:48 |
| AJaeger | efried, mordred, http://logs.openstack.org/33/494533/2/check/gate-project-config-jenkins-project/2e5f35c/console.html fails - that's the service type change | 18:49 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: Parent to publish-openstack-artifacts branch tarballs https://review.openstack.org/494677 | 18:50 |
| pabelanger | mordred: jeblair: I believe https://review.openstack.org/494677 and https://review.openstack.org/494672/ will do the same thing for our publish-openstack-python-branch-tarball jobs. First is untrusted, other is trusted. Both WIP, untested | 18:52 |
| *** jesusaurum is now known as jesusaur | 18:52 | |
| *** alexchadin has quit IRC | 18:56 | |
| *** phenixrizen has joined #openstack-infra | 18:56 | |
| *** ihrachys has quit IRC | 18:56 | |
| *** ihrachys has joined #openstack-infra | 18:57 | |
| mordred | AJaeger, efried: oh. hah. we made it use the remote data to make it more resilient ... but we've got a giant queue of post jobs so the service-types data hasn't been published yet :) | 18:58 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Fix zuul_stream callback dict modification https://review.openstack.org/494679 | 18:58 |
| *** EricGonc_ has joined #openstack-infra | 18:58 | |
| efried | mordred AJaeger Was just getting there, figured that was probably the case. | 18:58 |
| efried | So once that happens, we recheck, and then we're off to the races. | 18:59 |
| *** EricGonczer_ has quit IRC | 19:00 | |
| AJaeger | efried, mordred - ok... | 19:00 |
| *** baoli has quit IRC | 19:00 | |
| AJaeger | mordred: or can we promote that post job somehow? | 19:00 |
| mordred | pabelanger: can we promote post jobs? | 19:00 |
| *** amoralej is now known as amoralej|off | 19:01 | |
| efried | SSH into the server and hand-edit the file. | 19:01 |
| efried | (kidding) | 19:01 |
| pabelanger | mordred: don't believe so, they are pipeline independant | 19:01 |
| efried | They're not hierarchical? ;-) | 19:02 |
| mordred | AJaeger: we could also resubmit the project-config patch with remote_only turned off - then add a followup to do remote that we can land once the gate has settled | 19:03 |
| *** srobert has quit IRC | 19:04 | |
| efried | yeesh | 19:04 |
| *** dizquierdo has joined #openstack-infra | 19:05 | |
| AJaeger | mordred: not sure what's best. I'll be off now... | 19:05 |
| AJaeger | mordred, efried thanks for following up | 19:05 |
| mordred | efried: pushing up two patches ... | 19:06 |
| efried | mordred Okay; we'll need someone around to approve them. Sounds like AJaeger is out... | 19:07 |
| *** spzala has quit IRC | 19:07 | |
| *** hiller8 has joined #openstack-infra | 19:08 | |
| efried | mordred You'll need to rip out both session and only_remote. | 19:08 |
| *** baoli has joined #openstack-infra | 19:08 | |
| *** dizquierdo has quit IRC | 19:09 | |
| mordred | yup | 19:09 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Fetch service type data from the remote URL https://review.openstack.org/494533 | 19:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Switch to os-service-types https://review.openstack.org/494456 | 19:10 |
| *** iyamahat has joined #openstack-infra | 19:11 | |
| *** hiller8 has quit IRC | 19:13 | |
| SamYaple | AJaeger: mordred are those changes related to fixing http://logs.openstack.org/37/492637/4/check/gate-project-config-jenkins-project/58d9f2d/console.html#_2017-08-17_18_27_07_473123 ? | 19:14 |
| SamYaple | i dont think that error is related to my patch | 19:14 |
| mnaser | SamYaple same issue i think | 19:15 |
| SamYaple | cool | 19:15 |
| mnaser | there's a patch pending | 19:15 |
| mnaser | if mordred wants to confirm but i believe its once this lands - https://review.openstack.org/#/c/494533/ | 19:15 |
| mordred | SamYaple: yes | 19:16 |
| SamYaple | +1 | 19:16 |
| mordred | https://review.openstack.org/#/c/494456 actually at this point | 19:16 |
| mordred | but yes | 19:16 |
| mordred | and sorry this has drug on so long | 19:16 |
| SamYaple | excuses excuses | 19:17 |
| *** efried has quit IRC | 19:18 | |
| *** isaacb has joined #openstack-infra | 19:21 | |
| *** srobert has joined #openstack-infra | 19:24 | |
| *** slaweq_ has joined #openstack-infra | 19:24 | |
| *** efried has joined #openstack-infra | 19:24 | |
| *** eranrom has joined #openstack-infra | 19:25 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Allow configure-mirrors to do only homedir https://review.openstack.org/494684 | 19:30 |
| *** gomarivera has joined #openstack-infra | 19:30 | |
| *** kjackal has joined #openstack-infra | 19:32 | |
| pabelanger | jeblair: mordred: +3 on https://review.openstack.org/494288 to for internap mirror fixes | 19:36 |
| pabelanger | confirmed nodepool.cloud exists | 19:36 |
| *** gomarivera has quit IRC | 19:36 | |
| jeblair | pabelanger: thx | 19:37 |
| *** kjackal has quit IRC | 19:39 | |
| *** elmiko has joined #openstack-infra | 19:41 | |
| elmiko | hey folks, quick question. how long to upstream changes take to propogate into the content at specs.openstack.org ? | 19:42 |
| *** baoli has quit IRC | 19:43 | |
| *** baoli has joined #openstack-infra | 19:45 | |
| *** baoli has quit IRC | 19:45 | |
| openstackgerrit | Merged openstack-infra/project-config master: Use nodepool.cloud for zuul_site_mirror_fqdn https://review.openstack.org/494288 | 19:45 |
| *** trown is now known as trown|brb | 19:46 | |
| pabelanger | clarkb: mirror01.sto2.citycloud.o.o online and DNS has been update to point mirror.sto2.citycloud.o.o to it | 19:47 |
| *** e0ne has joined #openstack-infra | 19:47 | |
| mnaser | elmiko i believe that is a job in the post pipeline which doesnt have high priority (afaik?) - http://status.openstack.org/zuul/ | 19:52 |
| mnaser | there are 79 jobs queued there | 19:52 |
| *** baoli has joined #openstack-infra | 19:53 | |
| *** dizquierdo has joined #openstack-infra | 19:53 | |
| pabelanger | mnaser: they are low | 19:54 |
| clarkb | ok I am going to delete the old chocolate mirror now | 19:55 |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "TEMP - Disable voting on tripleo upgrade jobs" https://review.openstack.org/494596 | 19:56 |
| pabelanger | clarkb: /var/log/apache2/mirror01.regionone.infracloud-vanilla.openstack.org_8080_error.log showing some interesting errors. I suspect they are all networking related | 19:56 |
| *** srobert has quit IRC | 19:56 | |
| *** tumbarka__ has quit IRC | 19:56 | |
| clarkb | pabelanger: [Thu Aug 17 18:36:36.344042 2017] [proxy_http:error] [pid 1323:tid 140673055385344] (70014)End of file found: [client 15.184.66.223:59096] AH01102: error reading status line from remote server buildlogs.centos.org:443 ya I think that is failure to properly set up ssl | 19:57 |
| clarkb | and since its EOF guessing networking problems | 19:57 |
| pabelanger | ya | 19:57 |
| *** rkukura has quit IRC | 19:57 | |
| clarkb | pabelanger: I'm going to do kna1 now | 19:58 |
| pabelanger | clarkb: k | 19:58 |
| clarkb | lon and la are both not in use maybe you want to jump ahead to vexxhost? | 19:58 |
| clarkb | then we swing around and do the ones in idle regions | 19:58 |
| clarkb | or do you want to do kna1 and I do vexxhost? | 19:59 |
| clarkb | (since youalready know what flavors and everything to use in citycloud?0 | 19:59 |
| *** calebb has quit IRC | 19:59 | |
| *** trown|brb is now known as trown | 19:59 | |
| pabelanger | clarkb: I've also confimed yum failures in rax are because of DNS lookup failures | 19:59 |
| pabelanger | that will be fixed when we cut a new glean release with --skip-dns and we can test our new systemd service config files | 20:00 |
| clarkb | gotcha so its failures to look up against the cloud set dns? | 20:00 |
| pabelanger | ya | 20:01 |
| *** calebb has joined #openstack-infra | 20:01 | |
| pabelanger | so far xenial seems to be working nicely | 20:01 |
| elmiko | mnaser: thanks, i will practice patience =) | 20:01 |
| clarkb | ya and I definitely think it fixed the pip problems | 20:01 |
| clarkb | that hash mismatch is basically gone now | 20:01 |
| pabelanger | clarkb: cool | 20:02 |
| clarkb | pabelanger: should I do kna1 or do you want to do it? | 20:02 |
| *** marst has joined #openstack-infra | 20:04 | |
| pabelanger | clarkb: I won't be able to get to it for awhile, so feel free | 20:04 |
| clarkb | ok | 20:05 |
| *** dizquierdo has quit IRC | 20:05 | |
| clarkb | also was you that marked we can delete the old 2 core server in kna1? | 20:05 |
| clarkb | I'm going to start with that | 20:05 |
| *** jkilpatr has quit IRC | 20:09 | |
| *** e0ne has quit IRC | 20:12 | |
| *** armax has quit IRC | 20:13 | |
| *** rcernin has quit IRC | 20:16 | |
| *** kjackal has joined #openstack-infra | 20:17 | |
| clarkb | pabelanger: ianw fyi I just went and accepted the ssh host keys for all of the new mirrors built in the last day or so on puppetmaster | 20:20 |
| *** adisky__ has quit IRC | 20:20 | |
| pabelanger | clarkb: ack | 20:23 |
| *** jkilpatr has joined #openstack-infra | 20:30 | |
| *** kgiusti has quit IRC | 20:31 | |
| clarkb | putting kna1 mirror in service now (via DNS CNAME update) | 20:31 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove remote_tmp setting https://review.openstack.org/494695 | 20:36 |
| openstackgerrit | Merged openstack-infra/project-config master: Switch to os-service-types https://review.openstack.org/494456 | 20:37 |
| openstackgerrit | Divyansh Acharya proposed openstack/os-testr master: Just changed the opening of the file https://review.openstack.org/494697 | 20:38 |
| *** isaacb has quit IRC | 20:40 | |
| *** stevebaker has joined #openstack-infra | 20:41 | |
| clarkb | watching apache logs on the mirrors is like the heartbeat of ci | 20:42 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove remote_tmp setting https://review.openstack.org/494695 | 20:43 |
| clarkb | also sto2 is stubbornly still using the old one | 20:43 |
| *** srobert has joined #openstack-infra | 20:43 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul-jobs master: Add known_hosts generation for multiple nodes https://review.openstack.org/494700 | 20:46 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack-infra/project-config master: Add Kolla Ansible Plugins project https://review.openstack.org/491412 | 20:47 |
| *** atarakt has joined #openstack-infra | 20:47 | |
| *** heroux has quit IRC | 20:53 | |
| *** aeng has joined #openstack-infra | 20:57 | |
| *** trown is now known as trown|outtypewww | 21:02 | |
| ianw | clarkb: oh, i thought i'd done that for the rax ones by running a manual puppet run | 21:03 |
| clarkb | ianw: did you run it as root? | 21:03 |
| clarkb | I had to accept them fwiw so maybe that isn't sufficient for some reason | 21:03 |
| ianw | yep ... | 21:04 |
| ianw | oh well, i'll know that's not enough | 21:04 |
| clarkb | I'm about to delete the old sto2 mirror. Was last hit about 16 minutes ago | 21:04 |
| *** Apoorva_ has joined #openstack-infra | 21:04 | |
| *** tonyb has joined #openstack-infra | 21:05 | |
| clarkb | ianw: I think the next one we should do is vexxhost since ovh and the other two citycloud regions are not in use right now | 21:05 |
| *** mat128 has quit IRC | 21:05 | |
| *** eroux has joined #openstack-infra | 21:06 | |
| jeblair | infra-root: i'm adding all zuul executors to the emergency file | 21:06 |
| openstackgerrit | Merged openstack-infra/glean master: Remove config_exists function from glean.sh https://review.openstack.org/493330 | 21:06 |
| *** srobert has quit IRC | 21:06 | |
| ianw | clarkb: sounds good. i can get to it in a hour or so, if you want | 21:07 |
| mordred | jeblair: ++ | 21:07 |
| clarkb | ianw: that would be great. sto2 is all done now (will update etherpad) and I will make sure kna1 is claened up in a bit | 21:07 |
| ianw | i'll also cleanup rax | 21:07 |
| AJaeger | mordred, efried , we have the same problem in openstack-manuals see http://logs.openstack.org/87/494087/1/gate/gate-openstack-manuals-tox-doc-publish-checkbuild/64da766/console.html#_2017-08-17_21_04_32_593997 . | 21:07 |
| clarkb | ianw: ok | 21:07 |
| AJaeger | Sorry, completely signing off now... | 21:07 |
| *** Apoorva has quit IRC | 21:07 | |
| mordred | AJaeger: yes - sorry - that's next on my list to fix | 21:08 |
| openstackgerrit | Slawek Kaplonski proposed openstack-infra/shade master: DNM Test force cinder volume delete https://review.openstack.org/494705 | 21:08 |
| clarkb | ianw: to check things have migrated I'ev just been running sudo tail -f /var/log/apache2/*access.log | 21:10 |
| efried | mordred Zat guy also using duplicated code from os-service-types? | 21:10 |
| mordred | efried: yes. and also api-site | 21:10 |
| ianw | clarkb: i installed multitail but similar :) but none of the segfaults on rax afaics, so that's something | 21:11 |
| mordred | efried: I want to add a feature to api-site as well to add redirects for aliases | 21:11 |
| clarkb | ianw: ya on the e-r/logstash side we've seen the pip install hash mismatch go away completely | 21:11 |
| efried | mordred Please explicitly add me to reviews for those. They're not on my watchlist and I've got too many windows going to notice 'em as they come in. | 21:11 |
| mordred | efried: will do | 21:12 |
| clarkb | ianw: pabelanger is saying the yum failures are due to dns so we need to get the glean dns disabled stuff in place | 21:12 |
| clarkb | pabelanger: is there a topic those changes are under for easy reviewing? | 21:12 |
| ianw | i think i did look at them, we may need a release | 21:12 |
| clarkb | ya | 21:13 |
| *** Apoorva_ has quit IRC | 21:15 | |
| *** Apoorva has joined #openstack-infra | 21:15 | |
| *** dougwig has quit IRC | 21:16 | |
| *** ldnunes has quit IRC | 21:16 | |
| *** armax has joined #openstack-infra | 21:17 | |
| pabelanger | clarkb: ianw: https://review.openstack.org/#/q/topic:skip-dns | 21:18 |
| pabelanger | all merged now | 21:18 |
| pabelanger | clarkb: ianw: I want to then test https://review.openstack.org/493665/ | 21:19 |
| *** dprince has quit IRC | 21:19 | |
| clarkb | pabelanger: I see a bug, will do full review and post in a minute | 21:21 |
| *** baoli has quit IRC | 21:22 | |
| clarkb | its really annoying that systemd units have to always be fully qualified paths ... | 21:22 |
| clarkb | pabelanger: commented | 21:24 |
| *** tosky has joined #openstack-infra | 21:24 | |
| openstackgerrit | Merged openstack-infra/system-config master: Install python2 on newer Ubuntus https://review.openstack.org/494638 | 21:26 |
| *** slagle has quit IRC | 21:28 | |
| pabelanger | clarkb: thanks! I'll push up a change in a bit | 21:28 |
| clarkb | nodepool and zuul are busy today | 21:29 |
| clarkb | also someone pushed a ton of tempest patches that all just totally failed pep8 | 21:29 |
| clarkb | (legitimetely, not an infra problem) | 21:29 |
| pabelanger | clarkb: ianw: think we are going to keep seeing http://logs.openstack.org/96/489196/5/gate/gate-tripleo-ci-centos-7-undercloud-oooq/54b4169/logs/undercloud/home/jenkins/install_packages.sh.log.txt.gz#_2017-08-17_20_52_50 until we have primed the proxy cache | 21:31 |
| pabelanger | hopefully not much longer | 21:31 |
| clarkb | I'm also noticing that there are job out there with hour timeouts running tempest | 21:32 |
| clarkb | which I think is skewing the timeout tracking (there is no way you can run tempest in under an hour) | 21:32 |
| *** jcoufal has quit IRC | 21:32 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove remote_tmp setting https://review.openstack.org/494695 | 21:32 |
| clarkb | all that to say I think we are in relatively good shape despite nodepool running at full capacity all day | 21:32 |
| clarkb | definitely better off than we were last week | 21:32 |
| *** EricGonc_ has quit IRC | 21:34 | |
| pabelanger | Ya, tripleo change queue for gate isn't reseting as much also | 21:34 |
| pabelanger | that helps cut down on wasted resources too | 21:34 |
| pabelanger | they still 10hrs to land patches, but better then 24hrs a week ago | 21:35 |
| *** Julien-zte has quit IRC | 21:36 | |
| *** Julien-zte has joined #openstack-infra | 21:36 | |
| *** priteau has quit IRC | 21:38 | |
| *** slaweq_ has quit IRC | 21:40 | |
| *** slaweq_ has joined #openstack-infra | 21:41 | |
| *** bobh has quit IRC | 21:41 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix zuul_stream callback dict modification https://review.openstack.org/494679 | 21:42 |
| *** oanson has quit IRC | 21:42 | |
| *** oanson has joined #openstack-infra | 21:44 | |
| *** slaweq_ has quit IRC | 21:45 | |
| *** esberglu has quit IRC | 21:47 | |
| *** rtjure has quit IRC | 21:47 | |
| *** vhosakot has quit IRC | 21:48 | |
| *** vhosakot has joined #openstack-infra | 21:48 | |
| EmilienM | pabelanger, clarkb: we listenned and changed tripleo ci to use whitelist: https://review.openstack.org/#/c/493973/ | 21:50 |
| EmilienM | it's about to merge. | 21:50 |
| EmilienM | thanks for your feedback, please keep doing this, and we'll collaborate for sure | 21:50 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-zuul master: Allow executor to core dump https://review.openstack.org/494721 | 21:50 |
| jeblair | clarkb, pabelanger: can you +3 that ^ ? | 21:51 |
| *** xyang1 has quit IRC | 21:53 | |
| jeblair | infra-root: i have removed the zuul executors from the emergency file. ze01 is now running branch tip, with 'keep' enabled, and with a local modification to the init script to allow core dumps. | 21:53 |
| *** esberglu has joined #openstack-infra | 21:54 | |
| *** petevg has quit IRC | 21:57 | |
| *** vhosakot_ has joined #openstack-infra | 21:57 | |
| *** vhosakot has quit IRC | 21:58 | |
| *** EricGonczer_ has joined #openstack-infra | 21:59 | |
| *** thorst has quit IRC | 22:00 | |
| *** iyamahat_ has joined #openstack-infra | 22:02 | |
| pabelanger | jeblair: done | 22:04 |
| pabelanger | EmilienM: clarkb: cool, thanks! I'll look into it in a bit | 22:04 |
| *** iyamahat has quit IRC | 22:05 | |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/project-config master: Fix grafana configuration to neutron-dynamic-routing periodic job https://review.openstack.org/493709 | 22:05 |
| openstackgerrit | Merged openstack-infra/project-config master: Fail early if any Puppet preparation commands fail https://review.openstack.org/494244 | 22:07 |
| *** wolverineav has quit IRC | 22:07 | |
| openstackgerrit | Merged openstack-infra/project-config master: Provide unified gate configuration https://review.openstack.org/490790 | 22:07 |
| openstackgerrit | Merged openstack-infra/project-config master: Add missing publish-to-pypi to networking-baremetal https://review.openstack.org/494166 | 22:07 |
| ianw | pabelanger: hmm, 502 is a weird error though | 22:08 |
| pabelanger | ianw: ya, being infracloud only so far. I'm not too concerned | 22:08 |
| *** danger is now known as dangers_away | 22:10 | |
| *** slaweq_ has joined #openstack-infra | 22:10 | |
| *** thorst has joined #openstack-infra | 22:11 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Create glean@.service.d/override.conf https://review.openstack.org/493665 | 22:11 |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Make neutron functional job non-voting" https://review.openstack.org/494295 | 22:13 |
| *** bobh has joined #openstack-infra | 22:13 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add additional LOCI repos https://review.openstack.org/492637 | 22:13 |
| clarkb | EmilienM: thanks | 22:15 |
| clarkb | jeblair: its impressive that you got ansible to segfault | 22:15 |
| jeblair | clarkb: yeah... unfortunately it's been happening for a while. it seems to generally be causing post failures when it happens. | 22:16 |
| clarkb | I'm deleting the old kna1 mirror now | 22:17 |
| pabelanger | clarkb: ack | 22:17 |
| *** mattmceuen has quit IRC | 22:17 | |
| *** jascott1 has joined #openstack-infra | 22:17 | |
| *** claudiub|3 has joined #openstack-infra | 22:17 | |
| pabelanger | clarkb: cacti for rax: https://review.openstack.org/494370 | 22:17 |
| clarkb | oh thats what I've forgotten to do | 22:19 |
| openstackgerrit | Merged openstack-infra/project-config master: Add TripleO scenario007-container experimental job for OVN https://review.openstack.org/490622 | 22:19 |
| *** slaweq_ has quit IRC | 22:19 | |
| *** claudiub has quit IRC | 22:19 | |
| *** jascott1 has quit IRC | 22:22 | |
| *** jascott1 has joined #openstack-infra | 22:22 | |
| *** thorst has quit IRC | 22:23 | |
| *** gouthamr has joined #openstack-infra | 22:25 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Rename new infracloud and citycloud mirrors https://review.openstack.org/494728 | 22:27 |
| clarkb | pabelanger: ianw ^ that covers the other ones done today | 22:27 |
| *** thorst has joined #openstack-infra | 22:28 | |
| pabelanger | +2 thanks | 22:29 |
| *** felipemonteiro_ has quit IRC | 22:31 | |
| openstackgerrit | Merged openstack-infra/project-config master: Dragonflow: Add a gate-hook to tempest tests https://review.openstack.org/494155 | 22:31 |
| *** openstackgerrit has quit IRC | 22:33 | |
| *** edmondsw has quit IRC | 22:36 | |
| pabelanger | clarkb: jeblair: ze01 has load of 40+ right now, 15mins is 72.0 | 22:39 |
| pabelanger | it is pushing over 250 jobs at one point | 22:39 |
| jeblair | pabelanger: zl01? | 22:40 |
| pabelanger | yes, sorry, zl01 | 22:40 |
| pabelanger | should be bring zl07 / zl08 / zl09 back online? Or did we delete them | 22:40 |
| jeblair | pabelanger: i deleted them. are we seeing timeouts? | 22:40 |
| pabelanger | jeblair: I think so: telnet://198.72.124.244:19885 has been hung a while on SUCCESS | 22:41 |
| pabelanger | but hasn't completed | 22:41 |
| jeblair | pabelanger: what about the other launchers? | 22:41 |
| clarkb | 250 is definitely skewed | 22:41 |
| pabelanger | others are lower, 150 jobs | 22:42 |
| clarkb | there are 6 launchers and 900 instances so should be ~150 per launcher right? | 22:42 |
| jeblair | yeah, so this might be an unusual situation | 22:42 |
| pabelanger | should we graceful zl01, to spread some of the jobs out? | 22:43 |
| pabelanger | then bring it back online | 22:43 |
| pabelanger | actually, pause will stop new jobs right? | 22:43 |
| jeblair | pabelanger: it should do that on its own | 22:43 |
| pabelanger | it has a large amount of ready nodes assigned to it also, 25 others are 5 | 22:43 |
| pabelanger | http://grafana.openstack.org/dashboard/db/nodepool | 22:44 |
| clarkb | did it just end up with a bunch of multinode jobs I wonder | 22:45 |
| jamielennox | pabelanger: have you got more stuff you still want to merge into glean before a release? | 22:45 |
| pabelanger | jamielennox: not currently | 22:45 |
| pabelanger | clarkb: maybe? didn't we say that was part of the issue last time? | 22:45 |
| jamielennox | pabelanger: ok, cool, who do i ping to get a tag? glean doesn't seem to be handled out of releases | 22:46 |
| clarkb | ya I think so, basically getting laoded up on longer bigger jobs | 22:46 |
| clarkb | since gearman doesn't really distinguish those | 22:46 |
| clarkb | jamielennox: I think any infra-root can make them | 22:46 |
| jamielennox | clarkb: is that volunteering? | 22:47 |
| jamielennox | need a 1.9.3 | 22:47 |
| pabelanger | likely needs to be 1.10.0 now, new cli args | 22:47 |
| *** openstackgerrit has joined #openstack-infra | 22:47 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove callback_whitelist setting https://review.openstack.org/488214 | 22:47 |
| clarkb | pabelanger: ya | 22:49 |
| pabelanger | 100+ load on zl01 now | 22:49 |
| clarkb | jamielennox: in what case do you not boot with config drive? glean isn't useful without config drive | 22:49 |
| *** slaweq_ has joined #openstack-infra | 22:50 | |
| jamielennox | clarkb: it's being pulled in as simple-init from dib | 22:50 |
| clarkb | jamielennox: right but why use simple-init then? | 22:50 |
| jamielennox | and uses metadata i think | 22:50 |
| clarkb | jamielennox: glean only supports config drive | 22:50 |
| jamielennox | clarkb: honestly? history most likely | 22:50 |
| clarkb | and if you aren't config driving there is zero reason to use glean | 22:50 |
| pabelanger | ya, likely can drop simple-init in this case | 22:50 |
| clarkb | (either use dhcp all interfaces or cloud init) | 22:50 |
| jamielennox | SpamapS: ^ ? | 22:51 |
| clarkb | I mean happy to get the fix in too, just trying to make sure something else isn't broken | 22:51 |
| pabelanger | glean will generate the dhcp files however | 22:51 |
| *** thorst has quit IRC | 22:51 | |
| clarkb | pabelanger: ya but so will the dhcp element | 22:52 |
| pabelanger | yes | 22:52 |
| jamielennox | so i'm guessing this is just something we inheritted from looking at the -infra elements and it worked so we kept it | 22:52 |
| pabelanger | ya, you can also have simple-init install from git repo too | 22:53 |
| jamielennox | my understanding was that there (used?) to be dependency concerns with cloud-init so we uesd the same as upstream | 22:53 |
| pabelanger | if needed | 22:53 |
| pabelanger | ya, cloud-init bring is a large dependency list for python | 22:53 |
| pabelanger | glean has zero | 22:53 |
| clarkb | ya but you shouldn't need glean at all or cloud init if you just want dhcp | 22:53 |
| clarkb | anyways /me wanders off to get key | 22:54 |
| jamielennox | right, and so we don't actually do anything cloud-init-y we just put everything in the image | 22:54 |
| *** Apoorva_ has joined #openstack-infra | 22:55 | |
| *** rbrndt has quit IRC | 22:55 | |
| *** iyamahat has joined #openstack-infra | 22:56 | |
| *** iyamahat_ has quit IRC | 22:56 | |
| *** Apoorva has quit IRC | 22:59 | |
| *** slaweq_ has quit IRC | 22:59 | |
| clarkb | pabelanger: I just realized in reading the diff to write a tag message that the skip dns option won't work if dhcp is enabled | 22:59 |
| *** gildub has joined #openstack-infra | 22:59 | |
| clarkb | pabelanger: so it will fix rackspace but that is it | 22:59 |
| clarkb | pabelanger: I'm going to go ahead and tag it though and we can work on disabling dhcp setting resolvers later | 23:00 |
| *** Apoorva_ has quit IRC | 23:00 | |
| *** Apoorva has joined #openstack-infra | 23:00 | |
| clarkb | jamielennox: pabelanger 1.10.0 tag is pushed | 23:01 |
| *** pfallenop has joined #openstack-infra | 23:02 | |
| *** pfallenop has joined #openstack-infra | 23:02 | |
| pabelanger | clarkb: looking to see why dhcp will fail | 23:02 |
| jamielennox | clarkb: thanks - i'll look into replacing glean, but it's easy for now just to have it run | 23:02 |
| clarkb | pabelanger: dhcp will "fail" beacuse dhcp is writing the resolv.conf in that case | 23:02 |
| clarkb | pabelanger: so glean will skip it but then dhcpcd or dhclient or whatever will write it too | 23:03 |
| pabelanger | clarkb: Oh, right. | 23:03 |
| pabelanger | Ya, we have dhclient.conf disable that | 23:03 |
| clarkb | ah ok | 23:03 |
| clarkb | pabelanger: mgiht be worth a doc update to glean to explain that this won't affect dhcp clients | 23:04 |
| pabelanger | clarkb: http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/nodepool-base/finalise.d/89-unbound#n140 | 23:04 |
| clarkb | (so you'll have to separately configure those) | 23:04 |
| pabelanger | clarkb: but ya, something to test | 23:04 |
| pabelanger | clarkb: agree | 23:04 |
| *** Douhet has joined #openstack-infra | 23:11 | |
| clarkb | pabelanger: is zl01 managing to chew through things (if slowly?) or do you know if it is throwing errors out? | 23:12 |
| clarkb | I notice that my tarball build for glean 1.10.0 is successful but hasn't properly reported/recorded yet | 23:12 |
| pabelanger | clarkb: it is still accepting jobs as it finishes them, so staying the same | 23:13 |
| pabelanger | 215 jobs right now according to grafana | 23:13 |
| pabelanger | load 75 | 23:13 |
| *** Julien-zte has quit IRC | 23:18 | |
| *** vhosakot_ has quit IRC | 23:19 | |
| *** Julien-zte has joined #openstack-infra | 23:19 | |
| openstackgerrit | Merged openstack-infra/puppet-zuul master: Allow executor to core dump https://review.openstack.org/494721 | 23:20 |
| clarkb | pabelanger: jamielennox https://pypi.python.org/pypi/glean/ should be good to go now | 23:21 |
| pabelanger | clarkb: ya, confirmed dns is setup correctly on dhcp providers, dhclient.conf still working as expected | 23:21 |
| pabelanger | inap has the issue also | 23:22 |
| pabelanger | clarkb: thanks! I'll be building some images tomorrow | 23:22 |
| *** Julien-zte has quit IRC | 23:23 | |
| clarkb | also looks like we are launching jobs about as quickly as they are coming in and even running post jobs now | 23:23 |
| jamielennox | clarkb: thanks, running a nodepool build now | 23:23 |
| clarkb | so I think we've hit the tail end of demand for the day and things should catch up and calm down a bit | 23:23 |
| *** sflanigan has joined #openstack-infra | 23:24 | |
| *** sflanigan has joined #openstack-infra | 23:24 | |
| *** dave-mccowan has quit IRC | 23:25 | |
| *** Julien-zte has joined #openstack-infra | 23:25 | |
| *** freerunner has quit IRC | 23:27 | |
| jamielennox | clarkb: fixes it for me | 23:29 |
| *** freerunner has joined #openstack-infra | 23:29 | |
| *** tosky has quit IRC | 23:29 | |
| *** ekcs has quit IRC | 23:30 | |
| *** numans has quit IRC | 23:30 | |
| mordred | clarkb: with demand falling off, the post job for service-types-authority ran and latest is published, so i rechecked https://review.openstack.org/#/c/494533 | 23:31 |
| mordred | if it comes back green, it's good to land now | 23:31 |
| clarkb | mordred: ok I wasn't able to follow that much earlier as I was trying to not be too distracted on the mirrors front. Is there a tldr? | 23:32 |
| clarkb | just a bug in the check? | 23:32 |
| mordred | clarkb: oh golly no - and it's likely best you didn't pay attention :) | 23:35 |
| ianw | clarkb: vexxhost -> Error in creating instance (Inner Exception: Quota exceeded for cores: Requested 8, but already used 14 of 20 cores (HTTP 403) ... hmm | 23:36 |
| *** dingyichen has joined #openstack-infra | 23:36 | |
| clarkb | ianw: nb03 and existing mirror? | 23:37 |
| ianw | yeah, i guess so | 23:37 |
| clarkb | ianw: maybe we do that one when things settle down and we can just turn the region off in nodepool for an hour | 23:37 |
| mordred | clarkb: we wound up landing a change to service-types-authority that os-service-types knew how to deal with, but didn't anticipate how it would break the docs check scripts - we've updated the scripts and the data so everything is in a good state and should stay that way ... but we've also got a new list of gate jobs we're gonna want to put on service-types-authority patches moving forward | 23:37 |
| mordred | clarkb: tl;dr I guess is that it actually started being a thing that was dependent on slightly before sufficient safeguards were in place | 23:39 |
| *** markvoelker has quit IRC | 23:40 | |
| mordred | efried: https://review.openstack.org/494734 openstack-manuals fix | 23:40 |
| clarkb | mordred: I've approved the project-config fix | 23:41 |
| mordred | woot | 23:41 |
| mnaser | ianw should i raise it? | 23:41 |
| SpamapS | jamielennox: we have a cloud we use that doesn't do config drive? | 23:42 |
| mnaser | i can do that if you'd like | 23:42 |
| clarkb | mnaser: if you don't mind that would make life a little easier for us | 23:42 |
| mordred | config-drive is the best | 23:42 |
| mnaser | clarkb np one moment | 23:42 |
| clarkb | mnaser: its on the openstackci-vexxhost user | 23:42 |
| clarkb | at least that is what we call it in our clouds.yaml not sure of the actual name | 23:42 |
| mordred | clarkb: in the morning I will add a job on s-t-a to make sure it can't break project-config | 23:42 |
| mnaser | clarkb yeah there is a openstackci and openstackjenkins | 23:42 |
| mnaser | clarkb any reasonable number you have in mind? | 23:42 |
| mnaser | for quota | 23:43 |
| jamielennox | SpamapS: so this is not an area i know much about (and a channel i'll probably regret asking on), but isn't config drive mostly rax? | 23:43 |
| clarkb | mnaser: uh we will be replacing the old server so we don't have to go super high for now. 25 should be plenty and is a nice round number | 23:43 |
| *** dizquierdo has joined #openstack-infra | 23:43 | |
| jamielennox | i don't remember it being used on any of the deploys we've targetted - but i wouldn't have really checked | 23:43 |
| clarkb | jamielennox: rackspace doesn't run metadata service so your metadata option in rax is config drive. So in that way its almost a necessity on rax | 23:44 |
| mnaser | clarkb set to unlimited cores/instances and 100gb ram | 23:44 |
| mnaser | for openstackci | 23:44 |
| clarkb | jamielennox: generally though it gives you a far more reliable way to access config drive data, the downside is you can't get updated metadata and for a long time you couldn't live migrate instances with config drive | 23:44 |
| mnaser | jamielennox you can get configdrive on any cloud. it's an image metadata (unless the cloud has it set to required) | 23:44 |
| clarkb | jamielennox: metadata service goes through a mess of dark ip magic and NAT and http proxies its just a lot more complicated and less reliable in our experience | 23:45 |
| clarkb | mnaser: thank you@ | 23:45 |
| clarkb | er ! | 23:45 |
| mnaser | so if anything, configdrive is much more of a common denominator between clouds rather than metadata availablity | 23:45 |
| mnaser | cause nova just builds that in the compute node so as clarkb said, no magic involved in getting the data in | 23:45 |
| clarkb | jamielennox: iirc the path is VM -> 169.252.x.y -> through nat -> neutron proxy -> nova api server | 23:45 |
| jamielennox | so config drive is dfeault enabled? we're using a bluebox cloud and they generally leave things to default | 23:45 |
| clarkb | jamielennox: I don't think it is. mriedem has talked about changing that though | 23:46 |
| jamielennox | yea, i remember some of the security discussion around metadata service | 23:46 |
| clarkb | jamielennox: you can boot with config drive with a boot option even if it isn't enabled by default though | 23:46 |
| mnaser | jamielennox it is not enabled by default, there is a force_config_drive option which *forces* it for all VMs | 23:46 |
| mnaser | but as clarkb said, it is a thing you can select per boot (or per image too as an image property) | 23:46 |
| clarkb | this is one of those places where I'm glad we didn't stick to the way aws does it and we ended up wtih something far better imo | 23:47 |
| jamielennox | ok, as zuul is booting these instances i expect there'll be a flag there or in os-c-c that i've missed | 23:47 |
| SpamapS | I've always felt that we could invent a far simpler metadata service | 23:47 |
| SpamapS | with a config drive handing the instance a nonce.. and then just contacting a nice stateless webapp from there. | 23:48 |
| jamielennox | so i remember one summit listening to the rabbit hole of how you securely put a nonce on a new vm and it was painful | 23:49 |
| mordred | jamielennox: config-drive is default enabled in nodepool iirc | 23:50 |
| clarkb | mordred: I don't think it is | 23:51 |
| clarkb | someone else was bit by this bug too | 23:51 |
| mordred | clarkb: we still haven't thrown that switch? | 23:51 |
| mordred | golly. we should throw that switch | 23:51 |
| clarkb | ++ | 23:51 |
| clarkb | except does it error if cloud doesn't support it for some reason? | 23:51 |
| clarkb | (not sure if nova allows that) | 23:52 |
| mordred | hrm. maybe I'll put that on my list to investigate | 23:52 |
| openstackgerrit | Merged openstack-infra/project-config master: Fetch service type data from the remote URL https://review.openstack.org/494533 | 23:52 |
| clarkb | jamielennox: you can explicitly enable it on images in nodepool's config https://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/nodepool.yaml#n293 | 23:52 |
| ianw | mnaser: thanks, we really only need 22, but that works too :) | 23:53 |
| ianw | and even then only for a little :) | 23:54 |
| mnaser | ianw figured that infra might have other needs for control plane so i'll leave it there for when needed :) | 23:54 |
| clarkb | mnaser: yes in fact I plan to write the spec to move logs.o.o tomorrow | 23:54 |
| mnaser | clarkb oh cool, we can talk more about that at the ptg too (when you and infra are not putting out zuulv3 fires :)) | 23:55 |
| clarkb | sounds good | 23:55 |
| *** slaweq_ has joined #openstack-infra | 23:56 | |
| *** dizquierdo has quit IRC | 23:58 | |
| mnaser | anyways, just finished upgrading keystone and glance today from newton to ocata without anyone noticing, that's a win | 23:59 |
| * mnaser is out | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!