| fungi | i have to step away for a bit and tackle some chores i've been neglecting before tc office hour starts in an hour | 00:00 |
|---|---|---|
| clarkb | probably what we want to do is ask hpe if they can check for unhappy switch tables/hub behavior. And also tell us if there is some bw cap we fall under | 00:02 |
| fungi | though worth noting the pattern of gaps on the http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=4992&rra_id=all graphs would suggest packet loss increases during americas daytime hours | 00:02 |
| fungi | not sure whether that's because of our load or someone else's though | 00:02 |
| *** dingyichen has joined #openstack-infra | 00:03 | |
| *** gmann has quit IRC | 00:18 | |
| *** gmann has joined #openstack-infra | 00:18 | |
| *** sbezverk has quit IRC | 00:24 | |
| persia | Is it possible for people to have negative gerrit IDs? If so, is this something more complicated than incorrect conversion between signed and unsigned integers? | 00:28 |
| fungi | persia: as i explained a few hours ago in #openstack-dev those are fabricated for the purposes of indexing extra_atcs from governance who don't have gerrit accounts | 00:29 |
| persia | fungi: Apologies for the duplicate then: I had thought I was reading backscroll, but appear to have been mistaken. | 00:29 |
| fungi | still gives us an integer index and guaranteed not to conflict wit actual gerrit ids | 00:29 |
| persia | Indeed, and a sensible way to manage that. | 00:29 |
| fungi | and no apologies needed, it's a lot of complex detail to merge all these data sources and avoid duplicates | 00:30 |
| *** thorst has joined #openstack-infra | 00:39 | |
| *** thorst has quit IRC | 00:44 | |
| *** LindaWang has joined #openstack-infra | 00:44 | |
| *** gouthamr has quit IRC | 00:48 | |
| *** jpena|off is now known as jpena | 00:49 | |
| *** shu-mutou-AWAY is now known as shu-mutou | 00:49 | |
| openstackgerrit | Matthew Treinish proposed openstack/os-testr master: Switch to stestr under the covers https://review.openstack.org/488441 | 00:51 |
| *** dougwig has quit IRC | 00:56 | |
| *** bobh has joined #openstack-infra | 00:56 | |
| *** bobh has quit IRC | 00:57 | |
| *** iyamahat_ has joined #openstack-infra | 00:57 | |
| *** larainema has quit IRC | 00:58 | |
| *** pradk has joined #openstack-infra | 00:58 | |
| *** baoli_ has joined #openstack-infra | 00:58 | |
| *** iyamahat has quit IRC | 00:58 | |
| *** thorst has joined #openstack-infra | 00:58 | |
| *** jmorgan1_ has quit IRC | 00:59 | |
| *** adriant has quit IRC | 00:59 | |
| *** jmorgan1 has joined #openstack-infra | 00:59 | |
| *** baoli has quit IRC | 01:00 | |
| *** quite has quit IRC | 01:00 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/system-config master: Fix nova-conductor log collection for superconductor mode https://review.openstack.org/491955 | 01:01 |
| *** quite has joined #openstack-infra | 01:01 | |
| *** bobh has joined #openstack-infra | 01:05 | |
| *** hongbin has joined #openstack-infra | 01:06 | |
| *** markvoelker has joined #openstack-infra | 01:07 | |
| *** claudiub has quit IRC | 01:08 | |
| AJaeger_ | infra-root, can we remove OSIC cloud from system-config? There're two +2s but no +A yet on https://review.openstack.org/#/c/491192/ | 01:09 |
| *** liujiong has joined #openstack-infra | 01:09 | |
| AJaeger_ | clarkb: do you want to review this footer change again, please: https://review.openstack.org/490989 | 01:09 |
| fungi | AJaeger_: you should be vacationing! and sure, i'll take another look at the osic change | 01:11 |
| *** LindaWang has quit IRC | 01:12 | |
| *** LindaWang has joined #openstack-infra | 01:13 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for citcloud-lon1 slow nodes / n-cpu unavailable https://review.openstack.org/491956 | 01:13 |
| openstackgerrit | Merged openstack-infra/project-config master: Reduce infra-cloud usage https://review.openstack.org/491949 | 01:14 |
| *** liujiong has quit IRC | 01:15 | |
| AJaeger_ | fungi: team meeting - not vacatoin... | 01:15 |
| AJaeger_ | fungi: thanks for looking. | 01:15 |
| AJaeger_ | fungi: just doing a quick check over before grabbing a beer or two ;) | 01:16 |
| fungi | AJaeger_: oh, you'll need a vacation after this ;) | 01:16 |
| AJaeger_ | ;) | 01:16 |
| ianw | AJaeger_ / fungi : yeah, i didn't want to do that at a random hour in case of unintended consequences, sorry should have commented | 01:17 |
| mriedem | are there any public cloud providers that infra is using which expose nested virt? | 01:18 |
| openstackgerrit | Duong Ha-Quang proposed openstack-infra/project-config master: Add Kolla Ansible Plugins project https://review.openstack.org/491412 | 01:18 |
| mriedem | or is everything like kvm, infra gets a xenial 16.04 vm on it which has nested virt enabled in the image, and then we deploy openstack on that using devstack? | 01:18 |
| *** yamamoto has quit IRC | 01:19 | |
| clarkb | some expose it but its quite unreliable so we dont enable it in devstack-gate | 01:20 |
| clarkb | linux kernel disables by default for intel cpus and seems like for good reason based on experience | 01:21 |
| fungi | mriedem: some of them _may_ have nested virt exposed, but if so i don't know which ones. we 1. don't recommend using it for testing (for several reasons) and 2. can't easily limit specific jobs to specific providers | 01:21 |
| mriedem | clarkb: yeah i remember fixing that one live migration bug in devstack-gate b/c of some bash issue which meant we were missing that check | 01:22 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-client-config master: Updated from global requirements https://review.openstack.org/491294 | 01:22 |
| *** liujiong has joined #openstack-infra | 01:22 | |
| *** Julien-z_ has quit IRC | 01:22 | |
| mriedem | this https://review.openstack.org/#/c/435154/ | 01:22 |
| *** funzo has quit IRC | 01:25 | |
| *** zhurong has joined #openstack-infra | 01:26 | |
| *** Julien-zte has joined #openstack-infra | 01:26 | |
| *** yamamoto has joined #openstack-infra | 01:26 | |
| *** EricGonczer_ has joined #openstack-infra | 01:26 | |
| *** gcb has quit IRC | 01:32 | |
| *** Apoorva_ has joined #openstack-infra | 01:37 | |
| openstackgerrit | Merged openstack-infra/system-config master: Remove OSIC cloud https://review.openstack.org/491192 | 01:39 |
| *** Apoorva has quit IRC | 01:40 | |
| *** Apoorva_ has quit IRC | 01:42 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for citcloud-lon1 slow nodes / n-cpu unavailable https://review.openstack.org/491956 | 01:43 |
| *** Goneri has joined #openstack-infra | 01:43 | |
| openstackgerrit | Merged openstack-infra/system-config master: Regenerate nodepool-builder log https://review.openstack.org/491193 | 01:43 |
| openstackgerrit | Merged openstack-infra/system-config master: Remove tools/clean-fips.py https://review.openstack.org/491194 | 01:44 |
| *** gongysh has joined #openstack-infra | 01:44 | |
| *** jpena is now known as jpena|off | 01:44 | |
| *** gcb has joined #openstack-infra | 01:46 | |
| *** stakeda has joined #openstack-infra | 01:54 | |
| *** liujiong has quit IRC | 01:54 | |
| *** esberglu has quit IRC | 01:56 | |
| *** esberglu has joined #openstack-infra | 01:57 | |
| *** esberglu has quit IRC | 01:58 | |
| *** esberglu has joined #openstack-infra | 01:58 | |
| *** Goneri has quit IRC | 02:00 | |
| *** EricGonczer_ has quit IRC | 02:02 | |
| *** mriedem has quit IRC | 02:04 | |
| *** s-shiono has joined #openstack-infra | 02:08 | |
| *** lewo has quit IRC | 02:14 | |
| *** bobh has quit IRC | 02:16 | |
| *** thorst has quit IRC | 02:16 | |
| *** pradk has quit IRC | 02:19 | |
| *** sree has joined #openstack-infra | 02:20 | |
| *** sree has quit IRC | 02:24 | |
| *** bobh has joined #openstack-infra | 02:25 | |
| *** markvoelker has quit IRC | 02:30 | |
| *** annegentle has quit IRC | 02:33 | |
| *** markvoelker has joined #openstack-infra | 02:33 | |
| *** tuanluong has joined #openstack-infra | 02:41 | |
| *** annegentle has joined #openstack-infra | 02:42 | |
| mordred | 708038 | 02:42 |
| *** gildub has quit IRC | 02:43 | |
| *** esberglu has quit IRC | 02:43 | |
| *** zhurong has quit IRC | 02:44 | |
| *** cuongnv has joined #openstack-infra | 02:45 | |
| *** EricGonczer_ has joined #openstack-infra | 02:46 | |
| *** larainema has joined #openstack-infra | 02:55 | |
| *** karthikp has quit IRC | 02:57 | |
| *** zhurong has joined #openstack-infra | 03:01 | |
| *** spligak has quit IRC | 03:01 | |
| *** sbezverk has joined #openstack-infra | 03:06 | |
| *** yamahata has quit IRC | 03:06 | |
| *** bobh has quit IRC | 03:06 | |
| *** iyamahat_ has quit IRC | 03:06 | |
| *** markvoelker has quit IRC | 03:07 | |
| *** esberglu has joined #openstack-infra | 03:13 | |
| *** links has joined #openstack-infra | 03:15 | |
| *** annegentle has quit IRC | 03:15 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Clear up debian element documentation https://review.openstack.org/491971 | 03:15 |
| *** thorst has joined #openstack-infra | 03:17 | |
| *** esberglu has quit IRC | 03:17 | |
| *** sbezverk has quit IRC | 03:18 | |
| *** thorst has quit IRC | 03:21 | |
| *** EricGonczer_ has quit IRC | 03:24 | |
| *** iyamahat has joined #openstack-infra | 03:25 | |
| *** dave-mccowan has quit IRC | 03:28 | |
| *** iyamahat has quit IRC | 03:30 | |
| *** iyamahat has joined #openstack-infra | 03:30 | |
| *** iyamahat has quit IRC | 03:31 | |
| *** iyamahat has joined #openstack-infra | 03:31 | |
| *** baoli_ has quit IRC | 03:32 | |
| *** iyamahat has quit IRC | 03:34 | |
| *** iyamahat has joined #openstack-infra | 03:34 | |
| *** EricGonczer_ has joined #openstack-infra | 03:37 | |
| *** markvoelker_ has joined #openstack-infra | 03:49 | |
| *** adisky__ has joined #openstack-infra | 03:52 | |
| *** ykarel has joined #openstack-infra | 03:53 | |
| *** markvoelker_ has quit IRC | 03:54 | |
| *** nicolasbock has joined #openstack-infra | 03:57 | |
| openstackgerrit | Tony Breeds proposed openstack-infra/project-config master: Add periodic-stable jobs to oslo projecst that assert stable:follows-policy https://review.openstack.org/491980 | 04:00 |
| *** xarses_ has joined #openstack-infra | 04:02 | |
| *** gongysh has quit IRC | 04:03 | |
| *** bobh has joined #openstack-infra | 04:07 | |
| *** bobh has quit IRC | 04:11 | |
| *** hongbin has quit IRC | 04:14 | |
| *** zhurong has quit IRC | 04:19 | |
| *** gcb has quit IRC | 04:19 | |
| *** zhurong has joined #openstack-infra | 04:20 | |
| *** gcb has joined #openstack-infra | 04:20 | |
| *** sree has joined #openstack-infra | 04:21 | |
| *** iyamahat has quit IRC | 04:21 | |
| *** EricGonczer_ has quit IRC | 04:21 | |
| *** iyamahat has joined #openstack-infra | 04:21 | |
| *** iyamahat has quit IRC | 04:23 | |
| *** iyamahat has joined #openstack-infra | 04:23 | |
| *** slaweq has joined #openstack-infra | 04:25 | |
| *** sbezverk has joined #openstack-infra | 04:26 | |
| *** hareesh has joined #openstack-infra | 04:29 | |
| *** slaweq has quit IRC | 04:30 | |
| *** gongysh has joined #openstack-infra | 04:39 | |
| *** sbezverk has quit IRC | 04:46 | |
| *** gildub has joined #openstack-infra | 04:55 | |
| *** iyamahat has quit IRC | 04:59 | |
| *** armax has quit IRC | 05:06 | |
| *** markvoelker has joined #openstack-infra | 05:06 | |
| openstackgerrit | Artur Basiak proposed openstack-infra/project-config master: Provide unified gate configuration https://review.openstack.org/490790 | 05:09 |
| *** claudiub has joined #openstack-infra | 05:13 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [DNM] lvm testing of spanned vg https://review.openstack.org/491986 | 05:13 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [DNM] refactoring lvm creation into node objects https://review.openstack.org/491987 | 05:13 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Try to early terminate streaming on ansible errors https://review.openstack.org/491027 | 05:16 |
| *** thorst has joined #openstack-infra | 05:17 | |
| *** yamamoto has quit IRC | 05:19 | |
| *** thorst has quit IRC | 05:22 | |
| *** arturb has joined #openstack-infra | 05:23 | |
| *** psachin has joined #openstack-infra | 05:24 | |
| *** slaweq has joined #openstack-infra | 05:26 | |
| *** sshnaidm has quit IRC | 05:28 | |
| *** sshnaidm has joined #openstack-infra | 05:31 | |
| *** slaweq has quit IRC | 05:31 | |
| *** gildub has quit IRC | 05:31 | |
| *** yamamoto has joined #openstack-infra | 05:31 | |
| *** markvoelker has quit IRC | 05:32 | |
| *** markvoelker has joined #openstack-infra | 05:32 | |
| *** inc0 has quit IRC | 05:43 | |
| *** esberglu has joined #openstack-infra | 05:44 | |
| *** inc0 has joined #openstack-infra | 05:44 | |
| *** esberglu has quit IRC | 05:49 | |
| *** iyamahat has joined #openstack-infra | 05:49 | |
| *** cshastri has joined #openstack-infra | 05:51 | |
| *** rcernin has joined #openstack-infra | 05:52 | |
| *** iyamahat_ has joined #openstack-infra | 05:53 | |
| *** iyamahat_ has quit IRC | 05:54 | |
| *** iyamahat_ has joined #openstack-infra | 05:54 | |
| *** iyamahat has quit IRC | 05:56 | |
| *** iyamahat__ has joined #openstack-infra | 05:57 | |
| *** iyamahat_ has quit IRC | 06:01 | |
| *** coolsvap has joined #openstack-infra | 06:04 | |
| *** yamamoto_ has joined #openstack-infra | 06:05 | |
| openstackgerrit | Tony Breeds proposed openstack-infra/devstack-gate master: Add python-virtualenv to the list of required packages https://review.openstack.org/491994 | 06:07 |
| *** yamamoto has quit IRC | 06:09 | |
| *** bobh has joined #openstack-infra | 06:10 | |
| *** pgadiya has joined #openstack-infra | 06:14 | |
| *** bobh has quit IRC | 06:17 | |
| *** ykarel is now known as ykarel|afk | 06:19 | |
| *** inc0 has quit IRC | 06:28 | |
| *** aeng_ has quit IRC | 06:28 | |
| *** slaweq has joined #openstack-infra | 06:33 | |
| *** tesseract has joined #openstack-infra | 06:35 | |
| *** slaweq has quit IRC | 06:35 | |
| *** gildub has joined #openstack-infra | 06:37 | |
| *** liujiong has joined #openstack-infra | 06:52 | |
| *** iyamahat_ has joined #openstack-infra | 07:00 | |
| *** kjackal_ has joined #openstack-infra | 07:03 | |
| *** markvoelker has quit IRC | 07:03 | |
| *** markvoelker has joined #openstack-infra | 07:06 | |
| *** iyamahat__ has quit IRC | 07:06 | |
| *** gildub has quit IRC | 07:07 | |
| *** markvoelker has quit IRC | 07:09 | |
| *** florianf has joined #openstack-infra | 07:13 | |
| *** pcaruana has joined #openstack-infra | 07:24 | |
| *** tnovacik has quit IRC | 07:26 | |
| *** iyamahat has joined #openstack-infra | 07:26 | |
| *** markus_z has joined #openstack-infra | 07:31 | |
| openstackgerrit | Claudiu Belu proposed openstack-infra/project-config master: cloudbase-init: Adds releasenotes jobs https://review.openstack.org/491821 | 07:31 |
| openstackgerrit | Artur Basiak proposed openstack-infra/project-config master: Provide unified gate configuration https://review.openstack.org/490790 | 07:31 |
| *** iyamahat_ has quit IRC | 07:35 | |
| *** rossella_ has joined #openstack-infra | 07:36 | |
| *** thorst has joined #openstack-infra | 07:42 | |
| *** shardy has joined #openstack-infra | 07:42 | |
| *** thorst has quit IRC | 07:44 | |
| *** slaweq has joined #openstack-infra | 07:52 | |
| *** shardy has quit IRC | 07:53 | |
| *** alexchadin has joined #openstack-infra | 07:54 | |
| *** rossella_s has quit IRC | 07:54 | |
| *** esberglu has joined #openstack-infra | 07:55 | |
| *** slaweq has quit IRC | 07:55 | |
| *** slaweq has joined #openstack-infra | 08:14 | |
| *** shu-mutou is now known as shu-mutou-AWAY | 08:14 | |
| *** akscram has quit IRC | 08:15 | |
| *** akscram has joined #openstack-infra | 08:16 | |
| *** esberglu has quit IRC | 08:16 | |
| *** makowals has quit IRC | 08:16 | |
| *** ykarel|afk is now known as ykarel | 08:25 | |
| *** dizquierdo has joined #openstack-infra | 08:33 | |
| *** ralonsoh has joined #openstack-infra | 08:34 | |
| *** sflanigan has quit IRC | 08:35 | |
| *** Douhet has quit IRC | 08:47 | |
| *** mwhahaha has quit IRC | 08:47 | |
| *** dasanind has quit IRC | 08:47 | |
| *** rockstar has quit IRC | 08:47 | |
| *** fdegir has quit IRC | 08:47 | |
| *** simonmcc has quit IRC | 08:47 | |
| *** TheJulia has quit IRC | 08:47 | |
| *** puiterwijk|cld has quit IRC | 08:47 | |
| *** betherly has quit IRC | 08:47 | |
| *** berendt has quit IRC | 08:47 | |
| *** akscram has quit IRC | 08:47 | |
| *** slaweq has quit IRC | 08:47 | |
| *** kjackal_ has quit IRC | 08:47 | |
| *** psachin has quit IRC | 08:47 | |
| *** tuanluong has quit IRC | 08:47 | |
| *** jmorgan1 has quit IRC | 08:47 | |
| *** odyssey4me has quit IRC | 08:47 | |
| *** stevebaker has quit IRC | 08:47 | |
| *** Dinesh_Bhor has quit IRC | 08:47 | |
| *** beisner has quit IRC | 08:47 | |
| *** robled has quit IRC | 08:47 | |
| *** apetrich has quit IRC | 08:47 | |
| *** reed has quit IRC | 08:47 | |
| *** thingee has quit IRC | 08:47 | |
| *** gorba_ has quit IRC | 08:47 | |
| *** jokke_ has quit IRC | 08:47 | |
| *** SergK has quit IRC | 08:47 | |
| *** cburgess has quit IRC | 08:47 | |
| *** kong has quit IRC | 08:47 | |
| *** Shrews has quit IRC | 08:47 | |
| *** antonym has quit IRC | 08:47 | |
| *** coolsvap has quit IRC | 08:47 | |
| *** sree has quit IRC | 08:47 | |
| *** gcb has quit IRC | 08:47 | |
| *** larainema has quit IRC | 08:47 | |
| *** stakeda has quit IRC | 08:47 | |
| *** Julien-zte has quit IRC | 08:47 | |
| *** krtaylor has quit IRC | 08:47 | |
| *** jkilpatr has quit IRC | 08:47 | |
| *** dutsmoc has quit IRC | 08:47 | |
| *** asettle has quit IRC | 08:47 | |
| *** cloudnull has quit IRC | 08:47 | |
| *** mnencia has quit IRC | 08:47 | |
| *** fmccrthy has quit IRC | 08:47 | |
| *** dims has quit IRC | 08:47 | |
| *** seongsoocho has quit IRC | 08:47 | |
| *** uberjay has quit IRC | 08:47 | |
| *** ianychoi has quit IRC | 08:47 | |
| *** wfoster has quit IRC | 08:47 | |
| *** jamielennox has quit IRC | 08:47 | |
| *** yankcrime has quit IRC | 08:47 | |
| *** mpjetta has quit IRC | 08:47 | |
| *** dgonzalez has quit IRC | 08:47 | |
| *** benj_ has quit IRC | 08:47 | |
| *** schang has quit IRC | 08:47 | |
| *** ttx has quit IRC | 08:47 | |
| *** lamt has quit IRC | 08:47 | |
| *** andymccr has quit IRC | 08:47 | |
| *** wznoinsk has quit IRC | 08:47 | |
| *** NikitaKonovalov has quit IRC | 08:47 | |
| *** laurelm has quit IRC | 08:47 | |
| *** vgnbkr has quit IRC | 08:47 | |
| *** peterstac has quit IRC | 08:47 | |
| *** dougshelley66 has quit IRC | 08:47 | |
| *** ltomasbo has quit IRC | 08:47 | |
| *** bkero has quit IRC | 08:47 | |
| *** mmedvede has quit IRC | 08:47 | |
| *** ralonsoh has quit IRC | 08:47 | |
| *** yamamoto_ has quit IRC | 08:47 | |
| *** cuongnv has quit IRC | 08:47 | |
| *** hamzy has quit IRC | 08:47 | |
| *** dhill_ has quit IRC | 08:47 | |
| *** pblaho has quit IRC | 08:47 | |
| *** olaph has quit IRC | 08:47 | |
| *** watersoul has quit IRC | 08:47 | |
| *** isq_ has quit IRC | 08:47 | |
| *** niska has quit IRC | 08:47 | |
| *** masber has quit IRC | 08:47 | |
| *** dosaboy has quit IRC | 08:47 | |
| *** kaisers2 has quit IRC | 08:47 | |
| *** kzaitsev_pi has quit IRC | 08:47 | |
| *** pcrews has quit IRC | 08:47 | |
| *** gibi has quit IRC | 08:47 | |
| *** yuanying has quit IRC | 08:47 | |
| *** bradm has quit IRC | 08:47 | |
| *** rook has quit IRC | 08:47 | |
| *** dmsimard has quit IRC | 08:47 | |
| *** devananda has quit IRC | 08:47 | |
| *** timothyb89 has quit IRC | 08:47 | |
| *** EmilienM has quit IRC | 08:47 | |
| *** zigo has quit IRC | 08:47 | |
| *** tesseract has quit IRC | 08:47 | |
| *** jamesdenton has quit IRC | 08:47 | |
| *** yolanda has quit IRC | 08:47 | |
| *** tinwood has quit IRC | 08:47 | |
| *** wendar has quit IRC | 08:47 | |
| *** numans has quit IRC | 08:47 | |
| *** zz_ja has quit IRC | 08:47 | |
| *** kuromagi^ has quit IRC | 08:47 | |
| *** hughsaunders has quit IRC | 08:47 | |
| *** csatari has quit IRC | 08:47 | |
| *** DuncanT has quit IRC | 08:47 | |
| *** arxcruz has quit IRC | 08:47 | |
| *** edwarnicke has quit IRC | 08:47 | |
| *** jcook has quit IRC | 08:47 | |
| *** mrmartin has quit IRC | 08:47 | |
| *** lastmikoi has quit IRC | 08:47 | |
| *** dulek has quit IRC | 08:47 | |
| *** gongysh has quit IRC | 08:47 | |
| *** LindaWang has quit IRC | 08:47 | |
| *** mpranjic has quit IRC | 08:47 | |
| *** aarefiev has quit IRC | 08:47 | |
| *** Guest93343 has quit IRC | 08:47 | |
| *** raissa has quit IRC | 08:47 | |
| *** peterlisak has quit IRC | 08:47 | |
| *** dimak has quit IRC | 08:47 | |
| *** deduped has quit IRC | 08:47 | |
| *** sambetts|afk has quit IRC | 08:47 | |
| *** rvba has quit IRC | 08:47 | |
| *** tristanC has quit IRC | 08:47 | |
| *** rhagarty has quit IRC | 08:47 | |
| *** timburke has quit IRC | 08:47 | |
| *** katyafervent_awa has quit IRC | 08:47 | |
| *** mika has quit IRC | 08:47 | |
| *** vtapia has quit IRC | 08:47 | |
| *** frickler has quit IRC | 08:47 | |
| *** toabctl has quit IRC | 08:47 | |
| *** freerunner has quit IRC | 08:47 | |
| *** Swimingly has quit IRC | 08:47 | |
| *** jeblair has quit IRC | 08:47 | |
| *** tonyb has quit IRC | 08:47 | |
| *** mhayden has quit IRC | 08:47 | |
| *** johnthetubaguy has quit IRC | 08:47 | |
| *** eventingmonkey has quit IRC | 08:47 | |
| *** Keitaro has quit IRC | 08:47 | |
| *** jlk has quit IRC | 08:47 | |
| *** ericyoung has quit IRC | 08:47 | |
| *** zer0c00l_ has quit IRC | 08:47 | |
| *** SotK has quit IRC | 08:47 | |
| *** zeroDivisible has quit IRC | 08:47 | |
| *** bradjones has quit IRC | 08:47 | |
| *** ggillies has quit IRC | 08:47 | |
| *** rtjure has quit IRC | 08:47 | |
| *** ramishra has quit IRC | 08:47 | |
| *** fbouliane has quit IRC | 08:47 | |
| *** adam_g has quit IRC | 08:47 | |
| *** pfallenop has quit IRC | 08:47 | |
| *** jpena|off has quit IRC | 08:47 | |
| *** eglute has quit IRC | 08:47 | |
| *** d34dh0r53 has quit IRC | 08:47 | |
| *** dhellmann has quit IRC | 08:47 | |
| *** jmccrory has quit IRC | 08:47 | |
| *** notmyname has quit IRC | 08:47 | |
| *** lucas-afk has quit IRC | 08:47 | |
| *** sai has quit IRC | 08:47 | |
| *** greghaynes has quit IRC | 08:47 | |
| *** petevg has quit IRC | 08:47 | |
| *** hogepodge has quit IRC | 08:47 | |
| *** madhuvishy has quit IRC | 08:47 | |
| *** dirk has quit IRC | 08:47 | |
| *** ildikov has quit IRC | 08:47 | |
| *** SamYaple has quit IRC | 08:47 | |
| *** dtantsur|afk has quit IRC | 08:47 | |
| *** tommylikehu has quit IRC | 08:47 | |
| *** geekinutah has quit IRC | 08:47 | |
| *** verdurin has quit IRC | 08:47 | |
| *** sulo has quit IRC | 08:47 | |
| *** evgenyl has quit IRC | 08:47 | |
| *** asilenkov has quit IRC | 08:47 | |
| *** StevenK has quit IRC | 08:47 | |
| *** dalvarez has quit IRC | 08:47 | |
| *** kencjohnston_ has quit IRC | 08:47 | |
| *** gaelL_ has quit IRC | 08:47 | |
| *** andrewbogott has quit IRC | 08:47 | |
| *** rguillebert has quit IRC | 08:47 | |
| *** ddurst has quit IRC | 08:47 | |
| *** zul has quit IRC | 08:47 | |
| *** sweston has quit IRC | 08:47 | |
| *** petems has quit IRC | 08:47 | |
| *** iyamahat has quit IRC | 08:47 | |
| *** florianf has quit IRC | 08:47 | |
| *** pgadiya has quit IRC | 08:47 | |
| *** rcernin has quit IRC | 08:47 | |
| *** xarses_ has quit IRC | 08:47 | |
| *** links has quit IRC | 08:47 | |
| *** dingyichen has quit IRC | 08:47 | |
| *** pbourke has quit IRC | 08:47 | |
| *** bswartz has quit IRC | 08:47 | |
| *** jaosorior has quit IRC | 08:47 | |
| *** liusheng has quit IRC | 08:47 | |
| *** mnaser has quit IRC | 08:47 | |
| *** fabo has quit IRC | 08:47 | |
| *** weshay_ has quit IRC | 08:47 | |
| *** mandre_mtg has quit IRC | 08:47 | |
| *** atarakt has quit IRC | 08:47 | |
| *** vponomaryov has quit IRC | 08:47 | |
| *** persia has quit IRC | 08:47 | |
| *** askb has quit IRC | 08:47 | |
| *** med_ has quit IRC | 08:47 | |
| *** alex_xu has quit IRC | 08:47 | |
| *** jogo has quit IRC | 08:47 | |
| *** raginbajin has quit IRC | 08:47 | |
| *** mdbooth has quit IRC | 08:47 | |
| *** larsks has quit IRC | 08:47 | |
| *** migi has quit IRC | 08:47 | |
| *** _ari_ has quit IRC | 08:47 | |
| *** aspiers has quit IRC | 08:47 | |
| *** trown|outtypewww has quit IRC | 08:47 | |
| *** pcaruana has quit IRC | 08:47 | |
| *** claudiub has quit IRC | 08:47 | |
| *** Qiming has quit IRC | 08:47 | |
| *** masayukig[m] has quit IRC | 08:47 | |
| *** thorre has quit IRC | 08:47 | |
| *** serverascode has quit IRC | 08:47 | |
| *** junbo has quit IRC | 08:47 | |
| *** d0ugal has quit IRC | 08:47 | |
| *** arif-ali has quit IRC | 08:47 | |
| *** oomichi has quit IRC | 08:47 | |
| *** robcresswell has quit IRC | 08:47 | |
| *** srwilkers_ has quit IRC | 08:47 | |
| *** mugsie has quit IRC | 08:47 | |
| *** zxiiro has quit IRC | 08:47 | |
| *** scottda has quit IRC | 08:47 | |
| *** coreycb has quit IRC | 08:47 | |
| *** patrickeast has quit IRC | 08:47 | |
| *** ameade has quit IRC | 08:47 | |
| *** samueldmq has quit IRC | 08:47 | |
| *** clayg has quit IRC | 08:47 | |
| *** mgkwill has quit IRC | 08:47 | |
| *** xgerman_ has quit IRC | 08:47 | |
| *** fenar has quit IRC | 08:47 | |
| *** zaro has quit IRC | 08:47 | |
| *** cargonza has quit IRC | 08:47 | |
| *** srwilkers has quit IRC | 08:47 | |
| *** vkmc has quit IRC | 08:47 | |
| *** hrybacki has quit IRC | 08:47 | |
| *** v1k0d3n has quit IRC | 08:47 | |
| *** ilpianista has quit IRC | 08:47 | |
| *** MasterofJOKers has quit IRC | 08:47 | |
| *** mgagne has quit IRC | 08:47 | |
| *** kwazar has quit IRC | 08:47 | |
| *** gus has quit IRC | 08:47 | |
| *** fungi has quit IRC | 08:47 | |
| *** leifmadsen has quit IRC | 08:47 | |
| *** tomhambleton_ has quit IRC | 08:47 | |
| *** davidlenwell has quit IRC | 08:47 | |
| *** lin_yang has quit IRC | 08:47 | |
| *** johnsom has quit IRC | 08:47 | |
| *** vdrok has quit IRC | 08:47 | |
| *** fkautz has quit IRC | 08:47 | |
| *** kklimonda has quit IRC | 08:47 | |
| *** NobodyCam has quit IRC | 08:47 | |
| *** zhongjun has quit IRC | 08:47 | |
| *** portdirect has quit IRC | 08:47 | |
| *** markus_z has quit IRC | 08:47 | |
| *** Odd_Bloke has quit IRC | 08:47 | |
| *** ruhe has quit IRC | 08:47 | |
| *** rodrigods has quit IRC | 08:47 | |
| *** isviridoz has quit IRC | 08:47 | |
| *** sdake_ has quit IRC | 08:47 | |
| *** Nakato has quit IRC | 08:47 | |
| *** bstinson has quit IRC | 08:47 | |
| *** markmcclain has quit IRC | 08:47 | |
| *** markmcd has quit IRC | 08:47 | |
| *** danpawlik has quit IRC | 08:47 | |
| *** mrunge has quit IRC | 08:47 | |
| *** hrubi has quit IRC | 08:47 | |
| *** zeus has quit IRC | 08:47 | |
| *** clarkb has quit IRC | 08:47 | |
| *** AJaeger_ has quit IRC | 08:47 | |
| *** anthonyper has quit IRC | 08:47 | |
| *** zaneb has quit IRC | 08:47 | |
| *** andreaf has quit IRC | 08:47 | |
| *** vaidy has quit IRC | 08:47 | |
| *** knikolla has quit IRC | 08:47 | |
| *** bmjen has quit IRC | 08:47 | |
| *** rm_work has quit IRC | 08:47 | |
| *** puiterwijk has quit IRC | 08:47 | |
| *** GregHouse has quit IRC | 08:47 | |
| *** pots has quit IRC | 08:47 | |
| *** mrhillsman has quit IRC | 08:47 | |
| *** ilpianista_ has quit IRC | 08:47 | |
| *** jhesketh has quit IRC | 08:47 | |
| *** dizquierdo has quit IRC | 08:47 | |
| *** mikal has quit IRC | 08:47 | |
| *** spotz has quit IRC | 08:47 | |
| *** gagehugo has quit IRC | 08:47 | |
| *** strigazi has quit IRC | 08:47 | |
| *** tobiash has quit IRC | 08:47 | |
| *** pleia2 has quit IRC | 08:47 | |
| *** jlvillal has quit IRC | 08:47 | |
| *** kevinbenton has quit IRC | 08:47 | |
| *** bexelbie has quit IRC | 08:47 | |
| *** admcleod has quit IRC | 08:47 | |
| *** jzerebec1i has quit IRC | 08:47 | |
| *** Daviey has quit IRC | 08:47 | |
| *** dansmith has quit IRC | 08:47 | |
| *** kazsh has quit IRC | 08:47 | |
| *** darvon has quit IRC | 08:47 | |
| *** jgriffith has quit IRC | 08:47 | |
| *** rnoriega has quit IRC | 08:47 | |
| *** bnemec has quit IRC | 08:47 | |
| *** kambiz has quit IRC | 08:47 | |
| *** timrc has quit IRC | 08:47 | |
| *** Hunner has quit IRC | 08:47 | |
| *** chandankumar has quit IRC | 08:47 | |
| *** adreznec has quit IRC | 08:47 | |
| *** Krenair has quit IRC | 08:47 | |
| *** yee37925 has quit IRC | 08:47 | |
| *** Jeffrey4l has quit IRC | 08:47 | |
| *** dmanchad has quit IRC | 08:47 | |
| *** alexchadin has quit IRC | 08:47 | |
| *** kornicameister has quit IRC | 08:47 | |
| *** aviau has quit IRC | 08:47 | |
| *** lennyb has quit IRC | 08:47 | |
| *** stevemar has quit IRC | 08:47 | |
| *** luzC has quit IRC | 08:47 | |
| *** calebb has quit IRC | 08:47 | |
| *** onovy has quit IRC | 08:47 | |
| *** hwoarang has quit IRC | 08:47 | |
| *** mdrabe has quit IRC | 08:47 | |
| *** stephenfin has quit IRC | 08:47 | |
| *** khappone has quit IRC | 08:47 | |
| *** dtroyer has quit IRC | 08:47 | |
| *** hyakuhei has quit IRC | 08:47 | |
| *** jklare has quit IRC | 08:47 | |
| *** therve has quit IRC | 08:47 | |
| *** jd_ has quit IRC | 08:47 | |
| *** mordred has quit IRC | 08:47 | |
| *** afazekas has quit IRC | 08:47 | |
| *** flwang has quit IRC | 08:47 | |
| *** hemna has quit IRC | 08:47 | |
| *** morgan has quit IRC | 08:47 | |
| *** Gorian has quit IRC | 08:47 | |
| *** briancurtin has quit IRC | 08:47 | |
| *** jamespage has quit IRC | 08:47 | |
| *** dham1 has quit IRC | 08:47 | |
| *** auggy has quit IRC | 08:47 | |
| *** kmARC has quit IRC | 08:47 | |
| *** neith has quit IRC | 08:47 | |
| *** jbryce has quit IRC | 08:47 | |
| *** ggherdov- has quit IRC | 08:47 | |
| *** liujiong has quit IRC | 08:47 | |
| *** zhurong has quit IRC | 08:47 | |
| *** nicolasbock has quit IRC | 08:47 | |
| *** gmann has quit IRC | 08:47 | |
| *** rhallisey has quit IRC | 08:47 | |
| *** xinliang has quit IRC | 08:47 | |
| *** clayton has quit IRC | 08:47 | |
| *** david-lyle has quit IRC | 08:47 | |
| *** lifeless has quit IRC | 08:47 | |
| *** xarses has quit IRC | 08:47 | |
| *** rkukura has quit IRC | 08:47 | |
| *** mudpuppy has quit IRC | 08:47 | |
| *** jistr has quit IRC | 08:47 | |
| *** openstackgerrit has quit IRC | 08:47 | |
| *** lindsayk has quit IRC | 08:47 | |
| *** spiffxp has quit IRC | 08:47 | |
| *** cinerama has quit IRC | 08:47 | |
| *** cshastri has quit IRC | 08:47 | |
| *** gtmanfred has quit IRC | 08:47 | |
| *** mtreinish has quit IRC | 08:47 | |
| *** lbragstad has quit IRC | 08:47 | |
| *** flaper87 has quit IRC | 08:47 | |
| *** jesusaur has quit IRC | 08:47 | |
| *** michaelxin has quit IRC | 08:47 | |
| *** bauzas has quit IRC | 08:47 | |
| *** rcarrillocruz has quit IRC | 08:47 | |
| *** hemanthm has quit IRC | 08:47 | |
| *** ajafo has quit IRC | 08:47 | |
| *** eeiden has quit IRC | 08:47 | |
| *** patriciadomin has quit IRC | 08:47 | |
| *** beagles has quit IRC | 08:47 | |
| *** sc` has quit IRC | 08:47 | |
| *** charz has quit IRC | 08:47 | |
| *** yuval has quit IRC | 08:47 | |
| *** szaher has quit IRC | 08:47 | |
| *** loquacities has quit IRC | 08:47 | |
| *** zerick has quit IRC | 08:47 | |
| *** smcginnis has quit IRC | 08:47 | |
| *** Vivek has quit IRC | 08:47 | |
| *** honza has quit IRC | 08:47 | |
| *** SpamapS has quit IRC | 08:47 | |
| *** Anticimex has quit IRC | 08:47 | |
| *** andreykurilin has quit IRC | 08:47 | |
| *** njohnston has quit IRC | 08:47 | |
| *** Guest6666 has quit IRC | 08:47 | |
| *** shu-mutou-AWAY has quit IRC | 08:47 | |
| *** ChanServ has quit IRC | 08:47 | |
| *** adisky__ has quit IRC | 08:47 | |
| *** lathiat_ has quit IRC | 08:47 | |
| *** Adri2000 has quit IRC | 08:47 | |
| *** vipul has quit IRC | 08:47 | |
| *** gaurangt has quit IRC | 08:47 | |
| *** Kevin_Zheng has quit IRC | 08:47 | |
| *** jbernard has quit IRC | 08:47 | |
| *** tdasilva has quit IRC | 08:47 | |
| *** mancdaz has quit IRC | 08:47 | |
| *** tbarron has quit IRC | 08:47 | |
| *** cmurphy has quit IRC | 08:47 | |
| *** rfolco has quit IRC | 08:47 | |
| *** melwitt has quit IRC | 08:47 | |
| *** zhenguo has quit IRC | 08:47 | |
| *** mrodden has quit IRC | 08:47 | |
| *** masayukig has quit IRC | 08:47 | |
| *** Zara has quit IRC | 08:47 | |
| *** petrovich has quit IRC | 08:47 | |
| *** gothicmindfood has quit IRC | 08:47 | |
| *** nhandler has quit IRC | 08:47 | |
| *** electrical has quit IRC | 08:47 | |
| *** rajinir has quit IRC | 08:47 | |
| *** jpmaxman has quit IRC | 08:47 | |
| *** amotoki has quit IRC | 08:47 | |
| *** Ng has quit IRC | 08:47 | |
| *** s-shiono has quit IRC | 08:56 | |
| *** sshnaidm has quit IRC | 09:30 | |
| *** openstack has joined #openstack-infra | 13:53 | |
| *** makowals has quit IRC | 13:56 | |
| *** felipemonteiro has quit IRC | 13:56 | |
| *** pgadiya has joined #openstack-infra | 13:57 | |
| *** makowals has joined #openstack-infra | 13:57 | |
| *** bobh has joined #openstack-infra | 13:58 | |
| *** markmcd has joined #openstack-infra | 13:59 | |
| *** Adri2000 has joined #openstack-infra | 14:05 | |
| *** Adri2000 has quit IRC | 14:05 | |
| *** Adri2000 has joined #openstack-infra | 14:05 | |
| *** camunoz has joined #openstack-infra | 14:05 | |
| *** LindaWang has joined #openstack-infra | 14:07 | |
| *** kgiusti has quit IRC | 14:09 | |
| *** marst has joined #openstack-infra | 14:09 | |
| *** pgadiya has quit IRC | 14:10 | |
| *** kgiusti has joined #openstack-infra | 14:10 | |
| openstackgerrit | Mohammed Naser proposed openstack-infra/system-config master: Add Fedora Atomic mirrors https://review.openstack.org/491800 | 14:16 |
| *** rbrndt has joined #openstack-infra | 14:18 | |
| *** felipemonteiro has joined #openstack-infra | 14:20 | |
| mnaser | is there any network issues @ infracloud vanilla? | 14:20 |
| mnaser | or perhaps over the past few days? | 14:21 |
| *** cshastri has quit IRC | 14:25 | |
| *** armax has joined #openstack-infra | 14:26 | |
| *** Goneri has quit IRC | 14:28 | |
| pabelanger | mnaser: ya, see scroll back from last night. We believe the cloud is under load | 14:30 |
| robcresswell | On my latest CI adventures, I'm getting errors about Java missing on the images. I'm assuming one of the nodepool elements handles this; any idea on what I should be adding? | 14:30 |
| *** EricGonczer_ has joined #openstack-infra | 14:30 | |
| mnaser | pabelanger need to adjust my znc scrollback but i'll check eavesdrop | 14:32 |
| *** zhurong has quit IRC | 14:32 | |
| *** psachin has quit IRC | 14:33 | |
| *** ykarel is now known as ykarel|afk | 14:33 | |
| pabelanger | mnaser: we think we are hitting a network bottleneck someplace, I believe the plan is to follow up with HPE and have them confirm some things | 14:33 |
| fungi | persia: smcginnis: trying to piece the electoral roll discussion between you together from comments in scrollback here and in #-dev but i think there must be another piece i'm missing still... what bug have you found, or do you suspect? i've done paranoid levels of regression testing over all changes to the owners.py script and want to know if something has escaped my notice :/ | 14:33 |
| *** jkilpatr has quit IRC | 14:33 | |
| *** jkilpatr has joined #openstack-infra | 14:34 | |
| *** pblaho has quit IRC | 14:34 | |
| mnaser | pabelanger i see. let's hope that gets figured it! | 14:34 |
| mnaser | is nodepool still uploading images to our cloud? | 14:34 |
| *** ralonsoh_ has quit IRC | 14:35 | |
| mnaser | we're experimenting with new dedicated cpu flavors and i think we can throw 15-30 instances to see how it copes | 14:35 |
| pabelanger | mnaser: yes, we have some uploads going still | 14:35 |
| mnaser | 2 dedicated cores / 8 gb memory | 14:35 |
| pabelanger | pushing almost 6 hours too | 14:35 |
| *** baoli has quit IRC | 14:35 | |
| *** slaweq_ has joined #openstack-infra | 14:35 | |
| fungi | pabelanger: mnaser: nodepool isn't uploading images to vexxhost afaik | 14:35 |
| fungi | hasn't for months | 14:35 |
| pabelanger | fungi: infracloud-vanilla | 14:36 |
| *** liujiong has joined #openstack-infra | 14:36 | |
| persia | fungi: Running `tox -evenv2 -- generate-rolls` on a fresh zesty install with openstack/election 2f9f1754e575150d9547b741acd54bf24135ded0 produced a documentation.yaml file including gerrit ID 11904 without a member: key. | 14:36 |
| fungi | pabelanger: i took mnaser's reference to "our cloud" to be him asking about vexxhost | 14:36 |
| pabelanger | fungi: well, we also have a nodepool-builder in vexxhost too, nb03. Which is currently uploading for 6 hours | 14:37 |
| *** makowals has quit IRC | 14:37 | |
| pabelanger | fungi: Ohh, now I see | 14:37 |
| pabelanger | fungi: ignore me | 14:38 |
| mnaser | yeah, i meat to our cloud :p | 14:38 |
| fungi | pabelanger: sure, but that would upload _from_ his cloud, not _to_ his cloud | 14:38 |
| pabelanger | ya, lack of coffee | 14:38 |
| mnaser | so its not a easy min-servers: x bump we can do? | 14:38 |
| robcresswell | mnaser: Check out irccloud, I was using ZNC for a while but have found it pretty useful to not have to think about it. Plus the android app is handy, since most of the OSS IRC apps seemed to be abandoned. | 14:38 |
| mnaser | robcresswell: ive heard so much about irccloud indeed | 14:38 |
| pabelanger | mnaser: fungi: Ya, we can add vexxhost back since we still have clouds.yaml setup for it | 14:38 |
| clarkb | mnaser: we could probably revert the old stuff back in and double chwck mirror is still there | 14:38 |
| pabelanger | I think we deleted mirror | 14:39 |
| mnaser | i'm doing some testing now, we're doing high perf flavors with dedicated cores, cpu pinning, numa pininng, etc | 14:39 |
| mnaser | so in a way id like to see performance characteristics as well as help move the gate a bit .. its a win win | 14:39 |
| fungi | persia: thanks, i'm going to see if i can tell why the lookup to the member api isn't returning an entry for any of those three e-mail addresses | 14:40 |
| pabelanger | http://mirror.ca-ymq-1.vexxhost.openstack.org/ | 14:40 |
| *** makowals has joined #openstack-infra | 14:40 | |
| pabelanger | cool, still running | 14:40 |
| *** slaweq_ has quit IRC | 14:40 | |
| mnaser | yep timestamp.txt seems up to date | 14:40 |
| mnaser | if we can add the nodepool stuff with min-servers to -1 and when we'll finish our tests/checks, i can make small bumps | 14:41 |
| pabelanger | mnaser: you can add it with max-server: 0 and images to start the upload process | 14:41 |
| mnaser | i didnt find it in nodepool.yml so i guess ill go through the commits and revert | 14:42 |
| * mnaser digs | 14:42 | |
| persia | fungi: There are more examples: if you want to hunt this down immediately, then I can put together an exception report (with reproduction instructions, analysis scripts, etc.). If the single example is enough to decide between "Aha!" and "Hrm.", then in the "Hrm" case, I'm happy to dig through it some (although that will likely take many days). | 14:43 |
| pabelanger | mnaser: 328395 was removal | 14:44 |
| mnaser | ah thanks | 14:44 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/project-config master: Revert "Remove vexxhost-ca-ymq-1 from nodepool" https://review.openstack.org/492159 | 14:44 |
| pabelanger | mnaser: you'll need to refresh the image list | 14:45 |
| fungi | persia: it sounds like you e-mailed the preferred addresses of the contributors with no matching member entry, and the exceptions are the ones who e-mailed you back saying their foundation memberships were current or had updated e-mail addresses? | 14:45 |
| mnaser | pabelanger ok ill edit that up locally and put min-servers: 0 too | 14:45 |
| pabelanger | mnaser: ++ | 14:46 |
| persia | fungi: I mailed the preferred addresses of the contributors that did not have member: entries that would have participated in one of the contested elections if there had been a member: field. I have had replies from about 10% of the mails I sent, 2/3 being "Thanks, fixed now", and 1/3 being "My stuff looks right. Are you sure this isn't a bug?". I also received bounces from a couple addresses. | 14:47 |
| *** gouthamr has joined #openstack-infra | 14:47 | |
| persia | The total volume of entries without member: fields was about 10% of the electorate. | 14:48 |
| *** alexchadin has quit IRC | 14:48 | |
| fungi | persia: cool, so the bounces are likely not a huge surprise (people move on over the course of the year) but i'll take smcginnis as an initial canary to see if i can work out what is in the foundation member directory for him | 14:48 |
| *** pblaho has joined #openstack-infra | 14:48 | |
| fungi | and yeah, that 10% has improved (it was more like 13% when i first e-mailed a warning to the various mailing lists about it) | 14:49 |
| mnaser | `name-filter` is a filter of the flavor name, right? | 14:50 |
| persia | fungi: http://paste.openstack.org/show/617940/ is the quick snippet I used to generate the lists to email, in case there is a bug in that. | 14:51 |
| *** tmorin has quit IRC | 14:51 | |
| fungi | persia: so far i've positively confirmed that the foundation directory api doesn't return a corresponding foundation membership for any of the three e-mail addresses smcginnis has in gerrit | 14:52 |
| *** iyamahat has joined #openstack-infra | 14:53 | |
| pabelanger | mnaser: yes | 14:53 |
| persia | Excellent. That bounds the bug. | 14:53 |
| fungi | a lookup by name in the directory webui suggests he should be member 19313 | 14:55 |
| fungi | i think i have some access to the backend db for that system, i just need to suss out the table schema | 14:55 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/project-config master: Revert "Remove vexxhost-ca-ymq-1 from nodepool" https://review.openstack.org/492159 | 14:56 |
| mnaser | lets start getting images uploaded and then i can expose that flavor and we can start trying things out | 14:57 |
| *** felipemonteiro has quit IRC | 14:58 | |
| pabelanger | dirk: are we ready to remove opensuse-422 from nodepool? | 14:59 |
| *** ykarel|afk is now known as ykarel | 15:01 | |
| *** rlandy|meetup has joined #openstack-infra | 15:01 | |
| fungi | persia: after some hunting in the foundation db, it does look like smcginnis has matching email addresses in his profile for both his gmail and huawei addresses. i'm going to engage the foundation web dev team to see if they can work out why that api is not returning a match | 15:02 |
| *** jkilpatr has quit IRC | 15:03 | |
| persia | fungi: Thanks. That is not the only false negative, but it is probably a good example, since it really isn't clear why it didn't match (at least to my naive understanding of text matches). | 15:04 |
| fungi | persia: right, i want to see if they can work out the problem with this one first and then we'll see if that fix translates to the remainder | 15:04 |
| fungi | also i received your forwarded e-mail, thanks! | 15:05 |
| persia | OK. I have only a small number of confirmed false negatives, but enough that we should be able to perform that test. | 15:05 |
| *** tmorin has joined #openstack-infra | 15:05 | |
| *** sree has quit IRC | 15:06 | |
| *** dizquierdo has quit IRC | 15:06 | |
| *** sree has joined #openstack-infra | 15:07 | |
| odyssey4me | fungi any chance we can take a peek at https://review.openstack.org/492118 before I leave for the day? | 15:08 |
| odyssey4me | we're having particular trouble in infracloud where the downloads are *super* slow, causing our jobs to fail rather often thanks to the diminished nodepool provider set right now | 15:08 |
| odyssey4me | so the reverse proxy facility would be a great help | 15:09 |
| *** sree has quit IRC | 15:10 | |
| *** sree has joined #openstack-infra | 15:10 | |
| *** rlandy|meetup has quit IRC | 15:11 | |
| fungi | odyssey4me: sure, looking now | 15:11 |
| pabelanger | http://us.images.linuxcontainers.org/lxc-images/ 404s for me | 15:11 |
| fungi | wonder if they just don't have autoindexing enabled or something | 15:12 |
| fungi | the index files like http://us.images.linuxcontainers.org/meta/1.0/index-system seem to work fine | 15:13 |
| andymccr | yeah - for some reason we're getting 20K/s downloads from linuxcontainers.org on infracloud hosts (others seem ok) - so the 80MB file takes quite some time to download (and we end up timing out). | 15:14 |
| fungi | http://us.images.linuxcontainers.org/images/ does have an autoindex | 15:14 |
| fungi | so i agree, seems like the lxc-images path may be wrong? | 15:14 |
| *** tesseract has quit IRC | 15:14 | |
| pabelanger | ya, think so | 15:15 |
| fungi | andymccr: we think we may only have about 25mbps aggregate bandwidth available for both infracloud chocolate and vanilla deployments combined, and on top of that the switches hpe is providing there seem to be falling back to flood forwarding (bridge tables may be full or something) | 15:15 |
| fungi | so slow downloads aren't surprising | 15:16 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: WIP: Add upload-pypi job https://review.openstack.org/491926 | 15:16 |
| fungi | we're hoping to get some details as to what's going on there today | 15:16 |
| *** links has quit IRC | 15:16 | |
| *** sree has quit IRC | 15:17 | |
| andymccr | fungi: ahh thats good to know its something that isn't specific to us only - and that its being looked into already! Thanks | 15:17 |
| odyssey4me | oh, my bad - it seems the index shows images for this | 15:17 |
| odyssey4me | lemme fix that up | 15:18 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/system-config master: Add images.linuxcontainers.org reverse proxy https://review.openstack.org/492118 | 15:18 |
| *** LindaWang has quit IRC | 15:19 | |
| odyssey4me | fungi ^ updated with the right path... I got it mixed up with our own private path settings for our custom images ;) | 15:20 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: External Calendar Sync https://review.openstack.org/487683 | 15:22 |
| *** martinkopec has quit IRC | 15:22 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test https://review.openstack.org/475747 | 15:25 |
| *** ralonsoh has joined #openstack-infra | 15:25 | |
| dirk | pabelanger: from a devstack perspective yes, but there are still tons of ci jobs defined that reference 422 | 15:25 |
| dirk | pabelanger: the migration isn't complete. I was actually wondering what to doabout that going forward | 15:26 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/subunit2sql master: Update requirements https://review.openstack.org/492174 | 15:26 |
| fungi | odyssey4me: lgtm now | 15:26 |
| pabelanger | dirk: for fedora, we usually just forward port all the jobs to the latest image | 15:27 |
| pabelanger | dirk: I think we can do the same for opensuse-422 | 15:27 |
| pabelanger | does opensuse have LTS? | 15:27 |
| dirk | pabelanger: well, the thing is that there are some voting jobs and I don't know whether the -423 equivalent is passing already | 15:27 |
| dirk | pabelanger: I'll need to look through it and assess teh state | 15:27 |
| dirk | there isn't LTS, but 42.2 is still maintained for another 12 months or so | 15:28 |
| *** markvoelker has quit IRC | 15:28 | |
| dirk | so its not dead, it is maintained in parallel to 42.3 | 15:28 |
| dirk | pabelanger: do you want to do that cleanup before the big pike branching kicks in? | 15:28 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade master: DNM: Ignore https://review.openstack.org/490948 | 15:28 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade master: DNM: Ignore https://review.openstack.org/490948 | 15:29 |
| pabelanger | dirk: Ya, I think we should only carry a single image, like we do in fedora. So we should find out how is using it and port to opensuse-423 | 15:30 |
| pabelanger | let me look | 15:30 |
| pabelanger | dirk: mostly looks like openstack-ansible jobs | 15:31 |
| pabelanger | but they are also doing opensuse-423 already | 15:31 |
| pabelanger | so I think we can just propose a patch to drop opensuse-422 and have PTL +1 | 15:31 |
| pabelanger | let me work on that now | 15:31 |
| robcresswell | Sorry, just bumping an earlier question again: On my latest CI adventures, I'm getting errors about Java missing on the images. I'm assuming one of the nodepool elements handles this; any idea on what I should be adding? Jenkins seems to fallback to trying to use oracle's java, but fails on account requirements -.- | 15:32 |
| robcresswell | Won't ask again, cause I don't want to spam the channel :) | 15:32 |
| openstackgerrit | Spyros Trigazis (strigazi) proposed openstack-infra/project-config master: [magnum] Move -nv test to experimental https://review.openstack.org/492177 | 15:32 |
| openstackgerrit | Bogdan Dobrelya proposed openstack-infra/tripleo-ci master: Rework the getthelogs helper script for wget recursive https://review.openstack.org/492178 | 15:33 |
| odyssey4me | fungi who else is best suited to review https://review.openstack.org/492118 ? | 15:33 |
| dirk | pabelanger: cool, thansk for that | 15:33 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade master: DNM: Ignore https://review.openstack.org/490948 | 15:33 |
| pabelanger | robcresswell: ya, it was removed from our elements because we don't use puppet any more. You'll need to add it back into the jenkins-slave element. We should consider adding a flag for jenkins users who still need it | 15:33 |
| dirk | pabelanger: my secret plan is to add tumbleweed btw as a 2nd image :) | 15:34 |
| *** rwsu has joined #openstack-infra | 15:34 | |
| *** ykarel_ has joined #openstack-infra | 15:34 | |
| fungi | odyssey4me: pabelanger had started looking it over, maybe he can take a second look now? | 15:35 |
| fungi | persia: smcginnis: so! the issue for this one, at least, seems to be that the most recent e-mail address update wasn't verified. when you enter an e-mail address into your member profile the foundation member system is supposed to send a token e-mail to that address and you have to follow the url or whatever to verify the address | 15:35 |
| robcresswell | pabelanger: Ah, gotcha, thanks again | 15:36 |
| jeblair | fungi: any word from ovh? | 15:36 |
| fungi | persia: smcginnis: the foundation web dev team has temporarily removed the implicit filter on the emailverified flag, so that account is turning up a member match now. others may have been in the same boat? | 15:36 |
| smcginnis | fungi: I did receive two of those last I changed something (even though I don't think I changed my email) and I think I did one of them but deleted the other. | 15:36 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Try to early terminate streaming on ansible errors https://review.openstack.org/491027 | 15:36 |
| *** ykarel has quit IRC | 15:37 | |
| *** slaweq_ has joined #openstack-infra | 15:37 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade master: DNM: Ignore https://review.openstack.org/490948 | 15:37 |
| persia | fungi: Oh, interesting. So, if people didn't confirm their email, it wouldn't match? | 15:37 |
| * persia will start a run to see how much the results differ | 15:38 | |
| *** aarefiev is now known as aarefiev_afk | 15:38 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Remove opensuse-422 from jobs https://review.openstack.org/492181 | 15:38 |
| pabelanger | dirk: ^ | 15:38 |
| *** markvoelker has joined #openstack-infra | 15:39 | |
| *** tmorin has left #openstack-infra | 15:40 | |
| fungi | jeblair: still no reply from jean-daniel to me, nor anything looking like an account reinstatement in our infra-root inbox for ovh | 15:41 |
| odyssey4me | thanks pabelanger | 15:41 |
| *** slaweq_ has quit IRC | 15:41 | |
| pabelanger | odyssey4me: fungi: +3. You'll also want to update project-config (configure-mirrors.sh) adding the new proxy to /etc/ci/mirror_info.sh | 15:41 |
| pabelanger | fungi: I've used the #ovh channel on freenode before to get some support, we could try there too | 15:42 |
| odyssey4me | pabelanger yeah, not sure exactly how that should be put in there because it's not use by apt, yum or zypper | 15:42 |
| odyssey4me | it would be a hostname/URL provided when doing lxc-create | 15:43 |
| jeblair | fungi: should i send an email too? | 15:43 |
| pabelanger | odyssey4me: ya, /etc/ci/mirror_info.sh would just have a varible with the proper mirror info and port for it | 15:43 |
| *** yamamoto has quit IRC | 15:43 | |
| fungi | persia: yes, by way of details, there's a single emailverified bool in the db for the account record, which is false if an e-mail verification has been triggered for at least one address and not completed. the query api was implicitly filtering matches to only return records where that flag was true. it's since been switched to ignore the emailverified column, and longer term they're adding a | 15:43 |
| fungi | separate explicit filter option for that in the api | 15:43 |
| pabelanger | odyssey4me: just a central place to source that info | 15:43 |
| openstackgerrit | Bogdan Dobrelya proposed openstack-infra/tripleo-ci master: Rework the getthelogs helper script for wget recursive https://review.openstack.org/492178 | 15:43 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Update Get Members Endpoint Filters https://review.openstack.org/492183 | 15:44 |
| persia | My thought is that people with complex email setups might have addresses which they only read in exceptional cases (debugging something), where verification messages there may not be read (or even noticed). | 15:44 |
| fungi | jeblair: maybe? i cc'd clarkb since he was on the previous thread for the similar situation last year but will gladly bounce you a copy of my message. it's possibel j-d is just on vacation this week or something too, i suppose | 15:44 |
| persia | Anyway, I should have results on the effect of that in about 5 hours. | 15:44 |
| pabelanger | andymccr: mind reviewing: https://review.openstack.org/492181 replacing opensuse-422 with opensuse-423 for jobs. | 15:45 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Update Get Members Endpoint Filters https://review.openstack.org/492183 | 15:45 |
| *** yamamoto has joined #openstack-infra | 15:45 | |
| *** yamamoto has quit IRC | 15:45 | |
| *** yamamoto has joined #openstack-infra | 15:46 | |
| fungi | persia: yeah, i can see us wanting to still filter on that longer term so that people can't falsely claim other people's gerrit contributions or something (i haven't quite worked through the risk potential on this), but regardless for now i agree we shouldn't filter out unverified addresses until we can more clearly communicate that requirement and the instructions on how to identify/complete pending | 15:46 |
| fungi | verifications | 15:46 |
| *** Apoorva has joined #openstack-infra | 15:47 | |
| persia | Yes. Key is folk being able to tell whether things are confirmed, or if there is an action they can take to resolve the problem. | 15:47 |
| fungi | jeblair: i can also try opening a trouble ticket i suppose. looking through the last time this happened i think it started with a trouble ticket (mainly because we didn't realize the account had been suspended) and then ended with j-d e-mailing several of us about the reinstatement | 15:47 |
| *** baoli has joined #openstack-infra | 15:47 | |
| fungi | persia: also, it may be useful to hand-query the api if you aren't already, like: https://openstackid-resources.openstack.org/api/public/v1/members?filter=group_slug==foundation-members&filter[]=email==fungi@yuggoth.org' | 15:49 |
| persia | I had not been. Thanks for that. This may be a shortcut for those people who may not take the time to reply to my email. | 15:50 |
| jeblair | fungi: maybe let's do both? :) if you want to bounce me a copy of your mail so i have the details, i'll write a new one | 15:50 |
| fungi | persia: er, i guess more correctly like: https://openstackid-resources.openstack.org/api/public/v1/members?filter[]=group_slug==foundation-members&filter[]=email==fungi@yuggoth.org' | 15:50 |
| fungi | the array bracketing may matter to the api backend, i can't quite tell | 15:50 |
| fungi | jeblair: sending now, just a sec | 15:51 |
| *** yamamoto has quit IRC | 15:51 | |
| andymccr | persia: of course - taking a look now | 15:51 |
| *** kjackal_ has quit IRC | 15:51 | |
| pabelanger | jeblair: fungi: do you mind a +3 on https://review.openstack.org/492159 mnaser wants to experiment again for potentially running jobs for openstack-infra again | 15:52 |
| pabelanger | dirk: https://review.openstack.org/492181 removal of opensuse-422 from jobs | 15:53 |
| robcresswell | What's the status with zuulv3 nowdays? Wondering if we should still be relying on jenkins from the puppet-openstackci setup, or looking to newer setups. | 15:55 |
| *** markvoelker has quit IRC | 15:56 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Remove base job https://review.openstack.org/491907 | 15:56 |
| fungi | robcresswell: i think the idea is to work on making puppet-openstackci recommend zuulv3 once we've switched to it in production upstream (hopefully at the ptg, per announcements to the -dev ml) | 15:57 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test https://review.openstack.org/475747 | 15:57 |
| *** yamamoto has joined #openstack-infra | 15:57 | |
| robcresswell | fungi: Ah cool, sorry I'll go chase the mailer before bugging any more. Thanks. | 15:57 |
| *** slaweq_ has joined #openstack-infra | 15:57 | |
| fungi | robcresswell: though we expect to still support zuul v2.x with jenkins for third-party users for a while. didn't want to recommend zuul v3 for third-party ci systems until we had the kinks mostly worked out and were sure we had something we could support in a stable fashion | 15:58 |
| *** Sukhdev_ has joined #openstack-infra | 15:58 | |
| robcresswell | fungi: Makes sense | 15:58 |
| *** vhosakot has joined #openstack-infra | 15:59 | |
| fungi | robcresswell: http://lists.openstack.org/pipermail/openstack-dev/2017-August/120499.html | 15:59 |
| *** Diabelko has joined #openstack-infra | 15:59 | |
| fungi | that's the announcement to which i was referring | 15:59 |
| Diabelko | I'm having an issue where Zuul completely ignores my layout.yaml file | 15:59 |
| Diabelko | nothing in the logs about it being incorrect, no errors, nothing | 16:00 |
| Diabelko | do you guys have any ideas how to debug something like that? | 16:00 |
| fungi | Diabelko: usually it starts with looking at the zuul-debug.log from around the time you start or reconfigure the service | 16:01 |
| *** yamamoto has quit IRC | 16:01 | |
| robcresswell | pabelanger: Any idea where that java dep comes from? Sorry, I'm at a loss. Can't see anything in the jenkins-slave element in older commits. | 16:02 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config master: Add mirror host for images.linuxcontainers.org reverse proxy https://review.openstack.org/492192 | 16:02 |
| odyssey4me | pabelanger fungi ^ hope I did the right thing | 16:02 |
| pabelanger | robcresswell: they were managed by openstack-infra/puppet-jenkins | 16:02 |
| *** markvoelker has joined #openstack-infra | 16:03 | |
| fungi | robcresswell: have you tried asking in #openstack-third-party-ci? i'm not sure how third-party ci operators have been getting the java dep for the jenkins slave agent preinstalled on their images these days (we stopped using puppet to configure our base images some months back) | 16:04 |
| robcresswell | fungi: Ah, sorry, didn't realise there was a more appropriate channel | 16:05 |
| robcresswell | Thanks | 16:05 |
| *** dtantsur is now known as dtantsur|brb | 16:05 | |
| jeblair | Diabelko: it may help to show us a copy of zuul.conf and layout.yaml, along with the debug log. | 16:05 |
| fungi | robcresswell: i don't know that it's necessarily always a more appropriate channel, but they do have some background in dealing with current challenges which we have only historical info for how we used to solve in the upstream ci back when we still used jenkins a year or so ago | 16:06 |
| robcresswell | gotcha | 16:06 |
| mmedvede | robcresswell: yes, it is a low traffic channel, I try to watch it for questions, you just got to hang in there to get an answer. We can discuss it there | 16:07 |
| robcresswell | mmedvede: Sure, I've posted. I can bug again later. I've got time :) | 16:08 |
| jeblair | fungi, clarkb: how should we proceed with infracloud? | 16:09 |
| pabelanger | odyssey4me: sorry, I -1 492118. I think we can just add http://us.images.linuxcontainers.org into images.linuxcontainers proxypass / proxypassreverse to simplify things | 16:09 |
| pabelanger | odyssey4me: /images is likely to generic | 16:10 |
| *** katkapilatova has left #openstack-infra | 16:10 | |
| odyssey4me | pabelanger unfortunately the URL path is not customisable | 16:11 |
| odyssey4me | from a client standpoint | 16:11 |
| odyssey4me | only the hostname can be changed, and hopefully the port with it - that's still to be verified | 16:11 |
| pabelanger | odyssey4me: what is the client? | 16:11 |
| odyssey4me | lxc-create with the 'download' template | 16:12 |
| Diabelko | fungi, jeblair: I've found zuul-server -t... basically verifies layout.yaml file | 16:12 |
| odyssey4me | https://github.com/lxc/lxc/blob/master/templates/lxc-download.in | 16:12 |
| fungi | jeblair: i can e-mail kevin coker (data center manager) who still owes me the photos/list/count of network connections for that environment, but he's the only direct contact i personally have there at the moment | 16:12 |
| Diabelko | I was upgrading zuul 2.0 -> 2.5 | 16:12 |
| *** ykarel_ is now known as ykarel|away | 16:12 | |
| odyssey4me | pabelanger specifically we can set https://github.com/lxc/lxc/blob/master/templates/lxc-download.in#L212 | 16:12 |
| fungi | jeblair: happy to cc you, clarkb and whoever else wants to be on that thread | 16:12 |
| jeblair | fungi: ok, worth a shot, thanks | 16:14 |
| *** lucasagomes is now known as lucas-afk | 16:14 | |
| fungi | pabelanger: when you switched yout +3 to a -1 on 492118 you forgot to drop your workflow approval. i just now removed my +2 so it won't merge | 16:15 |
| pabelanger | fungi: ah, thanks | 16:15 |
| *** jlibosva has quit IRC | 16:17 | |
| *** ykarel|away has quit IRC | 16:17 | |
| pabelanger | odyssey4me: so, if we set mirror.regionone.infracloud-vanilla.openstack.org:8080/images.linuxcontainers to proxy http://us.images.linuxcontainers.org I believe it should work. Because you'd pass mirror.regionone.infracloud-vanilla.openstack.org:8080/images.linuxcontainers as your server and it would actually hit: mirror.regionone.infracloud-vanilla.openstack.org:8080/images.linuxcontainers/images and | 16:17 |
| pabelanger | meta | 16:17 |
| pabelanger | actually, it would drop the http bits, but same logic applies | 16:18 |
| odyssey4me | pabelanger hmm, it's worth a try methinks | 16:19 |
| odyssey4me | yeah, that looks like it might work | 16:19 |
| odyssey4me | let's give it a go | 16:19 |
| *** thingee_ has joined #openstack-infra | 16:21 | |
| *** mjblack has joined #openstack-infra | 16:21 | |
| *** rwsu has quit IRC | 16:21 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/system-config master: Add images.linuxcontainers.org reverse proxy https://review.openstack.org/492118 | 16:21 |
| odyssey4me | pabelanger like so? ^ | 16:21 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config master: Add mirror host for images.linuxcontainers.org reverse proxy https://review.openstack.org/492192 | 16:22 |
| pabelanger | odyssey4me: ya, +2 | 16:22 |
| *** dizquierdo has joined #openstack-infra | 16:23 | |
| odyssey4me | fungi ^ :) asseblief | 16:23 |
| *** markvoelker has quit IRC | 16:23 | |
| odyssey4me | (means 'please' in afrikaans) | 16:23 |
| fungi | neat. i don't know enough afrikaans | 16:23 |
| *** e0ne has quit IRC | 16:24 | |
| *** florianf has quit IRC | 16:28 | |
| *** markvoelker has joined #openstack-infra | 16:32 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack-infra/tripleo-ci master: Rework the getthelogs helper script for wget recursive https://review.openstack.org/492178 | 16:33 |
| *** slaweq_ has quit IRC | 16:34 | |
| *** slaweq_ has joined #openstack-infra | 16:35 | |
| *** eharney has joined #openstack-infra | 16:37 | |
| *** rwsu has joined #openstack-infra | 16:38 | |
| *** slaweq_ has quit IRC | 16:38 | |
| *** slaweq_ has joined #openstack-infra | 16:39 | |
| *** baoli has quit IRC | 16:39 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: WIP: Add upload-pypi job https://review.openstack.org/491926 | 16:40 |
| *** kornicameister has quit IRC | 16:40 | |
| openstackgerrit | Merged openstack-infra/zuul-sphinx master: Import directives/roles from zuul https://review.openstack.org/491635 | 16:41 |
| *** rcernin has quit IRC | 16:41 | |
| *** baoli has joined #openstack-infra | 16:42 | |
| *** Nil_ has joined #openstack-infra | 16:44 | |
| *** Goneri has joined #openstack-infra | 16:44 | |
| *** kornicameister has joined #openstack-infra | 16:45 | |
| mordred | dirk: I think that's a great secret plan | 16:45 |
| *** markvoelker has quit IRC | 16:46 | |
| robcresswell | So, the issue that I *think* we've run into is that in a Jenkins + Nodepool world, this commit https://github.com/openstack-infra/project-config/commit/2fe9c2125dc57c1d5a60e1492afd5d483a6abe4c implies that the jenkins-slave element would give you an image that works with jenkins. As far as we can tell so far, it doesn't, since the extra packages aren't specified. This may be the result of us having a | 16:49 |
| robcresswell | version mismatch, I'm unsure. We've got a pretty standard puppet-openstackci setup with latest versions, but nodepools Xenial images are missing Java. | 16:49 |
| robcresswell | Just wanted to relay that from the discussion in #openstack-third-party-ci | 16:49 |
| *** sshnaidm is now known as sshnaidm|off | 16:50 | |
| dirk | mordred: shh, its secret ;) | 16:52 |
| *** sbezverk has joined #openstack-infra | 16:53 | |
| pabelanger | robcresswell: ya, we should update that to have a flag for install java. maybe NODEPOOL_JENKINS_SUPPORT, check if true and then install java package | 16:54 |
| *** markvoelker has joined #openstack-infra | 16:55 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-sphinx master: Mark zuul-sphinx as supporting python3 https://review.openstack.org/492203 | 16:57 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-sphinx master: Add CONTRIBUTING file to repo and to docs https://review.openstack.org/492204 | 16:57 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-sphinx master: Add dist dir to .gitignore https://review.openstack.org/492205 | 16:57 |
| *** yamamoto has joined #openstack-infra | 16:58 | |
| robcresswell | pabelanger: That would certainly be useful :) | 16:58 |
| *** jkilpatr has joined #openstack-infra | 16:59 | |
| *** spligak has joined #openstack-infra | 17:01 | |
| *** gcb has quit IRC | 17:02 | |
| *** xarses_ has joined #openstack-infra | 17:02 | |
| *** yamamoto has quit IRC | 17:03 | |
| *** krtaylor has quit IRC | 17:05 | |
| *** eharney has quit IRC | 17:08 | |
| *** liujiong has quit IRC | 17:08 | |
| *** david-lyle has quit IRC | 17:08 | |
| *** david-lyle has joined #openstack-infra | 17:08 | |
| *** Sukhdev_ has quit IRC | 17:09 | |
| *** iyamahat has quit IRC | 17:09 | |
| mnaser | this queue is rough :9 | 17:10 |
| mnaser | :( | 17:10 |
| *** trown is now known as trown|lunch | 17:11 | |
| *** jpena|off is now known as jpena | 17:12 | |
| robcresswell | mnaser: Day before RC1 I guess. Its like being in a florists the day before valentines day. | 17:12 |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: WIP: Add upload-pypi job https://review.openstack.org/491926 | 17:14 |
| dmsimard | robcresswell: TIL #openstack-third-party-ci is a thing | 17:14 |
| dmsimard | also, yes, we actually have an extra element that (amongst other things) installs java | 17:14 |
| robcresswell | dmsimard: I don't suppose you could share in paste.openstack.org or something? | 17:15 |
| robcresswell | dmsimard: Frustation levels have just about hit cruising altitude for us :) | 17:15 |
| dmsimard | robcresswell: hm, there isn't much worth sharing I think -- we basically just have a nodepool DIB element with a package-installs.yaml with Java in it and then that element is part of the image | 17:17 |
| robcresswell | dmsimard: Fair enough. Thanks for the input anyway, I think that's what we'll go with. | 17:17 |
| dmsimard | robcresswell: looks a bit like this: https://github.com/rdo-infra/review.rdoproject.org-config/tree/master/nodepool/elements/rdo-base | 17:17 |
| *** dtantsur|brb is now known as dtantsur | 17:18 | |
| *** EricGonczer_ has quit IRC | 17:19 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: WIP: Add upload-pypi job https://review.openstack.org/491926 | 17:20 |
| *** mriedem has quit IRC | 17:22 | |
| pabelanger | jeblair: fungi: should we consider disabling infracloud (or 1 region) until we hear back from HPE? We are getting a large number of jobs failing due to job timeouts | 17:23 |
| *** dizquierdo has quit IRC | 17:23 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: WIP: Add upload-pypi job https://review.openstack.org/491926 | 17:25 |
| *** iyamahat has joined #openstack-infra | 17:27 | |
| *** mriedem has joined #openstack-infra | 17:29 | |
| *** ralonsoh has quit IRC | 17:29 | |
| fungi | pabelanger: or at least severely chopping max-servers (in half maybe?). i'm working on an e-mail to kevin about it now | 17:31 |
| *** markvoelker has quit IRC | 17:31 | |
| pabelanger | fungi: sure, lets do 1/2 for both | 17:32 |
| *** sambetts is now known as sambetts|afk | 17:32 | |
| fungi | i have a feeling the primary issue is aggregate bandwidth to/from the internet, so there's probably some level of activity we can stabilize on which won't overtax the link for now | 17:33 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Reduce infracloud by 50% https://review.openstack.org/492207 | 17:34 |
| pabelanger | fungi: jeblair: ^. I can also place nodepool.o.o in emergency and manually apply until that merges | 17:34 |
| clarkb | if we stopped uploading new images we might cut out some bw usage too | 17:34 |
| pabelanger | since we're at 3+ hours to clear check ATM | 17:34 |
| *** jpena is now known as jpena|off | 17:35 | |
| pabelanger | clarkb: Ya, we have 16 uploads going right now | 17:35 |
| *** rbrndt has quit IRC | 17:35 | |
| pabelanger | clarkb: fungi: if we want to land https://review.openstack.org/#/c/492181/ that removes opensuse-422 from jobs, we can then remove it from nodepool | 17:36 |
| fungi | really, the backlog in zuul isn't all that terrible, considering. we've seen _far_ worse in past cycles. we just got spoiled with having so many more nodes available | 17:36 |
| pabelanger | will save us an image upload | 17:36 |
| clarkb | well we can pause them in those regions right? | 17:36 |
| clarkb | and save all the uploads | 17:36 |
| pabelanger | Yes | 17:37 |
| pabelanger | I can do that patch now too | 17:37 |
| *** kgiusti has quit IRC | 17:38 | |
| *** jkilpatr has quit IRC | 17:38 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Pause image uploads in infracloud (both regions) https://review.openstack.org/492208 | 17:39 |
| pabelanger | there | 17:39 |
| *** rcernin has joined #openstack-infra | 17:41 | |
| *** jkilpatr has joined #openstack-infra | 17:42 | |
| *** jamesdenton has quit IRC | 17:42 | |
| *** kornicameister has quit IRC | 17:43 | |
| *** sdague has quit IRC | 17:43 | |
| *** yamahata has joined #openstack-infra | 17:44 | |
| *** vhosakot_ has joined #openstack-infra | 17:45 | |
| *** kornicameister has joined #openstack-infra | 17:46 | |
| *** kgiusti has joined #openstack-infra | 17:46 | |
| *** vhosakot has quit IRC | 17:47 | |
| *** jamesdenton has joined #openstack-infra | 17:47 | |
| *** Goneri has quit IRC | 17:50 | |
| *** arxcruz is now known as arxcruz|off | 17:50 | |
| *** kgiusti has left #openstack-infra | 17:53 | |
| *** kgiusti has joined #openstack-infra | 17:53 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/openstack-zuul-jobs master: WIP: Add upload-pypi job https://review.openstack.org/491926 | 17:53 |
| *** Sukhdev_ has joined #openstack-infra | 17:54 | |
| *** electrofelix has quit IRC | 17:57 | |
| *** yamamoto has joined #openstack-infra | 17:59 | |
| *** cshastri has joined #openstack-infra | 18:00 | |
| *** sdague has joined #openstack-infra | 18:01 | |
| *** jpena|off is now known as jpena | 18:02 | |
| *** yamamoto has quit IRC | 18:04 | |
| *** Goneri has joined #openstack-infra | 18:05 | |
| *** jamesmcarthur has quit IRC | 18:07 | |
| fungi | jeblair: clarkb: i cc'd you on my e-mail to kevin at hpe just now | 18:09 |
| mgagne | I'm running Zuul 2.5.2. Is there a way to reimplement the reverify mechanism ? I found an old infra config but I'm having a hard time understanding the logic: http://paste.openstack.org/show/617969/ | 18:10 |
| *** felipemonteiro has joined #openstack-infra | 18:10 | |
| mgagne | is the "approved" approval now Workflow+1 ? | 18:10 |
| *** Goneri has quit IRC | 18:10 | |
| *** jrist has quit IRC | 18:11 | |
| fungi | mgagne: yes, that will probably do what we used to | 18:11 |
| mgagne | ok, will try and test then | 18:12 |
| jeblair | mgagne: should be -- nothing about zuul caused us to remove that, but rather we did so because we added the 'clean check' requirement, so it didn't make sense anymore. | 18:12 |
| *** felipemonteiro_ has joined #openstack-infra | 18:12 | |
| jeblair | (i should say: we added 'clean check' to our gate pipeline definition) | 18:12 |
| mgagne | yes, I understand the reasons and read fungi's recent explanation | 18:12 |
| jeblair | ah ok, background has been covered :) | 18:12 |
| mgagne | =) | 18:13 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test https://review.openstack.org/475747 | 18:13 |
| pabelanger | fungi: clarkb: I've promoted 491878,3 in the gate for tripleo, that fixes a gate reset issue they are having with networking-bagpipe project. Should help reduce wasting of nodepool resources | 18:13 |
| fungi | thanks pabelanger | 18:14 |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Remove vexxhost-ca-ymq-1 from nodepool" https://review.openstack.org/492159 | 18:15 |
| *** EricGonczer_ has joined #openstack-infra | 18:15 | |
| pabelanger | mnaser: ^ should see uploads starting in about 15-30mins | 18:15 |
| *** dtantsur is now known as dtantsur|afk | 18:15 | |
| *** felipemonteiro has quit IRC | 18:16 | |
| mnaser | pabelanger cool. i'm doing my final tests and should be able to bump it bit by bit (i also want to make sure we dont end up causing more problems) | 18:17 |
| *** trown|lunch is now known as trown | 18:18 | |
| mgagne | would this config be enough or is there something to change to gate pipeline too? http://paste.openstack.org/show/617971/ | 18:18 |
| *** ldnunes has quit IRC | 18:20 | |
| jeblair | mgagne: that config contains the 'clean check' requirement as well (gate.require.approval[0].verified)... what do you want 'reverify' to do? | 18:21 |
| mgagne | restart gate pipeline like the old good time ;) | 18:21 |
| jeblair | mgagne: do you want it to bypass check? | 18:21 |
| *** kzaitsev1pi has joined #openstack-infra | 18:22 | |
| mgagne | yes, assuming check passed once (in a not so distant past) | 18:22 |
| *** kzaitsev_pi has quit IRC | 18:22 | |
| *** e0ne has joined #openstack-infra | 18:22 | |
| jeblair | mgagne: that's not possible to know -- gerrit doesn't supply older votes, only the current one, so if it failed check or gate (presumably yes, as that's generally why you would want to reverify) then it'll have a negative verified vote | 18:23 |
| fungi | the way we used to infer it, if you read through that old config, is that if there's a verify -2 then that's evidence it was already in the gate | 18:24 |
| mgagne | ok, so can I assume -2 means check passed at least once but gate failed, hence the -2 vote? | 18:24 |
| fungi | right, that's basically what we did | 18:24 |
| jeblair | ya that works | 18:24 |
| mgagne | ok, should I then move the trigger to gate pipeline instead? | 18:24 |
| jeblair | mgagne: https://etherpad.openstack.org/p/ReLYD5K1wV | 18:24 |
| jeblair | hang on | 18:24 |
| *** shardy has quit IRC | 18:24 | |
| jeblair | mgagne: okay, i think those are the changes you'd want to make | 18:25 |
| jeblair | move the reverify trigger, but also relax the gate pipeline requirement | 18:26 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool feature/zuulv3: Allow launcher to stop quicker when asked https://review.openstack.org/492219 | 18:26 |
| mgagne | so move trigger to gate since pipelines are not longer trigger by Jenkins commenting. remove clean check from gate pipeline. ? | 18:26 |
| mgagne | I guess I understand the logic now =) | 18:26 |
| *** xinliang has quit IRC | 18:26 | |
| mgagne | tyvm, will test this config | 18:26 |
| jeblair | mgagne: yeah; you might be able to add -2 to the gate pipeline requirement instead of removing clean check completely. | 18:27 |
| mgagne | is the check an AND or OR ? | 18:28 |
| mgagne | I saw what you did | 18:28 |
| jeblair | mgagne: that may be worthwhile, as it would still prevent a verified-1 from check from going into gate, but still allow reverify to work on -2. | 18:28 |
| *** cshastri has quit IRC | 18:28 | |
| mgagne | excellent | 18:28 |
| *** EricGonczer_ has quit IRC | 18:29 | |
| *** EricGonczer_ has joined #openstack-infra | 18:29 | |
| *** jcoufal has quit IRC | 18:29 | |
| openstackgerrit | Jamey McCabe proposed openstack-infra/irc-meetings master: Adding new meeting for LCOO WG https://review.openstack.org/492220 | 18:29 |
| jeblair | mgagne: all pipeline requirements are AND (so all of the items under approval must match) | 18:29 |
| *** camunoz has quit IRC | 18:30 | |
| mgagne | but [] check if value is in the list =) | 18:30 |
| jeblair | right, the individual items may have some flexibility | 18:30 |
| jeblair | so the verified is an OR list | 18:30 |
| mgagne | cool, starting a dev stack to test that =) | 18:30 |
| *** rbrndt has joined #openstack-infra | 18:31 | |
| *** markvoelker has joined #openstack-infra | 18:32 | |
| *** ldnunes has joined #openstack-infra | 18:32 | |
| *** nicolasbock has quit IRC | 18:33 | |
| openstackgerrit | Jamey McCabe proposed openstack-infra/irc-meetings master: Adding new meeting for LCOO WG https://review.openstack.org/492220 | 18:34 |
| *** nicolasbock has joined #openstack-infra | 18:34 | |
| *** ldnunes has quit IRC | 18:39 | |
| *** vhosakot has joined #openstack-infra | 18:40 | |
| mnaser | pabelanger looks like uploads have started coming in | 18:42 |
| mnaser | i see really old images, will nodepool clean those up or would we have to do that at some point? | 18:42 |
| *** vhosakot_ has quit IRC | 18:42 | |
| openstackgerrit | Leticia Wanderley proposed openstack-infra/project-config master: Update LDAP domain driver CI job to run tempest full https://review.openstack.org/492223 | 18:43 |
| mnaser | nevermind, those were our nodepool images, aha. | 18:43 |
| *** jamesmcarthur has joined #openstack-infra | 18:44 | |
| pabelanger | ya, nothing yet | 18:44 |
| pabelanger | let me check puppet | 18:44 |
| *** morgan is now known as kmalloc | 18:46 | |
| pabelanger | mnaser: should start up in 15mins, puppet just running now | 18:46 |
| kklimonda | could someone familiar with zuul launcher tell me, is zuul_log ansible module used only in ansiblelaunchserver.py to write a job status in post_playbook? I have to add/hack windows support to zuul, and I'm evaluating what needs to be done and priorities. | 18:46 |
| fungi | kklimonda: the ansible launcher in 2.5 isn't really officially supported. it's an experimental feature which will be dropped in 3.0.0 | 18:49 |
| mhayden | mordred: did you find anything on the project-config version shenanigans from earlier? | 18:49 |
| *** nicolasbock has quit IRC | 18:50 | |
| fungi | kklimonda: and yes, all the ansible-related code in zuul 2.5.x is for the ansible launcher, so not necessary if you're not going to port it | 18:50 |
| *** eranrom has joined #openstack-infra | 18:50 | |
| kklimonda | fungi: what would be a more future-proof way to add windows support to 2.5, that will give me a corner stone for 3.0? | 18:51 |
| *** ldnunes has joined #openstack-infra | 18:51 | |
| fungi | kklimonda: you likely want to look at porting the feature/zuulv3 branch directly and separately if you want to get a head start before 3.0.0 is released. there are some substantial changes in configuration syntax and overall design/operation anyway | 18:52 |
| jeblair | kklimonda: oh we actually had some windows related discussion in #zuul earlier today | 18:52 |
| jeblair | kklimonda: if you want to join #zuul i can catch you up | 18:52 |
| fungi | agreed, that channel is probably a better venue for such discussions | 18:53 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/project-config master: Skip py27 job on nodepool features/zuulv3 branch https://review.openstack.org/492225 | 18:53 |
| Shrews | pabelanger: is that enough to drop py2 support? ^^^^ | 18:53 |
| Shrews | s/support/jobs/ | 18:53 |
| *** jamesmcarthur has quit IRC | 18:53 | |
| pabelanger | Shrews: Ya, think so | 18:55 |
| inc0 | hey guys, lots of our jobs times out:( | 18:55 |
| *** vhosakot has quit IRC | 18:55 | |
| Shrews | pabelanger: great. mordred, you'll probably be interested in that, too | 18:55 |
| pabelanger | inc0: yes, we are seeing a large amount of jobs timeout currently. http://status.openstack.org/elastic-recheck/#1686542 We're working to resolve it | 18:56 |
| inc0 | cool, let me know if I can help with anythign | 18:56 |
| inc0 | thank you guys | 18:56 |
| *** slaweq__ has joined #openstack-infra | 18:57 | |
| *** slaweq_ has quit IRC | 18:57 | |
| *** vhosakot has joined #openstack-infra | 18:57 | |
| *** yamamoto has joined #openstack-infra | 19:00 | |
| openstackgerrit | Ramamani Yeleswarapu proposed openstack-infra/project-config master: Enable TLS in ironic gate jobs except grenade https://review.openstack.org/492231 | 19:00 |
| *** felipemonteiro_ has quit IRC | 19:01 | |
| *** baoli has quit IRC | 19:01 | |
| *** baoli has joined #openstack-infra | 19:02 | |
| *** yamamoto has quit IRC | 19:05 | |
| *** markvoelker has quit IRC | 19:06 | |
| *** krtaylor has joined #openstack-infra | 19:08 | |
| *** ldnunes has quit IRC | 19:08 | |
| pabelanger | mnaser: and uploading now | 19:08 |
| *** markvoelker has joined #openstack-infra | 19:08 | |
| mnaser | pabelanger should be able to slowly bump it up once it's done. | 19:11 |
| mnaser | question: how does nodepool cope with instance types without ephermal/root disk? | 19:11 |
| fungi | mnaser: you mean like flavors that only support boot-from-volume? | 19:12 |
| mnaser | fungi yep | 19:12 |
| fungi | i don't think we've implemented boot-from-volume support in nodepool, though i could be mistaken | 19:12 |
| fungi | afaik we only support boot-from-image at the moment | 19:13 |
| mordred | pabelanger, Shrews: feel like +Aing this? https://review.openstack.org/#/c/491266/ | 19:13 |
| *** slaweq_ has joined #openstack-infra | 19:13 | |
| mnaser | fungi i am doing boot from image on a instance type with no root/ephermal disk | 19:13 |
| mnaser | and it's booting fine | 19:13 |
| mnaser | i wonder what size it gets | 19:14 |
| mnaser | i think it just uses the original image size | 19:14 |
| Shrews | mordred: done | 19:14 |
| pabelanger | mnaser: fungi: We do support boot-from-volume, but it would need to be backported to master. I know rdo is running it today for RDOcloud | 19:14 |
| *** baoli has quit IRC | 19:15 | |
| pabelanger | mnaser: fungi: https://review.openstack.org/464283 | 19:15 |
| mnaser | ok it looks like boot from image when no root/ephermal disk is configured results in it getting a root disk the size of the image | 19:15 |
| *** slaweq__ has quit IRC | 19:15 | |
| fungi | mnaser: okay, so maybe there's some background transparency which turns boot-from-image into image-to-volume,boot-from-volume i'm unfamiliar with | 19:15 |
| mnaser | hm, this might be an issue, how big is the root disk for the nodepool image builds? | 19:16 |
| fungi | mnaser: i don't believe we make it any larger than necessary to include the files we insert, and expect it to get resized to the backing block device size at boot | 19:17 |
| pabelanger | ya we have growroot setup on all our images | 19:17 |
| mnaser | well that's a slight bump | 19:18 |
| * mnaser thinks | 19:18 | |
| *** tnovacik has joined #openstack-infra | 19:20 | |
| *** adisky__ has quit IRC | 19:20 | |
| *** ldnunes has joined #openstack-infra | 19:20 | |
| openstackgerrit | Merged openstack-infra/project-config master: Skip py27 job on nodepool features/zuulv3 branch https://review.openstack.org/492225 | 19:22 |
| openstackgerrit | Merged openstack-infra/project-config master: Add QoS service plugin to be enabled in shade tests https://review.openstack.org/491266 | 19:24 |
| mordred | slaweq: ^^ | 19:24 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test https://review.openstack.org/475747 | 19:25 |
| *** sbezverk has quit IRC | 19:29 | |
| *** EricGonczer_ has quit IRC | 19:32 | |
| *** EricGonczer_ has joined #openstack-infra | 19:32 | |
| *** ociuhandu has joined #openstack-infra | 19:33 | |
| *** EricGonc_ has joined #openstack-infra | 19:33 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add images.linuxcontainers.org reverse proxy https://review.openstack.org/492118 | 19:34 |
| pabelanger | mnaser: in nodepool we can pass volume-size to shade, which should return a volume based on that value | 19:36 |
| *** EricGonczer_ has quit IRC | 19:36 | |
| mnaser | pabelanger is there a way to manually pass values like that to shade? | 19:37 |
| pabelanger | mnaser: ya, create_server takes both boot_from_volume and volume_size as args | 19:40 |
| pabelanger | we just pass those setting straight through | 19:40 |
| mnaser | pabelanger but that involves backporting that patch, right? | 19:41 |
| mnaser | because looking at the codebase it looks like everything is being passed strictly (no dict pass through or so) | 19:42 |
| mnaser | https://review.openstack.org/#/c/474607/ -- looks like that would need to merge for us to be able to do it | 19:43 |
| *** rmcall has joined #openstack-infra | 19:43 | |
| pabelanger | mnaser: ya, we could bring it up with other infra-root, but it is increases capacity, it might be something to consider doing | 19:45 |
| pabelanger | mnaser: I belive dmsimard is running that patch in production | 19:46 |
| openstackgerrit | Merged openstack-infra/zuul-sphinx master: Mark zuul-sphinx as supporting python3 https://review.openstack.org/492203 | 19:46 |
| dmsimard | we don't run anything in production | 19:46 |
| * dmsimard hides | 19:46 | |
| dmsimard | oh, boot from volume, yeah, we do that | 19:46 |
| mnaser | lol | 19:46 |
| mnaser | pabelanger i will create a custom flavor type with 80gb of root disk | 19:46 |
| mnaser | and i will update the filter | 19:46 |
| dmsimard | mnaser: it's built-in to SF 2.6 | 19:46 |
| mnaser | ooou nice | 19:47 |
| dmsimard | mnaser: https://github.com/rdo-infra/review.rdoproject.org-config/commit/a4b17ac546b9a4bb376a5af3319991aa4408b7cb | 19:47 |
| dmsimard | mnaser: you can see the patches carried by SF in the distgit repos | 19:47 |
| dmsimard | like https://github.com/softwarefactory-project/nodepool-distgit and https://github.com/softwarefactory-project/zuul-distgit | 19:47 |
| *** funzo has joined #openstack-infra | 19:47 | |
| * mnaser needs to learn how to properly carry patches with distgits soon | 19:47 | |
| *** Sukhdev_ has quit IRC | 19:48 | |
| mnaser | anyone in infra-root can verify if v1-performance-2-openstackci is visible in the `openstackci` tenant? | 19:50 |
| pabelanger | sure, give me a second | 19:51 |
| mnaser | pabelanger actually, ill rename it to v1-performance-2-80gb .. so we can reuse it, sorry one second | 19:52 |
| *** baoli has joined #openstack-infra | 19:52 | |
| mnaser | ok it should be there now as v1-performance-2-80gb | 19:52 |
| pabelanger | mnaser: I don't see it under flavor list | 19:53 |
| pabelanger | just v1 / v2 standard | 19:53 |
| mnaser | pabelanger do you mind confirming the project id you have? | 19:53 |
| pabelanger | sure | 19:53 |
| *** jamesmcarthur has joined #openstack-infra | 19:53 | |
| *** rwsu_ has joined #openstack-infra | 19:53 | |
| mnaser | i think i gave it to the openstackci tenant, not openstackjenkins (the one that runs the jobs) | 19:54 |
| pabelanger | | 86bbbcfa8ad043109d2d7af530225c72 | openstackjenkins | | 19:54 |
| mnaser | yeah | 19:54 |
| mnaser | pabelanger should be good now | 19:54 |
| mnaser | let me know if you see it | 19:54 |
| *** baoli has quit IRC | 19:55 | |
| pabelanger | mnaser: yup, see it | 19:55 |
| *** baoli has joined #openstack-infra | 19:55 | |
| mnaser | ok ill submit a bump to max-servers: 10 and appropriate name-filter (in case it breaks things, because we'll have diskless instance types) | 19:55 |
| pabelanger | ack | 19:56 |
| openstackgerrit | Leticia Wanderley proposed openstack-infra/project-config master: Update LDAP domain driver CI job to run tempest full https://review.openstack.org/492223 | 19:56 |
| pabelanger | ianw: so, I'd like to see if we could move this into devstack http://git.openstack.org/cgit/openstack-infra/system-config/tree/install_puppet.sh#n166 and do what we do today for jobs that need EPEL. | 19:57 |
| *** rwsu has quit IRC | 19:57 | |
| pabelanger | ianw: cause I think those packages are installed a job runtime, not DIB creation? | 19:57 |
| *** priteau has quit IRC | 19:58 | |
| openstackgerrit | Mohammed Naser proposed openstack-infra/project-config master: Update instance type and bump vexxhost max-servers to 10 https://review.openstack.org/492240 | 19:58 |
| mnaser | pabelanger ^ feel free to merge once uploads are done. i'll be around to make sure this goes up smoothly | 19:58 |
| pabelanger | looking | 19:58 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test https://review.openstack.org/475747 | 19:58 |
| *** eranrom has quit IRC | 20:00 | |
| pabelanger | fungi: clarkb: ^ cause to look at 492240. Adds 10 nodes back to vexxhost | 20:02 |
| *** yamamoto has joined #openstack-infra | 20:02 | |
| *** sbezverk has joined #openstack-infra | 20:04 | |
| *** yamamoto has quit IRC | 20:06 | |
| *** kjackal_ has joined #openstack-infra | 20:09 | |
| *** slaweq_ has quit IRC | 20:10 | |
| mnaser | pabelanger would it be possible to promote 492240 so that we can be here when it merges to make sure everything is going okay? | 20:14 |
| mnaser | rather than it merging sometime much later and causing issues | 20:14 |
| *** slaweq_ has joined #openstack-infra | 20:14 | |
| fungi | mnaser: i can enqueue it into the gate now, sure | 20:15 |
| openstackgerrit | Merged openstack-infra/project-config master: Pause image uploads in infracloud (both regions) https://review.openstack.org/492208 | 20:16 |
| pabelanger | fungi: mnaser: wfm | 20:16 |
| mnaser | fungi thank you | 20:16 |
| fungi | mnaser: thank YOU! | 20:17 |
| mnaser | well, we needed someone to do some workloads to test some new instance types and their performance... :p | 20:18 |
| fungi | #status log OVH ticket 8344470555 has been opened to track voucher reinstatement/refresh | 20:18 |
| openstackstatus | fungi: finished logging | 20:18 |
| fungi | infra-root: ^ | 20:18 |
| pabelanger | clarkb: so looking at buildlogs reverse proxy more, I think we need to do the same thing we did for docker. I think we are getting redirected to their CDN network and not actually caching things | 20:18 |
| *** jrich has joined #openstack-infra | 20:19 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Also add buildlogs.cdn.centos.org https://review.openstack.org/492256 | 20:23 |
| pabelanger | fungi: Thanks! | 20:23 |
| pabelanger | infra-root: ^ I'd like to also add buildlogs.cdn.centos.org to our reverse proxy cache and experiment using it. I am seeing a lot of 302 redirects in our apache access logs ATM and fear we are not properly caching those requests | 20:24 |
| *** links has joined #openstack-infra | 20:25 | |
| mtreinish | fungi: if you get a sec: https://review.openstack.org/#/c/491615/ it's a straightforward doc patch | 20:25 |
| openstackgerrit | Merged openstack-infra/project-config master: Update instance type and bump vexxhost max-servers to 10 https://review.openstack.org/492240 | 20:25 |
| fungi | mnaser: pabelanger: ^ should take effect in the next 15-30 minutes i hope | 20:25 |
| pabelanger | ++ | 20:26 |
| mnaser | https://github.com/openstack-infra/system-config/blob/master/modules/openstack_project/manifests/puppetmaster.pp | 20:27 |
| mnaser | so in 3 minutes i guess? :p | 20:27 |
| pabelanger | assuming puppet isn't still running | 20:27 |
| pabelanger | ya, puppet running still, so we'll end up missing this loop | 20:28 |
| *** slaweq__ has joined #openstack-infra | 20:28 | |
| fungi | unless it wraps up in the next minute anyway | 20:28 |
| pabelanger | actually, if we haven't hit nodepool.o.o yet, it will go live sooner | 20:29 |
| pabelanger | since we git clone | 20:29 |
| *** vhosakot has quit IRC | 20:29 | |
| fungi | oh, fair, this being a project-config change it's not delayed to the next run like something in the puppet manifest would be | 20:29 |
| *** sbezverk has quit IRC | 20:29 | |
| *** vhosakot has joined #openstack-infra | 20:30 | |
| pabelanger | ya | 20:30 |
| *** bobh has quit IRC | 20:30 | |
| *** slaweq_ has quit IRC | 20:31 | |
| *** vhosakot has quit IRC | 20:32 | |
| *** bobh has joined #openstack-infra | 20:33 | |
| *** markvoelker has quit IRC | 20:35 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/infra-specs master: Gerrit ContactStore Removal is implemented https://review.openstack.org/492287 | 20:39 |
| pabelanger | Ah, we need to run cloud launcher against vexxhost | 20:40 |
| fungi | oh, to get our networks set? | 20:40 |
| pabelanger | we need to add infra-root ssh key | 20:40 |
| fungi | right, that | 20:40 |
| pabelanger | 1 sec, looking | 20:40 |
| fungi | we should probably get a procedure written up for provider additions so we have a checklist | 20:41 |
| pabelanger | ya, I completely forgot about it this time | 20:41 |
| *** markvoelker has joined #openstack-infra | 20:41 | |
| fungi | we have more on the way in coming weeks/months, and would be great to have other people act as root coordinators on those | 20:41 |
| fungi | for example, i haven't actually handled one yet | 20:42 |
| fungi | so would end up pestering those of you who have for a walkthrough | 20:42 |
| *** marst_ has joined #openstack-infra | 20:44 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test https://review.openstack.org/475747 | 20:44 |
| fungi | which i suppose i can still do, and then just document the steps i follow as i fumble my way through | 20:44 |
| *** kgiusti has left #openstack-infra | 20:46 | |
| *** marst has quit IRC | 20:47 | |
| *** baoli has quit IRC | 20:48 | |
| *** baoli has joined #openstack-infra | 20:49 | |
| *** markvoelker has quit IRC | 20:52 | |
| pabelanger | odd, getting 401 from dynamic inventory | 20:53 |
| pabelanger | we must have wrong credentials for a cloud | 20:54 |
| *** baoli has quit IRC | 20:55 | |
| *** links has quit IRC | 20:56 | |
| mnaser | pabelanger maybe suspended ovh accounts? | 20:56 |
| *** jpena is now known as jpena|off | 20:56 | |
| pabelanger | started failling everytime on 2017-07-26 | 20:58 |
| *** markvoelker has joined #openstack-infra | 20:59 | |
| *** trown is now known as trown|outtypewww | 20:59 | |
| *** slaweq_ has joined #openstack-infra | 21:00 | |
| *** jkilpatr has quit IRC | 21:01 | |
| *** krtaylor has quit IRC | 21:01 | |
| *** yamamoto_ has joined #openstack-infra | 21:03 | |
| *** slaweq__ has quit IRC | 21:04 | |
| *** krtaylor has joined #openstack-infra | 21:04 | |
| *** yamamoto_ has quit IRC | 21:08 | |
| *** arxcruz|off is now known as arxcruz | 21:10 | |
| *** rbrndt has quit IRC | 21:11 | |
| pabelanger | fungi: mnaser: okay, I have manually run cloud-launcher. Incoming patch in a moment, going to try and trace down are bad credentials | 21:12 |
| *** EricGonc_ has quit IRC | 21:14 | |
| *** dougwig has joined #openstack-infra | 21:16 | |
| *** ldnunes has quit IRC | 21:16 | |
| mnaser | pabelanger ok i see VMs are going up and also deletions | 21:16 |
| *** yamamoto_ has joined #openstack-infra | 21:16 | |
| mnaser | is everything going okay or is it likely failing and deleting them? | 21:16 |
| *** e0ne has quit IRC | 21:16 | |
| *** rtjure has quit IRC | 21:17 | |
| mnaser | dang, didnt specify raw image type | 21:17 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Remove internap-nyj01 https://review.openstack.org/492301 | 21:17 |
| mnaser | so launches are so slooow | 21:17 |
| pabelanger | mnaser: dns failed for one | 21:18 |
| pabelanger | Host git.openstack.org not found: 3(NXDOMAIN)^ | 21:19 |
| mnaser | that's weird. they're launched on the public network? | 21:19 |
| pabelanger | checking network settings now | 21:19 |
| mtreinish | pabelanger: is this something we're tracking somewhere: http://logs.openstack.org/15/491615/1/gate/gate-infra-puppet-apply-3-centos-7/548215f/puppetapplytest29.final.out.FAILED ? | 21:19 |
| mtreinish | I do like that it's failing on pleia2's key | 21:19 |
| pabelanger | mtreinish: will have to check, I think it is something to do with puppet-jenkins and | 21:20 |
| pabelanger | hardcoded file | 21:20 |
| mtreinish | my docs change failed on that | 21:20 |
| *** yamamoto_ has quit IRC | 21:21 | |
| pleia2 | you're welcome | 21:21 |
| pabelanger | mnaser: looks like public network? I see both ipv6 and ipv4 | 21:22 |
| pabelanger | going to manually boot a node and see | 21:22 |
| mnaser | pabelanger i cant imagine why dns would fail because afaik fungi mentioned you run unbound and it seems to forward to opendns/google dns | 21:23 |
| pabelanger | mnaser: right, I can see we are using it | 21:23 |
| mnaser | security groups? | 21:23 |
| * mtreinish tries a blind recheck | 21:23 | |
| pabelanger | numans: shouldn't be, cloud launcher should be managing it | 21:23 |
| pabelanger | numans: sorry | 21:23 |
| mnaser | i guess thats for me :p | 21:23 |
| pabelanger | mnaser: we open default rules | 21:24 |
| mnaser | hmm | 21:24 |
| *** aeng has joined #openstack-infra | 21:24 | |
| pabelanger | booting instances now | 21:26 |
| *** jamesmcarthur has quit IRC | 21:26 | |
| mnaser | i'll have to submit a change to get raw images to speed up boots and cow efficency but ill do that once we solve this | 21:26 |
| *** derekh has quit IRC | 21:29 | |
| *** askb has joined #openstack-infra | 21:31 | |
| *** thorst has quit IRC | 21:31 | |
| *** rbrndt has joined #openstack-infra | 21:33 | |
| pabelanger | okay, something is odd, our public keys are not getting added to root user | 21:34 |
| mnaser | pabelanger do you depend on configdrive or metadata service | 21:36 |
| *** slaweq_ has quit IRC | 21:36 | |
| pabelanger | configdrive | 21:36 |
| mnaser | do you tag images with configdrive as required | 21:36 |
| pabelanger | I don't think so | 21:36 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: WIP: containers periodic test https://review.openstack.org/475747 | 21:36 |
| mnaser | that might be it | 21:36 |
| pabelanger | just pass configdrive=true to create_server | 21:36 |
| mnaser | oh if you pass that then it should provision it | 21:37 |
| pabelanger | let me log in via ze01 | 21:37 |
| mnaser | yep, i see the config drive attached | 21:37 |
| mnaser | so that should be okay | 21:37 |
| mnaser | pabelanger oh maybe you didnt pass configdrive when you just booted your test instance? | 21:38 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/project-config master: switch vexxhost to raw images https://review.openstack.org/492304 | 21:41 |
| pabelanger | mnaser: config-drive: true is setup in nodepool for vexxhost. I'm guess glean isn't running for some reason | 21:41 |
| mnaser | i think glean is running it because clearly zuul was able to login and launch a job | 21:42 |
| mnaser | i think when you tried to locally boot a vm (i assume using openstack server create) it didnt have the config drive option in the line and therefore didnt let you sign in? | 21:43 |
| *** priteau has joined #openstack-infra | 21:47 | |
| *** priteau has quit IRC | 21:47 | |
| *** esberglu has quit IRC | 21:48 | |
| *** esberglu has joined #openstack-infra | 21:48 | |
| *** Sukhdev_ has joined #openstack-infra | 21:48 | |
| *** thorst has joined #openstack-infra | 21:50 | |
| *** EricGonczer_ has joined #openstack-infra | 21:51 | |
| *** gouthamr has quit IRC | 21:52 | |
| *** esberglu has quit IRC | 21:53 | |
| *** thorst has quit IRC | 21:55 | |
| pabelanger | mnaser: we still bake in the zuul user SSH key, so that is why nodepool is able to log in | 21:55 |
| fungi | pabelanger: does glean actually do anything if it sees working dhcp? | 21:56 |
| fungi | just wondering if we're short-circuiting once working dhcp is concerned | 21:57 |
| fungi | er, confirmed | 21:57 |
| *** yamamoto_ has joined #openstack-infra | 21:57 | |
| *** EricGonczer_ has quit IRC | 21:58 | |
| mnaser | fungi https://github.com/openstack-infra/glean/blob/master/glean/init/glean.sh#L44-L57 ? | 21:58 |
| *** EricGonczer_ has joined #openstack-infra | 21:58 | |
| pabelanger | so, glean didn't mount configdrive for some reason | 21:59 |
| mnaser | oh i have a possible reason for you | 21:59 |
| fungi | aha, so it's the other way around. we try to use configdrive, and fall back on dhcp otherwise | 21:59 |
| mnaser | nvm, we used to run a different config_drive_format before but not anymore | 22:00 |
| *** arturb has quit IRC | 22:00 | |
| mnaser | so it should be default | 22:00 |
| *** xyang1 has quit IRC | 22:00 | |
| persia | fungi: Changing the verified flag status caused 8 additional people in the projects of interest to be tagged as members. I still have a false negative (someone I know to be a foundation member, where the email address matches), although our canary no longer appears. | 22:00 |
| *** arturb has joined #openstack-infra | 22:01 | |
| fungi | persia: what are the member id and gerrit id numbers for that false negative? | 22:01 |
| fungi | i'll see if i can spot anything out of the ordinary in the backend | 22:02 |
| mnaser | pabelanger fungi we use vfat config drives | 22:03 |
| mnaser | /dev/vdb: SEC_TYPE="msdos" LABEL="CONFIG-2" UUID="669B-C8F4" TYPE="vfat" | 22:04 |
| mnaser | the label is all uppercase :( | 22:04 |
| fungi | mnaser: that might explain it | 22:04 |
| *** vipul has quit IRC | 22:04 | |
| mnaser | (there is a longstanding bug that doesnt let us live migrate with without vfat config drives) | 22:04 |
| mnaser | blkid -t LABEL="config-2" returns nothing | 22:04 |
| persia | fungi: If I'm reading the output correctly, it's gerrit 11692, member 19118. If that doesn't work, asking the foundation API for the preferred email address for that gerrit ID does return a result. | 22:04 |
| *** xarses has quit IRC | 22:04 | |
| mnaser | so its just running it with dhcp only | 22:04 |
| fungi | persia: checking, thanks | 22:05 |
| *** xarses has joined #openstack-infra | 22:05 | |
| mnaser | i mean, that still doesn't explain why dns resolution is failing | 22:05 |
| *** dklyle has joined #openstack-infra | 22:06 | |
| *** david-lyle has quit IRC | 22:06 | |
| *** adriant has joined #openstack-infra | 22:08 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Bind secrets to their playbooks https://review.openstack.org/492307 | 22:09 |
| *** pcrews has quit IRC | 22:09 | |
| fungi | persia: looks like that user has a foundation account but no foundation membership. i'll get you a useful query with additional detail | 22:09 |
| fungi | persia: so their foundation account id number is definitely 19118 but probably only community member or speaker or something | 22:10 |
| *** vipul has joined #openstack-infra | 22:10 | |
| fungi | i'll know in a sec | 22:10 |
| *** spiffxp has quit IRC | 22:10 | |
| pabelanger | fungi: mnaser: we seem to be only forwarding to ipv6 for unbound | 22:11 |
| *** pcrews has joined #openstack-infra | 22:12 | |
| fungi | pabelanger: i believe it tries to use ipv6 and if that doesn't work falls back to ipv4 | 22:12 |
| *** esberglu has joined #openstack-infra | 22:12 | |
| fungi | configuration-wise | 22:12 |
| pabelanger | nope | 22:12 |
| fungi | does the instance have a global v6 address/route? | 22:12 |
| pabelanger | we just add IPv6onlye | 22:12 |
| mnaser | instances have ipv6 addresses on our cloud | 22:13 |
| fungi | yeah, i mean if the instance has working v6 we set it up to query over v6, otherwise we set it up to query over v4 | 22:13 |
| pabelanger | fungi: no, we just only have ipv6 for forward addr: no ipv6 | 22:13 |
| *** spiffxp has joined #openstack-infra | 22:13 | |
| pabelanger | http://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/scripts/configure_mirror.sh#n38 | 22:13 |
| mnaser | o | 22:13 |
| fungi | because in places where we have global v6 but all ipv4 routed through an overload nat, we generally get better performance over v6 | 22:13 |
| pabelanger | that should be updated to include ipv4 too | 22:14 |
| clarkb | no | 22:14 |
| mnaser | that just means it will forward requests over ipv6 if i understand correctly? | 22:14 |
| clarkb | if you add ipv4 it is used too and we dont want that because nat | 22:14 |
| mnaser | ex: an A record will be looked up over IPv6 | 22:14 |
| clarkb | mnaser: yes | 22:14 |
| fungi | that's what we want if there's working v6 | 22:14 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove 'auth' dict from jobs https://review.openstack.org/492309 | 22:14 |
| mnaser | we did have to shutoff one of our ipv6 peers because of problems | 22:15 |
| mnaser | let me see if maybe its unreachable? | 22:15 |
| mnaser | one of our other ipv6 peers is *sigh* cogent | 22:15 |
| mnaser | dig -t a git.openstack.org @2620:0:ccc::2 responds from one of the VMs i have here so it should be reachable | 22:15 |
| mnaser | is /etc/resolv.conf configured with 127.0.0.1 (maybe thats the issue because the configdrive stuff is failing?) | 22:16 |
| fungi | persia: try querying like: wget -qO- 'https://openstackid-resources.openstack.org/api/public/v1/members?expand=groups&filter[]=email==fungi@yuggoth.org' and if you sub in their address for mine you'll see they're only in group 29 "Community Members" while i'm in lots of others (including, most importantly for this, group 5 "Foundation Members") | 22:17 |
| fungi | persia: the query we use in owners.py includes filter[]=group_slug==foundation-members which will cause it not to return non-members (the whole reason we're querying, in fact) | 22:18 |
| *** kjackal_ has quit IRC | 22:18 | |
| persia | fungi: Excellent, so what we have here is a case of user confusoin over membership status, rather than a false negative. Perfect. The other false negative was fixed by the change, and all the "thanks, I've updated" folk seem to have been updated. | 22:19 |
| persia | You have restored my faith in the depth of your paranoia :) | 22:20 |
| fungi | persia: awesome! thanks for taking the time for running those down | 22:20 |
| *** rcernin has quit IRC | 22:20 | |
| *** Apoorva_ has joined #openstack-infra | 22:20 | |
| *** sbezverk has joined #openstack-infra | 22:20 | |
| fungi | hey, i don't even trust myself to have sufficient paranoia for something important like this, which is why i love having others double-check | 22:20 |
| pabelanger | mnaser: no, we default to 127.0.0.1 to hit unbound | 22:20 |
| mnaser | thats so strange why git.openstack.org would return NXDOMAIN then | 22:21 |
| mnaser | dig -t a git.openstack.org @2620:0:ccc::2 on that machine responds correctly? | 22:21 |
| *** jkilpatr has joined #openstack-infra | 22:22 | |
| *** Apoorva has quit IRC | 22:23 | |
| fungi | unless unbound somehow managed to cache a negative response previously | 22:23 |
| pabelanger | dig -t a git.openstack.org @2001:4860:4860::8888 fails | 22:25 |
| pabelanger | but dig -t a git.openstack.org @2620:0:ccc::2 works | 22:25 |
| mnaser | ok | 22:25 |
| mnaser | let me try something | 22:25 |
| pabelanger | so we cannot hit google DNS | 22:25 |
| mnaser | ill bring up that second ipv6 bgp peer now | 22:25 |
| *** jkilpatr has quit IRC | 22:29 | |
| mnaser | pabelanger fixed now | 22:30 |
| pabelanger | clarkb: fungi: do you mind a review / +3 on 492256. Want to setup our reverse proxy cache on buildlogs.cdn | 22:30 |
| pabelanger | hey, node online now | 22:30 |
| pabelanger | and dig works | 22:31 |
| fungi | yay! | 22:31 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Expose final job attribute https://review.openstack.org/479382 | 22:32 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Prohibit inheritance with final https://review.openstack.org/492314 | 22:32 |
| pabelanger | nodepool is using ipv4 | 22:32 |
| pabelanger | but that is okay for now | 22:32 |
| pabelanger | nc 162.253.55.90 19885 | 22:32 |
| fungi | not sure if that has to do with which address is returned first in the nova list entry, or if we aren't getting the v6 addys in nova list | 22:33 |
| pabelanger | ya, need to double check what we prefer for vexxhost | 22:34 |
| pabelanger | mnaser: so, what is the issue with config-drive? | 22:35 |
| pabelanger | ah, label issue | 22:35 |
| mnaser | pabelanger we use vfat config drive (config_drive_format=vfat) and the label of it is CONFIG-2 and not config-2 | 22:35 |
| pabelanger | guess we need to update glean to account for that | 22:35 |
| mnaser | i think you could get away with something like blkid <foo> || blkid <bar> | 22:35 |
| *** bobh has quit IRC | 22:37 | |
| clarkb | fungi: pabelanger ipv6-preffered: true on the provider iirc | 22:37 |
| clarkb | *preferred | 22:37 |
| pabelanger | ya, that sounds right | 22:38 |
| *** _ryan_ has joined #openstack-infra | 22:38 | |
| fungi | clarkb: okay, makes sense. thanks | 22:39 |
| fungi | mnaser: or since we already have a vfat conditional in there we can just .upper() the blkid or something | 22:39 |
| mnaser | fungi but the lookup uses the blkid tool and i looked quikcly and didnt see any case conditionals | 22:40 |
| fungi | mnaser: i meant we already wrap the blkid call in a conditional | 22:41 |
| fungi | i guess it's in shell so we could re-set or callout echo with a pipe through tr a-z A-Z | 22:42 |
| *** jkilpatr has joined #openstack-infra | 22:42 | |
| fungi | oh, nevermind, the mount call is in a conditional | 22:42 |
| fungi | and we're using blkid to get it, so yeah i guess we need a logic operator there | 22:43 |
| *** dave-mccowan has quit IRC | 22:44 | |
| mnaser | i'd appreciate eyes on this - https://review.openstack.org/#/c/492304/ - before we can bump it up more -- cause it causes a ton of churn on our infra | 22:46 |
| mnaser | pretty much a ~13GB write on the cluster on every new VM | 22:46 |
| pabelanger | mnaser: might be worth adding that info in the commit message so we know why | 22:47 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/project-config master: Switch vexxhost to RAW image format https://review.openstack.org/492304 | 22:49 |
| mnaser | pabelanger ^ | 22:50 |
| *** kornicameister has quit IRC | 22:50 | |
| pabelanger | +2 | 22:50 |
| pabelanger | clarkb: ^any issues with going with raw for vexxhost | 22:50 |
| fungi | the only real downside is that it will take longer to upload images | 22:51 |
| *** gcb has joined #openstack-infra | 22:51 | |
| fungi | afaik | 22:51 |
| *** rbrndt has quit IRC | 22:51 | |
| clarkb | ya | 22:51 |
| clarkb | at rest size uis comparable to qcow2 due to zeroing | 22:52 |
| pabelanger | and more HDD on builders | 22:52 |
| *** krtaylor has quit IRC | 22:52 | |
| mnaser | it wont be that bad if you use the builder hosted with us :D | 22:52 |
| mnaser | haha | 22:52 |
| fungi | i think we create both on builders regardless, right? | 22:52 |
| fungi | we just choose which to upload depending on configuration | 22:53 |
| clarkb | either can build them | 22:53 |
| *** sdague has quit IRC | 22:53 | |
| pabelanger | ya, we have qcow2 and vhds atm | 22:53 |
| *** jamesmcarthur has joined #openstack-infra | 22:53 | |
| pabelanger | nb03 only have 500Gb, so something to keep an eye on | 22:53 |
| fungi | i mean we create both images types on the builders regardless, and choose which image type to upload to where depending on configuration | 22:54 |
| pabelanger | but images are down in size overall | 22:54 |
| fungi | or is the builder actually smart enough to only create the image types it knows it will need? | 22:54 |
| pabelanger | right, we don't have .raw ATM | 22:54 |
| pabelanger | so, if we land 492304, we'll need to wait until new builds for new images | 22:55 |
| *** kornicameister has joined #openstack-infra | 22:55 | |
| *** jamesmcarthur has quit IRC | 22:57 | |
| fungi | sure, i don't think we're in a huge hurry on our end | 23:00 |
| fungi | zuul is just about caught up from today's peak surge | 23:01 |
| pabelanger | ya, tripleo change pipeline is hurting a little, but working on helping them fix it | 23:01 |
| fungi | i'm willing to bet that by ~01:00 utc the only remaining backlog of note will be the gate pipeline because it can't be burned down as quickly due to resets serializing it | 23:01 |
| fungi | and, yeah, that seems to be much more the tripleo gate queue than the integrated gate queue | 23:02 |
| pabelanger | 491878 should help with the resets, but failed in gate | 23:03 |
| pabelanger | plan on testing buildlogs.cdn.centos.org this evening to make sure it is caching properly | 23:03 |
| pabelanger | then, they'll be able to make the switch | 23:03 |
| fungi | looks like we only have ~600 queued jobs right now, the rest have nodes (if the queue graph is to be believed) | 23:04 |
| *** armax has quit IRC | 23:05 | |
| pabelanger | and about 700 waiting, down from 1050 1 hour ago | 23:05 |
| fungi | yeah, inbound activity has trailed off so it's catching up quickly | 23:06 |
| *** pbourke has quit IRC | 23:09 | |
| *** pbourke has joined #openstack-infra | 23:11 | |
| mnaser | does graphite.openstack.org still include timing data for jobs? | 23:19 |
| mgagne | mnaser: looks like yes because I have graph for a tempest-dsvm-neutron-full job for our provider: http://grafana.openstack.org/dashboard/db/nodepool-internap | 23:20 |
| *** gcb has quit IRC | 23:21 | |
| mgagne | in stats.timers.nodepool.job I'm not sure if it's what you are looking for | 23:21 |
| *** gcb has joined #openstack-infra | 23:21 | |
| mnaser | mgagne found it, the nodepool graph in the revert had ubuntu-trusty in there | 23:22 |
| mnaser | thanks | 23:22 |
| fungi | oh, yep, that probably straddled the trusty->xenial migration | 23:30 |
| fungi | i didn't even catch that in review | 23:30 |
| *** claudiub has quit IRC | 23:32 | |
| fungi | hour-old jobs in check are getting nodes assigned now, so not much longer | 23:32 |
| *** hongbin has quit IRC | 23:37 | |
| *** krtaylor has joined #openstack-infra | 23:40 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add firehose schema docs for logstash workers https://review.openstack.org/491615 | 23:41 |
| *** jaypipes has quit IRC | 23:44 | |
| fungi | and now nearly everything in check has node assignments so we should start burning through that post pipeline backlog any minute | 23:45 |
| *** marst_ has quit IRC | 23:47 | |
| *** dougwig has quit IRC | 23:48 | |
| *** markvoelker has quit IRC | 23:49 | |
| fungi | and just as we were almost caught up on check, a 10-change-deep stack for nova got pushed | 23:49 |
| openstackgerrit | Merged openstack-infra/project-config master: Switch vexxhost to RAW image format https://review.openstack.org/492304 | 23:51 |
| fungi | mnaser: so that's in now ^ | 23:51 |
| *** gcb has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!