| *** tphummel has quit IRC | 00:00 | |
| *** s-shiono has joined #openstack-infra | 00:00 | |
| *** ssbarnea has quit IRC | 00:00 | |
| *** skelso has joined #openstack-infra | 00:00 | |
| *** baoli has joined #openstack-infra | 00:01 | |
| *** spzala has quit IRC | 00:02 | |
| *** dingyichen has joined #openstack-infra | 00:02 | |
| clarkb | mriedem: I'm looking at ^ and wondering if anything in nova config drive generation changed in the last week or so in nova | 00:03 |
|---|---|---|
| clarkb | mriedem: we are tripping over a code path that we didn't run before which implies something is different in he config drive data | 00:03 |
| clarkb | mriedem: and worried that maybe its backward incompatible? | 00:03 |
| *** flepied has quit IRC | 00:05 | |
| mriedem | clarkb: you're using config drive from trunk nova? | 00:07 |
| mriedem | s/config drive/code/ | 00:07 |
| clarkb | yes, found https://review.openstack.org/#/c/430910/ and think that may be it | 00:08 |
| mriedem | hmm, hadn't seen that | 00:08 |
| clarkb | I'm gonna try booting a cirros host and dump the config drive from it | 00:08 |
| clarkb | fungi: https://arstechnica.com/security/2017/03/google-takes-symantec-to-the-woodshed-for-mis-issuing-30000-https-certs/ that won't affect us right? | 00:10 |
| bkero | www.openstack.org's certificate is issued by the much more trustworthy COMODO | 00:11 |
| clarkb | ya I think all of ours are but I haven't checked recently and I think fungi di some reissues | 00:11 |
| *** jianghuaw has quit IRC | 00:13 | |
| *** nadya has joined #openstack-infra | 00:15 | |
| *** adrian_otto has quit IRC | 00:16 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Typo fix: curent => current https://review.openstack.org/448966 | 00:16 |
| *** Goneri has quit IRC | 00:17 | |
| *** baoli has quit IRC | 00:17 | |
| *** baoli has joined #openstack-infra | 00:17 | |
| *** sflanigan has joined #openstack-infra | 00:18 | |
| *** Goneri has joined #openstack-infra | 00:19 | |
| *** ZZelle has quit IRC | 00:19 | |
| *** nadya has quit IRC | 00:19 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/nodepool feature/zuulv3: Merge branch 'master' into feature/zuulv3 https://review.openstack.org/445325 | 00:22 |
| clarkb | mriedem: ya I think its that change, and we have to add support for new things (ipv6_dhcp) to glean so I think nova is ok | 00:22 |
| clarkb | kevinbenton: ^ this brings up a new question which is, is neutron's default ipv6 behavior to use dhcp and not RAs? | 00:22 |
| clarkb | kevinbenton: if so what is the motivation for that? that seems backwards | 00:23 |
| * clarkb guesses ebacuse dns and friends | 00:23 | |
| *** flepied has joined #openstack-infra | 00:25 | |
| clarkb | mriedem: kevinbenton actually the ipv6 subnet claims its ipv6_address_mode and ipv6_ra_mode are both slaac | 00:25 |
| clarkb | so this seems like maybe a bug in neutron/nova somewhere? | 00:25 |
| clarkb | mordred: ^ fyi | 00:25 |
| clarkb | I might just send email to the dev list about this because its getting later in the day | 00:26 |
| mordred | clarkb: ++ | 00:26 |
| *** dave-mccowan has quit IRC | 00:29 | |
| *** pvaneck has quit IRC | 00:29 | |
| *** armax has joined #openstack-infra | 00:30 | |
| *** esikachev has joined #openstack-infra | 00:32 | |
| *** Julien-zte has joined #openstack-infra | 00:32 | |
| clarkb | oh it has enable_dhcp set to true too | 00:34 |
| clarkb | this is so confusing | 00:34 |
| *** esikachev has quit IRC | 00:36 | |
| *** rlandy|afk is now known as rlandy | 00:36 | |
| *** armax has quit IRC | 00:38 | |
| clarkb | ok mail sent lets see what people say | 00:40 |
| mriedem | clarkb: sfinucan has been digging through all of this so he's probably the person i'd ask in the morning | 00:41 |
| mriedem | he's trying to untangle a bunch of old nova-network config cruft | 00:41 |
| mriedem | for the eventual removal | 00:42 |
| clarkb | mriedem: I think nova might be mostly doing the correct thing (possibly needs to also write the slaac network info too) but am more confused by what neutron is doing | 00:42 |
| clarkb | why would dhcp and slaac be enabled on the same network? | 00:42 |
| clarkb | anyways lets see what people say and I can try to get sfinucan's input too | 00:43 |
| *** esikachev has joined #openstack-infra | 00:43 | |
| *** tosky has quit IRC | 00:43 | |
| *** baoli has quit IRC | 00:44 | |
| *** baoli has joined #openstack-infra | 00:45 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/nodepool feature/zuulv3: Fix test_leaked_node_not_deleted for v3 https://review.openstack.org/449375 | 00:47 |
| *** dave-mccowan has joined #openstack-infra | 00:48 | |
| *** baoli has quit IRC | 00:49 | |
| *** esikachev has quit IRC | 00:53 | |
| *** dave-mcc_ has joined #openstack-infra | 00:55 | |
| *** dave-mccowan has quit IRC | 00:57 | |
| openstackgerrit | Merged openstack-infra/glean master: Always define link_type in debian interfaces https://review.openstack.org/449366 | 01:01 |
| *** armax has joined #openstack-infra | 01:02 | |
| pabelanger | clarkb: mordred ^ | 01:03 |
| pabelanger | sure we tag a release? | 01:03 |
| clarkb | well it still failed | 01:06 |
| clarkb | so maybe still broken | 01:06 |
| clarkb | possibly we ssh to the ipv6 and not v4? | 01:06 |
| clarkb | I had hopped on the host and v4 was reachable | 01:06 |
| openstackgerrit | Merged openstack/diskimage-builder master: functests: skip qcow2 generically but add specific test https://review.openstack.org/448837 | 01:07 |
| *** dangers_away has quit IRC | 01:08 | |
| *** spzala has joined #openstack-infra | 01:08 | |
| *** dave-mccowan has joined #openstack-infra | 01:10 | |
| *** dave-mcc_ has quit IRC | 01:11 | |
| *** baoli has joined #openstack-infra | 01:11 | |
| *** liujiong has joined #openstack-infra | 01:12 | |
| openstackgerrit | gecong proposed openstack/os-testr master: Remove support for py34 in os-testr https://review.openstack.org/449379 | 01:12 |
| *** baoli has quit IRC | 01:15 | |
| *** MasterOfBugs has joined #openstack-infra | 01:17 | |
| *** genq has joined #openstack-infra | 01:18 | |
| *** dangers_away has joined #openstack-infra | 01:18 | |
| *** spzala has quit IRC | 01:20 | |
| *** spzala has joined #openstack-infra | 01:20 | |
| *** spzala has quit IRC | 01:21 | |
| *** spzala has joined #openstack-infra | 01:21 | |
| *** armax has quit IRC | 01:22 | |
| *** Julien-zte has quit IRC | 01:25 | |
| *** zhurong has joined #openstack-infra | 01:26 | |
| openstackgerrit | Brian Moss proposed openstack-infra/project-config master: Set up openstack-manuals for ocata https://review.openstack.org/449385 | 01:27 |
| *** Julien-zte has joined #openstack-infra | 01:27 | |
| *** dangers_away has quit IRC | 01:27 | |
| *** hongbin_ has joined #openstack-infra | 01:28 | |
| pabelanger | clarkb: no, we just timed out on the job | 01:31 |
| *** kaisers has joined #openstack-infra | 01:31 | |
| pabelanger | clarkb: rechecked nodepool and it is green on src now | 01:31 |
| pabelanger | http://logs.openstack.org/47/447647/5/check/gate-dsvm-nodepool-src-nv/abff2ba/console.html | 01:31 |
| pabelanger | clarkb: timeout was because it was building 3 images, precise, trusty (both passed) | 01:31 |
| *** kaisers1 has quit IRC | 01:33 | |
| *** jamielennox is now known as jamielennox|away | 01:39 | |
| *** rbuzatu has joined #openstack-infra | 01:39 | |
| *** esikachev has joined #openstack-infra | 01:41 | |
| *** dangers_away has joined #openstack-infra | 01:41 | |
| *** yanyanhu has joined #openstack-infra | 01:41 | |
| *** SumitNaiksatam has left #openstack-infra | 01:41 | |
| *** rbuzatu has quit IRC | 01:43 | |
| *** zul has quit IRC | 01:43 | |
| *** zul has joined #openstack-infra | 01:44 | |
| *** esikachev has quit IRC | 01:45 | |
| *** Apoorva_ has quit IRC | 01:47 | |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Add support for requiring github pr head status https://review.openstack.org/449390 | 01:47 |
| *** jamielennox|away is now known as jamielennox | 01:48 | |
| *** gcb has joined #openstack-infra | 01:48 | |
| *** gongysh has joined #openstack-infra | 01:49 | |
| *** rlandy has quit IRC | 01:51 | |
| *** ihrachys has joined #openstack-infra | 01:52 | |
| *** esikachev has joined #openstack-infra | 01:55 | |
| *** spzala has quit IRC | 01:56 | |
| *** spzala has joined #openstack-infra | 01:56 | |
| *** knangia has quit IRC | 02:01 | |
| *** esikachev has quit IRC | 02:01 | |
| *** spzala has quit IRC | 02:02 | |
| *** sree has joined #openstack-infra | 02:13 | |
| *** nadya has joined #openstack-infra | 02:15 | |
| *** sree has quit IRC | 02:17 | |
| *** Sukhdev has quit IRC | 02:17 | |
| *** ihrachys has quit IRC | 02:18 | |
| *** spzala has joined #openstack-infra | 02:19 | |
| *** kukacz has quit IRC | 02:20 | |
| *** nadya has quit IRC | 02:20 | |
| *** spzala has quit IRC | 02:23 | |
| *** esikachev has joined #openstack-infra | 02:25 | |
| *** esikachev has quit IRC | 02:29 | |
| *** larainema has quit IRC | 02:32 | |
| *** yanyanhu has quit IRC | 02:33 | |
| *** aeng has quit IRC | 02:35 | |
| *** esikachev has joined #openstack-infra | 02:36 | |
| *** sflanigan is now known as sflanigan_afk | 02:36 | |
| *** yanyanhu has joined #openstack-infra | 02:38 | |
| *** tbarron has quit IRC | 02:38 | |
| *** knikolla has quit IRC | 02:38 | |
| *** timrc has quit IRC | 02:38 | |
| *** aleph1 has quit IRC | 02:38 | |
| *** esikachev has quit IRC | 02:41 | |
| *** yamahata has quit IRC | 02:43 | |
| *** esikachev has joined #openstack-infra | 02:46 | |
| *** aleph1 has joined #openstack-infra | 02:47 | |
| *** raginbajin has quit IRC | 02:47 | |
| *** tbarron has joined #openstack-infra | 02:48 | |
| *** timrc has joined #openstack-infra | 02:49 | |
| *** knikolla has joined #openstack-infra | 02:50 | |
| *** esikachev has quit IRC | 02:50 | |
| *** sflanigan_afk is now known as sflanigan | 02:52 | |
| *** raginbajin has joined #openstack-infra | 02:52 | |
| *** esberglu has joined #openstack-infra | 02:53 | |
| *** tuanluong has joined #openstack-infra | 02:56 | |
| *** yuanying has quit IRC | 02:58 | |
| *** esberglu has quit IRC | 02:58 | |
| *** gcb has quit IRC | 02:58 | |
| *** gcb has joined #openstack-infra | 02:59 | |
| *** spzala has joined #openstack-infra | 03:00 | |
| *** IRCFrEAK has joined #openstack-infra | 03:01 | |
| *** IRCFrEAK has quit IRC | 03:04 | |
| *** spzala has quit IRC | 03:05 | |
| *** dave-mccowan has quit IRC | 03:08 | |
| *** ramishra has joined #openstack-infra | 03:12 | |
| *** mrodden has quit IRC | 03:17 | |
| *** gtmanfred has quit IRC | 03:18 | |
| *** IRCFrEAK has joined #openstack-infra | 03:18 | |
| *** gtmanfred has joined #openstack-infra | 03:19 | |
| *** mrodden has joined #openstack-infra | 03:19 | |
| *** gongysh has quit IRC | 03:20 | |
| *** IRCFrEAK has left #openstack-infra | 03:22 | |
| *** hongbin_ has quit IRC | 03:24 | |
| *** rbuzatu has joined #openstack-infra | 03:28 | |
| *** erikmwilson has quit IRC | 03:32 | |
| *** erikmwilson has joined #openstack-infra | 03:32 | |
| *** rbuzatu has quit IRC | 03:32 | |
| *** Julien-zte has quit IRC | 03:33 | |
| *** IRCFrEAK has joined #openstack-infra | 03:41 | |
| *** spzala has joined #openstack-infra | 03:42 | |
| *** IRCFrEAK has quit IRC | 03:42 | |
| *** skelso has quit IRC | 03:44 | |
| *** spzala has quit IRC | 03:47 | |
| *** yamahata has joined #openstack-infra | 03:47 | |
| *** knangia has joined #openstack-infra | 03:52 | |
| *** fandi has joined #openstack-infra | 03:53 | |
| *** IRCFrEAK has joined #openstack-infra | 03:58 | |
| *** IRCFrEAK has left #openstack-infra | 04:00 | |
| *** zhurong has quit IRC | 04:07 | |
| *** spzala has joined #openstack-infra | 04:09 | |
| *** sree has joined #openstack-infra | 04:12 | |
| *** spzala has quit IRC | 04:13 | |
| *** EricGonczer_ has quit IRC | 04:15 | |
| *** nadya has joined #openstack-infra | 04:16 | |
| *** sree_ has joined #openstack-infra | 04:17 | |
| *** sree_ is now known as Guest91910 | 04:18 | |
| *** sree has quit IRC | 04:19 | |
| *** links has joined #openstack-infra | 04:20 | |
| *** yuanying has joined #openstack-infra | 04:20 | |
| *** nadya has quit IRC | 04:21 | |
| *** EricGonczer_ has joined #openstack-infra | 04:25 | |
| *** IRCFrEAK has joined #openstack-infra | 04:26 | |
| *** IRCFrEAK has left #openstack-infra | 04:30 | |
| *** amotoki has joined #openstack-infra | 04:31 | |
| *** marst has joined #openstack-infra | 04:31 | |
| *** ihrachys has joined #openstack-infra | 04:32 | |
| *** amotoki has quit IRC | 04:35 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Use correct Ubuntu distro url on non-x86 arches https://review.openstack.org/448848 | 04:43 |
| *** amotoki has joined #openstack-infra | 04:44 | |
| *** dhajare has joined #openstack-infra | 04:47 | |
| *** Julien-zte has joined #openstack-infra | 04:52 | |
| *** nadya has joined #openstack-infra | 04:53 | |
| *** beslemon has joined #openstack-infra | 04:53 | |
| *** beslemon has quit IRC | 04:58 | |
| *** amotoki has quit IRC | 04:58 | |
| *** beslemon has joined #openstack-infra | 04:58 | |
| *** beslemon has quit IRC | 04:59 | |
| *** beslemon has joined #openstack-infra | 04:59 | |
| *** esikachev has joined #openstack-infra | 05:01 | |
| *** amotoki has joined #openstack-infra | 05:01 | |
| *** spzala has joined #openstack-infra | 05:01 | |
| *** Sukhdev has joined #openstack-infra | 05:04 | |
| *** beslemon has quit IRC | 05:05 | |
| *** beslemon has joined #openstack-infra | 05:05 | |
| *** EricGonczer_ has quit IRC | 05:06 | |
| *** esikachev has quit IRC | 05:06 | |
| *** spzala has quit IRC | 05:06 | |
| *** fandi has quit IRC | 05:07 | |
| *** psachin has joined #openstack-infra | 05:07 | |
| *** amotoki has quit IRC | 05:09 | |
| *** armax has joined #openstack-infra | 05:11 | |
| *** vikrant has joined #openstack-infra | 05:11 | |
| *** kornica has joined #openstack-infra | 05:12 | |
| *** Kevin_Zheng has quit IRC | 05:13 | |
| *** nadya has quit IRC | 05:15 | |
| *** nadya has joined #openstack-infra | 05:15 | |
| *** zhurong has joined #openstack-infra | 05:16 | |
| *** IRCFrEAK has joined #openstack-infra | 05:18 | |
| *** pgadiya has joined #openstack-infra | 05:18 | |
| *** nadya has quit IRC | 05:19 | |
| *** spzala has joined #openstack-infra | 05:23 | |
| *** esikachev has joined #openstack-infra | 05:26 | |
| *** spzala has quit IRC | 05:27 | |
| *** Sukhdev has quit IRC | 05:29 | |
| *** gongysh has joined #openstack-infra | 05:29 | |
| *** prateek has joined #openstack-infra | 05:31 | |
| *** kornica has quit IRC | 05:34 | |
| *** anilvenkata has joined #openstack-infra | 05:41 | |
| *** kornica has joined #openstack-infra | 05:51 | |
| *** amotoki has joined #openstack-infra | 05:58 | |
| openstackgerrit | Merged openstack-infra/project-config master: Make the scenario multinode job multinode https://review.openstack.org/449300 | 05:58 |
| *** spzala has joined #openstack-infra | 06:00 | |
| *** knangia has quit IRC | 06:01 | |
| *** xarses_ has joined #openstack-infra | 06:02 | |
| *** Julien-zte has quit IRC | 06:05 | |
| *** beslemon has quit IRC | 06:05 | |
| *** spzala has quit IRC | 06:05 | |
| *** kornica has quit IRC | 06:05 | |
| *** yee37914 has quit IRC | 06:06 | |
| *** yee379 has joined #openstack-infra | 06:08 | |
| *** andreww has joined #openstack-infra | 06:10 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Create PReP boot partition for PPC https://review.openstack.org/447739 | 06:10 |
| *** yee379 has quit IRC | 06:12 | |
| *** yee379 has joined #openstack-infra | 06:12 | |
| *** xarses_ has quit IRC | 06:14 | |
| *** yee379 has quit IRC | 06:14 | |
| *** andreww has quit IRC | 06:14 | |
| *** jiaohaolin has joined #openstack-infra | 06:15 | |
| *** yee379 has joined #openstack-infra | 06:15 | |
| jiaohaolin | hey guys, I am setting up my CI ,and I found that the slave need to be ubuntu 14.04,but the devstack need to run in 16.04 rather than 14.04 . How to solve this problem ? | 06:18 |
| *** bhavik1 has joined #openstack-infra | 06:20 | |
| ianw | jiaohaolin: the slave doesn't have to be trusty. although there may be documentation that needs to be updated | 06:24 |
| *** gcb has quit IRC | 06:28 | |
| *** armax has quit IRC | 06:29 | |
| *** MasterOfBugs has quit IRC | 06:32 | |
| jiaohaolin | ianw: Can you give me an guide of Slave ? | 06:40 |
| *** camunoz has quit IRC | 06:40 | |
| *** spzala has joined #openstack-infra | 06:42 | |
| jiaohaolin | ianw: please? | 06:42 |
| *** slaweq__ is now known as slaweq_ | 06:43 | |
| *** yee379 has quit IRC | 06:44 | |
| *** gouthamr has joined #openstack-infra | 06:44 | |
| ianw | jiaohaolin: i'm not sure what you mean sorry | 06:44 |
| *** IRCFrEAK has quit IRC | 06:44 | |
| *** yee379 has joined #openstack-infra | 06:44 | |
| *** ihrachys has quit IRC | 06:44 | |
| *** gcb has joined #openstack-infra | 06:45 | |
| *** spzala has quit IRC | 06:47 | |
| jiaohaolin | ianw: I followed this article to set up the master ,but the slave can't not successfully runing | 06:48 |
| *** alexchadin has joined #openstack-infra | 06:48 | |
| jiaohaolin | ianw: http://www.joinfu.com/2014/02/setting-up-an-external-openstack-testing-system/ | 06:48 |
| jiaohaolin | ianw: Do you have any article about slave ? | 06:48 |
| jiaohaolin | ianw: it is very nice if you can share it | 06:50 |
| *** gongysh has quit IRC | 06:51 | |
| *** markus_z has joined #openstack-infra | 06:56 | |
| *** Guest91910 has quit IRC | 06:56 | |
| *** andreas_s has joined #openstack-infra | 06:58 | |
| *** Julien-zte has joined #openstack-infra | 07:00 | |
| *** jascott1_ has quit IRC | 07:01 | |
| *** jascott1_ has joined #openstack-infra | 07:01 | |
| *** gongysh has joined #openstack-infra | 07:02 | |
| openstackgerrit | Steven Dake proposed openstack-infra/system-config master: Add #openstack-deployment channel to logging https://review.openstack.org/438857 | 07:06 |
| *** jascott1_ has quit IRC | 07:06 | |
| *** spzala has joined #openstack-infra | 07:10 | |
| *** gouthamr has quit IRC | 07:12 | |
| *** spzala has quit IRC | 07:15 | |
| *** psachin has quit IRC | 07:18 | |
| *** psachin has joined #openstack-infra | 07:18 | |
| *** ralonsoh has joined #openstack-infra | 07:23 | |
| *** alexchadin has quit IRC | 07:25 | |
| *** jascott1_ has joined #openstack-infra | 07:25 | |
| *** tesseract has joined #openstack-infra | 07:31 | |
| *** bhavik1 has quit IRC | 07:38 | |
| *** jascott1_ has quit IRC | 07:44 | |
| *** liujiong has quit IRC | 07:45 | |
| *** jascott1_ has joined #openstack-infra | 07:45 | |
| *** kornica has joined #openstack-infra | 07:45 | |
| *** eroux has quit IRC | 07:47 | |
| *** jascott1_ has quit IRC | 07:50 | |
| *** spzala has joined #openstack-infra | 07:52 | |
| *** eroux has joined #openstack-infra | 07:53 | |
| *** matrohon has joined #openstack-infra | 07:54 | |
| *** alexchadin has joined #openstack-infra | 07:54 | |
| *** askb has quit IRC | 07:55 | |
| *** jaosorior has joined #openstack-infra | 07:55 | |
| *** spzala has quit IRC | 07:57 | |
| *** ccamacho has joined #openstack-infra | 07:58 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-infra | 08:01 | |
| *** hrubi has quit IRC | 08:01 | |
| *** Julien-zte has quit IRC | 08:04 | |
| *** nadya has joined #openstack-infra | 08:05 | |
| *** nadya has quit IRC | 08:06 | |
| *** nadya has joined #openstack-infra | 08:06 | |
| *** yamahata has quit IRC | 08:07 | |
| *** ralonsoh has quit IRC | 08:07 | |
| *** shardy_afk is now known as shardy | 08:09 | |
| *** florianf has joined #openstack-infra | 08:09 | |
| *** mriedem has quit IRC | 08:16 | |
| *** vikrant has quit IRC | 08:16 | |
| *** Julien-zte has joined #openstack-infra | 08:18 | |
| *** vikrant has joined #openstack-infra | 08:18 | |
| *** esikachev has quit IRC | 08:20 | |
| *** gongysh has quit IRC | 08:22 | |
| *** Kevin_Zheng has joined #openstack-infra | 08:22 | |
| AJaeger | yolanda: good morning! could you review https://review.openstack.org/446698 and https://review.openstack.org/#/c/446744/ as well, please? | 08:23 |
| *** amoralej|off is now known as amoralej | 08:23 | |
| *** esikachev has joined #openstack-infra | 08:24 | |
| yolanda | sure | 08:25 |
| *** sflanigan has quit IRC | 08:25 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add periodic-newton/ocata jobs to openstack/congress https://review.openstack.org/449321 | 08:26 |
| openstackgerrit | Merged openstack-infra/project-config master: Add Test Types to Patrole Jobs https://review.openstack.org/449289 | 08:27 |
| *** pahuang has quit IRC | 08:28 | |
| *** sree has joined #openstack-infra | 08:28 | |
| yolanda | oh, new neutron infra liaisons... | 08:30 |
| *** rbuzatu has joined #openstack-infra | 08:30 | |
| AJaeger | yolanda: yeah - new names to learn ;( | 08:30 |
| yolanda | oh, i'll need some time to get used to it | 08:31 |
| AJaeger | so do I... | 08:31 |
| yolanda | also glance changed | 08:31 |
| AJaeger | oh? Let me check... | 08:31 |
| AJaeger | thanks | 08:31 |
| *** sree has quit IRC | 08:32 | |
| * yolanda is thinking about printing the liaisons list | 08:32 | |
| *** spzala has joined #openstack-infra | 08:33 | |
| *** bauzas is now known as bauwser | 08:34 | |
| openstackgerrit | Merged openstack-infra/project-config master: add releasenotes jobs for shade https://review.openstack.org/446673 | 08:35 |
| openstackgerrit | Merged openstack-infra/project-config master: networking-ovn: add functional tests with python 3.5 in CI jobs https://review.openstack.org/435916 | 08:35 |
| openstackgerrit | Merged openstack-infra/project-config master: Rename #openstack-university to #openstack-upstream-institute https://review.openstack.org/446744 | 08:36 |
| openstackgerrit | Merged openstack-infra/project-config master: Fix VNC setup in neutron DVR multinode jobs https://review.openstack.org/449132 | 08:37 |
| *** matrohon has quit IRC | 08:37 | |
| openstackgerrit | Merged openstack-infra/project-config master: Make the UEFI job in Ironic voting https://review.openstack.org/449222 | 08:37 |
| *** gongysh has joined #openstack-infra | 08:38 | |
| *** spzala has quit IRC | 08:38 | |
| openstackgerrit | Merged openstack-infra/project-config master: networking-ovn: Make OVS master job non-voting. https://review.openstack.org/448776 | 08:38 |
| *** sree has joined #openstack-infra | 08:40 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add a galera_ssl scenario test for OpenStack-Ansible https://review.openstack.org/448721 | 08:40 |
| *** matrohon has joined #openstack-infra | 08:41 | |
| *** aarefiev_pto is now known as aarefiev | 08:42 | |
| *** gongysh has quit IRC | 08:43 | |
| openstackgerrit | Merged openstack-infra/project-config master: tripleo: add deploy-guide-jobs into tripleo-docs https://review.openstack.org/448740 | 08:45 |
| openstackgerrit | Merged openstack-infra/project-config master: Use local_conf for almanach and aodh https://review.openstack.org/446698 | 08:46 |
| *** bogdando has joined #openstack-infra | 08:52 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: Apply setfiles on all mountpoints https://review.openstack.org/447076 | 08:52 |
| *** ralonsoh has joined #openstack-infra | 08:52 | |
| bogdando | hello. Folks, are there tarballs live some place for faild CI jobs logs? | 08:52 |
| bogdando | given example http://logs.openstack.org/43/448543/3/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/9ece507/logs/ how could I fetch all for a local inspection but archived?.. | 08:53 |
| bogdando | or could you please share some kibana magic examples to restrict search *only* to a given job? | 08:53 |
| bogdando | and are those match 1:1 (data stored in ES/Kibana and in logs.openstack)? | 08:54 |
| *** markmcd has quit IRC | 08:57 | |
| *** alex_xu has quit IRC | 08:58 | |
| *** martinkopec has joined #openstack-infra | 08:58 | |
| *** alex_xu has joined #openstack-infra | 08:59 | |
| *** gildub has quit IRC | 08:59 | |
| *** alex_xu has quit IRC | 09:01 | |
| *** alex_xu has joined #openstack-infra | 09:02 | |
| *** ralonsoh_ has joined #openstack-infra | 09:05 | |
| *** alex_xu has quit IRC | 09:06 | |
| *** alex_xu has joined #openstack-infra | 09:07 | |
| *** claudiub|2 has joined #openstack-infra | 09:08 | |
| *** s-shiono has quit IRC | 09:09 | |
| *** ralonsoh has quit IRC | 09:09 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: WIP: Add lvm management to diskimage-builder https://review.openstack.org/444403 | 09:11 |
| *** yuanying has quit IRC | 09:12 | |
| *** ssbarnea has joined #openstack-infra | 09:13 | |
| *** spzala has joined #openstack-infra | 09:15 | |
| *** spzala has quit IRC | 09:19 | |
| *** gongysh has joined #openstack-infra | 09:20 | |
| *** tuanluong has quit IRC | 09:23 | |
| *** ggnel_t has joined #openstack-infra | 09:26 | |
| *** ralonsoh_ is now known as ralonsoh | 09:27 | |
| *** lucas-afk is now known as lucasagomes | 09:29 | |
| *** flepied has quit IRC | 09:29 | |
| *** esberglu has joined #openstack-infra | 09:30 | |
| *** derekh has joined #openstack-infra | 09:33 | |
| *** jrist has quit IRC | 09:34 | |
| *** esberglu has quit IRC | 09:35 | |
| *** Julien-zte has quit IRC | 09:37 | |
| *** jiaohaolin has quit IRC | 09:38 | |
| *** jiaohaolin has joined #openstack-infra | 09:38 | |
| *** ssbarnea has quit IRC | 09:40 | |
| *** electrofelix has joined #openstack-infra | 09:41 | |
| *** spzala has joined #openstack-infra | 09:48 | |
| *** yamamoto has quit IRC | 09:50 | |
| *** jiaohaolin has quit IRC | 09:50 | |
| *** jiaohaolin has joined #openstack-infra | 09:52 | |
| *** mat128 has joined #openstack-infra | 09:53 | |
| *** e0ne has joined #openstack-infra | 09:53 | |
| *** spzala has quit IRC | 09:53 | |
| *** yuanying has joined #openstack-infra | 09:54 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: WIP: Add lvm management to diskimage-builder https://review.openstack.org/444403 | 09:54 |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: Use stevedore for module config of block device https://review.openstack.org/447090 | 09:54 |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: Refactor: block-device filesystem creation, mount and fstab https://review.openstack.org/444586 | 09:54 |
| *** flepied has joined #openstack-infra | 09:59 | |
| *** nadya has quit IRC | 10:01 | |
| *** nadya has joined #openstack-infra | 10:01 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: Refactor: block-device filesystem creation, mount and fstab https://review.openstack.org/444586 | 10:02 |
| *** nadya has quit IRC | 10:05 | |
| *** pcaruana has joined #openstack-infra | 10:05 | |
| *** ociuhandu has joined #openstack-infra | 10:08 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Set up openstack-manuals for Ocata https://review.openstack.org/449385 | 10:14 |
| *** tellesnobrega has quit IRC | 10:14 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: WIP: Add lvm management to diskimage-builder https://review.openstack.org/444403 | 10:15 |
| *** sree has quit IRC | 10:19 | |
| openstackgerrit | Alfredo Moralejo proposed openstack-infra/tripleo-ci master: [DNM] Adding yum debug in oooq playbook https://review.openstack.org/449548 | 10:20 |
| *** sree has joined #openstack-infra | 10:20 | |
| *** sree has quit IRC | 10:21 | |
| *** sree has joined #openstack-infra | 10:22 | |
| *** sree has quit IRC | 10:26 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack-infra/tripleo-ci master: Adapt getthelogs UX for more use cases https://review.openstack.org/449552 | 10:27 |
| *** spzala has joined #openstack-infra | 10:29 | |
| *** matrohon has quit IRC | 10:30 | |
| *** ZZelle_ is now known as ZZelle | 10:31 | |
| *** sfinucan has joined #openstack-infra | 10:34 | |
| *** spzala has quit IRC | 10:34 | |
| *** alexchadin has quit IRC | 10:35 | |
| *** alexchadin has joined #openstack-infra | 10:35 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE: Testing novajoin authtoken https://review.openstack.org/446348 | 10:36 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE: Testing ensure dir for httpd certs https://review.openstack.org/446348 | 10:36 |
| *** gongysh has quit IRC | 10:37 | |
| *** nicolasbock has joined #openstack-infra | 10:38 | |
| *** tosky has joined #openstack-infra | 10:38 | |
| openstackgerrit | Andrey Volkov proposed openstack-infra/project-config master: Add placement api ref jenkins job https://review.openstack.org/449557 | 10:39 |
| *** alexchadin has quit IRC | 10:40 | |
| *** zhurong has quit IRC | 10:40 | |
| *** matrohon has joined #openstack-infra | 10:41 | |
| *** yanyanhu has quit IRC | 10:42 | |
| *** erikmwilson has quit IRC | 10:42 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE: Testing change dir for httpd certs/keys https://review.openstack.org/449560 | 10:42 |
| *** nadya has joined #openstack-infra | 10:44 | |
| *** yamamoto has joined #openstack-infra | 10:49 | |
| *** yamamoto_ has joined #openstack-infra | 10:50 | |
| sfinucan | dhellmann: Could ask you to take a gawk at https://review.openstack.org/#/c/448066/ at some point? I'd like to use reno in a couple of non-OpenStack projects and that adds a nice improvement for those projects. | 10:51 |
| * sfinucan isn't sure where reno discussion should take place | 10:51 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: Apply setfiles on all mountpoints https://review.openstack.org/447076 | 10:52 |
| *** yamamoto has quit IRC | 10:54 | |
| *** gcb has quit IRC | 10:56 | |
| AJaeger | sfinucan: #openstack-release - and dhellmann is on PTO this week, so won't answer | 10:57 |
| sfinucan | AJaeger: Ah :) Cheers! | 10:57 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE: Testing TLS-everywhere with keystone container https://review.openstack.org/449570 | 11:00 |
| *** armax has joined #openstack-infra | 11:02 | |
| *** yamamoto_ has quit IRC | 11:05 | |
| *** yamamoto has joined #openstack-infra | 11:05 | |
| *** Dinesh_Bhor has quit IRC | 11:07 | |
| *** e0ne has quit IRC | 11:10 | |
| *** e0ne has joined #openstack-infra | 11:10 | |
| *** armax has quit IRC | 11:16 | |
| *** yamamoto has quit IRC | 11:17 | |
| *** dtantsur|afk is now known as dtantsur | 11:18 | |
| *** Dinesh_Bhor has joined #openstack-infra | 11:20 | |
| *** sdague has joined #openstack-infra | 11:21 | |
| *** gongysh has joined #openstack-infra | 11:28 | |
| *** EricGonczer_ has joined #openstack-infra | 11:31 | |
| *** rfolco has joined #openstack-infra | 11:32 | |
| *** rfolco has quit IRC | 11:33 | |
| *** rfolco has joined #openstack-infra | 11:33 | |
| *** pgadiya has quit IRC | 11:38 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack-infra/tripleo-ci master: Adapt getthelogs UX for more use cases https://review.openstack.org/449552 | 11:43 |
| *** ldnunes has joined #openstack-infra | 11:44 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack-infra/tripleo-ci master: Adapt getthelogs UX for more use cases https://review.openstack.org/449552 | 11:45 |
| openstackgerrit | Sean Dague proposed openstack-infra/devstack-gate master: Capture any devstack unit logs https://review.openstack.org/449311 | 11:46 |
| *** alexchadin has joined #openstack-infra | 11:48 | |
| *** Hal has quit IRC | 11:49 | |
| openstackgerrit | Andrey Volkov proposed openstack-infra/project-config master: Add placement api ref jenkins job https://review.openstack.org/449557 | 11:50 |
| *** spzala has joined #openstack-infra | 11:50 | |
| *** rfolco has quit IRC | 11:51 | |
| *** spzala has quit IRC | 11:56 | |
| *** tellesnobrega has joined #openstack-infra | 11:58 | |
| *** hashar has joined #openstack-infra | 11:58 | |
| *** links has quit IRC | 12:02 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: WIP Fetch server console log if ssh connection fails https://review.openstack.org/358730 | 12:03 |
| *** EricGonczer_ has quit IRC | 12:05 | |
| *** links has joined #openstack-infra | 12:05 | |
| *** ssbarnea has joined #openstack-infra | 12:06 | |
| *** yamamoto has joined #openstack-infra | 12:06 | |
| mordred | clarkb: when you get up - let's see if we can't get that patch from TheJulia to glean in - it's been lurking for over a year and I feel bad | 12:08 |
| *** aysyd has joined #openstack-infra | 12:08 | |
| openstackgerrit | Merged openstack-infra/glean master: Remove support for py33/py26 https://review.openstack.org/424419 | 12:08 |
| *** sputnik13 has joined #openstack-infra | 12:10 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/glean master: Revise systemd determination to verify systemctl presence https://review.openstack.org/279818 | 12:12 |
| mordred | clarkb: ^^ fixed your elif concern | 12:12 |
| mordred | oh - you know what ... | 12:13 |
| mordred | clarkb, pabelanger: we should add a gate-nodepool-dsvm-src job to glean patches | 12:13 |
| *** lucasagomes is now known as lucas-hungry | 12:14 | |
| *** yamamoto has quit IRC | 12:15 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-log_processor master: Add MQTT support to the gearman worker https://review.openstack.org/449294 | 12:15 |
| openstackgerrit | Anton Studenov proposed openstack-infra/project-config master: [Rally] Make gate-rally-tox-self-* voting https://review.openstack.org/449282 | 12:15 |
| *** gongysh has quit IRC | 12:17 | |
| *** cshastri has joined #openstack-infra | 12:17 | |
| *** wolverineav has joined #openstack-infra | 12:20 | |
| *** katkapilatova has joined #openstack-infra | 12:20 | |
| *** katkapilatova has left #openstack-infra | 12:24 | |
| *** katkapilatova has joined #openstack-infra | 12:24 | |
| *** d0ugal has quit IRC | 12:24 | |
| *** dprince has joined #openstack-infra | 12:25 | |
| *** dprince has quit IRC | 12:26 | |
| *** dprince has joined #openstack-infra | 12:26 | |
| *** rfolco has joined #openstack-infra | 12:27 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-log_processor master: Add MQTT support to the gearman worker https://review.openstack.org/449294 | 12:27 |
| pabelanger | mordred: we kinda do already, gate-dsvm-nodepool-debian-src-nv and gate-dsvm-nodepool-redhat-src-nv will attempt to build each image we use in the gate. | 12:29 |
| *** shardy is now known as shardy_lunch | 12:31 | |
| *** rkukura has quit IRC | 12:32 | |
| *** rkukura_ has joined #openstack-infra | 12:32 | |
| *** spzala has joined #openstack-infra | 12:32 | |
| *** tellesnobrega has quit IRC | 12:34 | |
| *** jascott1_ has joined #openstack-infra | 12:36 | |
| *** flepied has quit IRC | 12:36 | |
| *** rkukura_ has quit IRC | 12:36 | |
| *** spzala has quit IRC | 12:37 | |
| *** jaypipes is now known as leakypipes | 12:37 | |
| *** trown|outtypewww is now known as trown | 12:39 | |
| *** mdrabe has joined #openstack-infra | 12:41 | |
| *** Julien-zte has joined #openstack-infra | 12:41 | |
| *** jascott1_ has quit IRC | 12:41 | |
| *** gordc has joined #openstack-infra | 12:44 | |
| *** tellesnobrega has joined #openstack-infra | 12:46 | |
| *** cshastri has quit IRC | 12:46 | |
| aspiers | hi all, I've been pointed here by AJaeger as the place to request that my two gerrit user accounts are merged into one | 12:51 |
| aspiers | I think I accidentally logged into gerrit once using the wrong launchpad OpenId account | 12:52 |
| *** rlandy has joined #openstack-infra | 12:52 | |
| aspiers | and that was enough for it to appear in gerrit account searches | 12:52 |
| *** flepied has joined #openstack-infra | 12:54 | |
| *** david-lyle has quit IRC | 12:55 | |
| *** esberglu has joined #openstack-infra | 12:55 | |
| *** ramishra has quit IRC | 12:56 | |
| *** xyang1 has joined #openstack-infra | 12:56 | |
| *** flepied has quit IRC | 13:03 | |
| *** anilvenkata has quit IRC | 13:04 | |
| *** genq has quit IRC | 13:04 | |
| *** mriedem has joined #openstack-infra | 13:06 | |
| *** spzala has joined #openstack-infra | 13:06 | |
| *** gouthamr has joined #openstack-infra | 13:08 | |
| *** sree has joined #openstack-infra | 13:08 | |
| *** EricGonczer_ has joined #openstack-infra | 13:09 | |
| *** kornica has quit IRC | 13:09 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack-infra/tripleo-ci master: Adapt getthelogs UX for more use cases https://review.openstack.org/449552 | 13:09 |
| *** ihrachys has joined #openstack-infra | 13:11 | |
| *** spzala has quit IRC | 13:12 | |
| *** ihrachys has quit IRC | 13:12 | |
| *** EricGonczer_ has quit IRC | 13:13 | |
| fungi | clarkb: yeah, i don't think we're directly impacted by the chrome/verisign situation | 13:15 |
| *** jcoufal has joined #openstack-infra | 13:15 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for AZ scheduling bug 1675607 https://review.openstack.org/449619 | 13:16 |
| openstack | bug 1675607 in OpenStack Compute (nova) "test_list_migrations_in_flavor_resize_situation fails with NoValidHost - AvailabilityZoneFilter returned 0 hosts" [Undecided,Confirmed] https://launchpad.net/bugs/1675607 | 13:16 |
| *** shardy_lunch is now known as shardy | 13:17 | |
| *** flepied has joined #openstack-infra | 13:17 | |
| *** erikmwilson has joined #openstack-infra | 13:18 | |
| fungi | clarkb: mriedem: as for ipv6 and dhcp... keep in mind there are two major dhcp6 modes. stateless dhcp6 is really just geared at handing out things like dns server addresses and still relies on slaac for hosts to pick their addresses and ra for them to figure out their prefixes and gateways. _stateful_ dhcp6 gets into direct address assignment and tracking (though even stateful dhcp6 still relies on | 13:19 |
| fungi | ra to establish routing and local gateway addresses) | 13:19 |
| *** yamamoto has joined #openstack-infra | 13:19 | |
| *** anilvenkata has joined #openstack-infra | 13:20 | |
| fungi | aspiers: i'll see if i can guess your account ids by peeking in the db | 13:20 |
| aspiers | fungi: ubuntu@adamspiers.org and aspiers@suse.com - thanks a lot! | 13:21 |
| fungi | aspiers: oh, good, because we also have an andrew spiers ;) | 13:21 |
| aspiers | fungi: yikes :) I want to keep aspiers@suse.com as the primary | 13:22 |
| *** jrist has joined #openstack-infra | 13:22 | |
| aspiers | actually I don't expect to use ubuntu@ for OpenStack at all, but I might occasionally accidentally log into launchpad using that | 13:22 |
| fungi | aspiers: yeah, looks like you have two accounts: 2394 from circa 2013 or earlier, and 24717 from around the end of last year | 13:23 |
| aspiers | 2394 must be @suse.com | 13:23 |
| fungi | yep | 13:23 |
| aspiers | 24717 is the accident | 13:23 |
| fungi | i'm cleaning it up, just a sec | 13:23 |
| aspiers | I don't think I used it much, if at all | 13:23 |
| aspiers | awesome thanks! | 13:23 |
| AJaeger | fungi, heads up: For the first time openstack-manuals used the releases repo to branch and it worked fine ;) | 13:25 |
| *** psachin has quit IRC | 13:25 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for AZ scheduling bug 1675607 https://review.openstack.org/449619 | 13:25 |
| openstack | bug 1675607 in OpenStack Compute (nova) "test_list_migrations_in_flavor_resize_situation fails with NoValidHost - AvailabilityZoneFilter returned 0 hosts" [Undecided,Confirmed] https://launchpad.net/bugs/1675607 | 13:25 |
| *** esikachev has quit IRC | 13:26 | |
| AJaeger | fungi, is the codesearch update still on your todo list? | 13:26 |
| *** EricGonczer_ has joined #openstack-infra | 13:26 | |
| fungi | AJaeger: oh, great news on the branch automation! | 13:27 |
| *** lucas-hungry is now known as lucasagomes | 13:27 | |
| fungi | AJaeger: i tried several ways to get those changes applied on codesearch.o.o and had no luck. tried telling ansible to directly run puppet on it, tried manually running the update commands, doesn't seem to have changed the repo config there | 13:28 |
| AJaeger | ;( | 13:28 |
| fungi | so there's something i'm missing, maybe | 13:28 |
| *** baoli has joined #openstack-infra | 13:28 | |
| AJaeger | so, what magic did pabelanger do last time? | 13:28 |
| fungi | i asked him, and then did exactly the same thing as far as i can tell | 13:28 |
| pabelanger | hmm | 13:28 |
| pabelanger | fungi: maybe I took it out of emergency for 30mins, to make sure puppet run properly | 13:29 |
| fungi | i tried updating it with the ad hoc playbook, then tried updating it with the kick.sh script (which seems to run exactly the same command), then tried again with the entry removed from the emergency disable file just in case that was preventing it | 13:29 |
| pabelanger | then added it back | 13:29 |
| fungi | i can try just taking it out of the emergency list for longer to give puppet a chance to update normally, but would be good to figure out what that does differently that kick.sh is missing | 13:30 |
| fungi | aspiers: okay, the newer accidental account is now disabled, so it shouldn't cause you any further trouble | 13:30 |
| *** gcb has joined #openstack-infra | 13:31 | |
| fungi | pabelanger: AJaeger: okay, i've enabled it, and will tail the syslog there to see when it updates | 13:31 |
| openstackgerrit | Sean Dague proposed openstack-infra/devstack-gate master: Capture any devstack unit logs https://review.openstack.org/449311 | 13:32 |
| AJaeger | thanks, fungi | 13:33 |
| *** lucasagomes is now known as lucas-brb | 13:35 | |
| *** tphummel has joined #openstack-infra | 13:36 | |
| *** prateek has quit IRC | 13:36 | |
| *** Dinesh_Bhor has quit IRC | 13:39 | |
| *** jamesdenton has joined #openstack-infra | 13:43 | |
| *** jkilpatr has quit IRC | 13:44 | |
| *** jcoufal_ has joined #openstack-infra | 13:44 | |
| jbernard | mriedem: for a given spike in the ceph failure rate, is there precise way of identifying all of the failing jobs withing a particular time window? | 13:44 |
| *** d0ugal has joined #openstack-infra | 13:45 | |
| *** andreas_s has quit IRC | 13:45 | |
| *** spzala has joined #openstack-infra | 13:45 | |
| *** jcoufal has quit IRC | 13:46 | |
| *** sree_ has joined #openstack-infra | 13:46 | |
| *** sree_ is now known as Guest85728 | 13:46 | |
| *** ihrachys has joined #openstack-infra | 13:47 | |
| *** sree has quit IRC | 13:48 | |
| mriedem | jbernard: logstash goes back 10 days, | 13:48 |
| mriedem | so if it started within 10 days and you have the right query for the failure, then that would tell you | 13:48 |
| jbernard | mriedem: logstash, how does it work? | 13:49 |
| *** dansmith is now known as superdan | 13:49 | |
| mriedem | http://logstash.openstack.org/#/dashboard/file/logstash.json | 13:49 |
| mriedem | jbernard: we track known CI failures here http://status.openstack.org/elastic-recheck/ | 13:49 |
| *** mtanino has joined #openstack-infra | 13:49 | |
| mriedem | e.g. http://status.openstack.org/elastic-recheck/#1648885 | 13:49 |
| jbernard | mriedem: oh this is cool | 13:50 |
| mriedem | click the logstash link for any of those to see the query over a 10 day run | 13:50 |
| *** spzala has quit IRC | 13:50 | |
| mriedem | https://docs.openstack.org/infra/elastic-recheck/readme.html has some more info on elastic-recheck | 13:50 |
| mriedem | http://lucene.apache.org/core/4_0_0/queryparser/org/apache/lucene/queryparser/classic/package-summary.html#package_description is the query syntax | 13:50 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config master: Enable firehose reporting on subunit workers https://review.openstack.org/449630 | 13:51 |
| *** links has quit IRC | 13:53 | |
| *** asselin has joined #openstack-infra | 13:53 | |
| *** rkukura has joined #openstack-infra | 13:53 | |
| *** dimtruck is now known as zz_dimtruck | 13:53 | |
| *** sshnaidm|off has quit IRC | 13:54 | |
| *** zz_dimtruck is now known as dimtruck | 13:54 | |
| *** rossella_s has joined #openstack-infra | 13:56 | |
| *** rbrndt has joined #openstack-infra | 13:59 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/project-config master: New Project - LOCI https://review.openstack.org/449224 | 13:59 |
| *** lucas-brb is now known as lucasagomes | 14:01 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config master: Enable firehose reporting on subunit workers https://review.openstack.org/449630 | 14:02 |
| *** anilvenkata has quit IRC | 14:02 | |
| *** erikmwilson has quit IRC | 14:03 | |
| *** dimtruck is now known as zz_dimtruck | 14:03 | |
| *** erikmwilson has joined #openstack-infra | 14:04 | |
| jbernard | ihrachys: for the grafana dashboard i added, is there a way to see it working before code review? | 14:04 |
| *** eharney has joined #openstack-infra | 14:04 | |
| *** dave-mccowan has joined #openstack-infra | 14:07 | |
| *** spzala has joined #openstack-infra | 14:08 | |
| sbezverk_ | Good morning infra team. I am looking for an expert (Ubunut/libvirtd) opinion on the issue I see when I use virsh commands in Ubuntu gate VM. (For CentOS the same sequence of commands works perfectly). Anybody you can suggest? | 14:09 |
| *** scottda has joined #openstack-infra | 14:09 | |
| *** alexchadin has quit IRC | 14:11 | |
| fungi | sbezverk_: maybe jamespage? | 14:12 |
| * jamespage is flattered | 14:13 | |
| jamespage | sbezverk_: reference to your issue? | 14:13 |
| jamespage | (might not be me but know a few guru's) | 14:13 |
| * fungi is also amused by the term "ubunut" | 14:13 | |
| * mordred is pretty sure jamespage is a guru in most things | 14:13 | |
| jamespage | sounds kinda tasty to me | 14:14 |
| mordred | yah. I want a bowl of ubunuts now | 14:14 |
| jamespage | lol | 14:14 |
| ihrachys | jbernard: no | 14:14 |
| ihrachys | jbernard: as most things in project-config :) | 14:14 |
| *** wolverineav has quit IRC | 14:14 | |
| mordred | jamespage: you should tell marketing that y'all should set out bowls of orange colored nuts at conferences labeled "Ubunuts" | 14:14 |
| jbernard | ihrachys: ok, so if it parses then im /probably/ okay? | 14:14 |
| ihrachys | prolly | 14:15 |
| AJaeger | jamespage: while I see you here, could you put https://review.openstack.org/443279 and https://review.openstack.org/443280 on your review queue, please? | 14:15 |
| ihrachys | :) | 14:15 |
| * jamespage looks | 14:15 | |
| sbezverk_ | jamespage: http://logs.openstack.org/50/441550/119/experimental/gate-kolla-kubernetes-deploy-ubuntu-source-3-ironic-nv/b1e8c22/console.html#_2017-03-23_04_33_26_695664 | 14:15 |
| jamespage | AJaeger: ta - shows how long those charms have been around :-) | 14:16 |
| sbezverk_ | jamespage: I already enabled strace to start debugging it | 14:16 |
| AJaeger | jamespage: thanks | 14:16 |
| jamespage | AJaeger: yw | 14:17 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config master: Kubernetes External Cloud Provider for OpenStack https://review.openstack.org/449641 | 14:17 |
| *** alexchadin has joined #openstack-infra | 14:17 | |
| *** skelso has joined #openstack-infra | 14:17 | |
| *** jbadiapa has quit IRC | 14:18 | |
| jamespage | sbezverk_: how do I see /home/jenkins/workspace/gate-kolla-kubernetes-deploy-ubuntu-source-3-ironic-nv/tests/bin/../conf/ironic/vm-1.xml ? | 14:19 |
| openstackgerrit | Dave McCowan proposed openstack-infra/project-config master: Adding Python3 Functional Test Gate for Barbican https://review.openstack.org/449343 | 14:19 |
| jamespage | I suspect its something in the libvirt xml that either the ubuntu libvirt version does not like, or the apparmor containment is rejecting | 14:19 |
| jamespage | sbezverk_: bear in mind that 16.04 has libvirt 1.3.1 which might be a bit older that on CentOS | 14:20 |
| jamespage | that said there is a newer version in the Ocata UCA if you need it | 14:20 |
| *** electrofelix has quit IRC | 14:21 | |
| SamYaple | is there a way I can trigger an action outside of infra when a new commit is made to master branch keystone for example? The only way I can think of is polling the repo which i do not wish to do | 14:21 |
| *** spzala has quit IRC | 14:21 | |
| *** electrofelix has joined #openstack-infra | 14:21 | |
| *** zz_dimtruck is now known as dimtruck | 14:21 | |
| *** amoralej is now known as amoralej|lunch | 14:21 | |
| fungi | SamYaple: you can set up an ssh socket to the gerrit event stream, or an mqtt listener on the firehose. lemme get you links to docs | 14:22 |
| SamYaple | fungi: awesome! that sounds workable. thanks | 14:22 |
| fungi | SamYaple: this is what the first option looks like https://docs.openstack.org/infra/system-config/third_party.html#reading-the-event-stream | 14:22 |
| sbezverk_ | jamespage: https://github.com/openstack/kolla-kubernetes/blob/master/tests/conf/ironic/vm-1.xml | 14:22 |
| sbezverk_ | jamespage: it happens with Ocata images | 14:23 |
| fungi | SamYaple: and here's the second option https://docs.openstack.org/infra/system-config/firehose.html#client-usage | 14:23 |
| *** bnemec is now known as beekneemech | 14:23 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config master: Enable firehose reporting on subunit workers https://review.openstack.org/449630 | 14:23 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Fix old-style mocking of nova_client https://review.openstack.org/449645 | 14:23 |
| *** nadya has quit IRC | 14:23 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Migrate server console tests to requests_mock https://review.openstack.org/449646 | 14:23 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade master: Migrate get_server_console to REST https://review.openstack.org/449647 | 14:23 |
| * mtreinish votes for using firehose | 14:24 | |
| *** rossella_s has quit IRC | 14:24 | |
| fungi | SamYaple: if you're going to be at the summit in may, mtreinish and i are giving a talk on firehose.openstack.org | 14:24 |
| fungi | in the upstream dev track,obviously | 14:24 |
| mordred | jeblair: ^^ those two patches to shade fix the issue with raw=True from the nodepool patch. we we can fix that as soon as we have new images with the new glean in them | 14:24 |
| *** chlong has joined #openstack-infra | 14:25 | |
| *** david-lyle has joined #openstack-infra | 14:26 | |
| jamespage | sbezverk_: re sudo virsh define /home/jenkins/workspace/gate-kolla-kubernetes-deploy-ubuntu-source-3-ironic-nv/tests/bin/../conf/ironic/vm-1.xml | 14:28 |
| jamespage | tried reproducing that on my local zesty install: | 14:28 |
| jamespage | error: Cannot check QEMU binary /usr/libexec/qemu-kvm: No such file or directory | 14:28 |
| jamespage | might need tweaking for the qemu-XXXX binary on Ubuntu | 14:28 |
| *** Guest85728 has quit IRC | 14:29 | |
| SamYaple | fungi: cool. yea firehose looks like what i want, having a bit of trouble with the python lib i think. getting network unreachable from the python lib only. but im still working through it | 14:29 |
| SamYaple | fungi: thanks again. this is what i needed | 14:29 |
| *** hashar is now known as hasharAway | 14:29 | |
| sbezverk_ | jamespage: I do it in the code here: https://github.com/openstack/kolla-kubernetes/blob/master/tests/bin/ironic_deploy_tests.sh#L117 | 14:29 |
| *** sree has joined #openstack-infra | 14:29 | |
| *** admcleod is now known as admcleod_afk | 14:31 | |
| jamespage | sbezverk_: ah right I see | 14:31 |
| *** ralonsoh_ has joined #openstack-infra | 14:31 | |
| fungi | SamYaple: mtreinish: yeah, i think something might be wrong with it. checking now | 14:31 |
| mordred | pabelanger: do you know if we cut a new glean with clarkb's patch yet? | 14:32 |
| SamYaple | fungi: the cli client works fine | 14:32 |
| pabelanger | mordred: Not that I know off | 14:32 |
| fungi | SamYaple: oh, it was the websockets ports i was getting that error from | 14:33 |
| *** jbadiapa has joined #openstack-infra | 14:33 | |
| *** tonytan_brb has joined #openstack-infra | 14:33 | |
| mtreinish | fungi: yeah we had to take those down because mosquitto was crashing in libwebsockets | 14:33 |
| mordred | pabelanger: we should probalby do that and kick off some new image builds | 14:33 |
| fungi | SamYaple: there's a note further up in the connection info section about how websockets is disabled pending fixes for a bug we encountered | 14:33 |
| SamYaple | gotcha | 14:33 |
| *** matrohon has quit IRC | 14:33 | |
| *** skelso has quit IRC | 14:33 | |
| mordred | infra-root: anybody have an issue with cutting a new glean release to pick up clarkb's fix for the nodepool-dsvm gate failures? | 14:33 |
| pabelanger | mordred: seems reasonable | 14:33 |
| *** skelso has joined #openstack-infra | 14:34 | |
| SamYaple | fungi: perfect! python lib working | 14:34 |
| SamYaple | fungi: thanks, apprecaite it | 14:34 |
| *** ralonsoh has quit IRC | 14:34 | |
| *** sree has quit IRC | 14:34 | |
| fungi | SamYaple: yeah, i guess our python lib examples use websockets, which explains the issue | 14:35 |
| *** amotoki has quit IRC | 14:35 | |
| mtreinish | fungi: I can add a non-websockets example to the docs | 14:35 |
| fungi | mordred: i'm in favor | 14:35 |
| *** tonytan4ever has quit IRC | 14:35 | |
| fungi | mtreinish: that might be good for now | 14:35 |
| mtreinish | it's pretty much the same thing just remove the port and transport args from the highlighted lines | 14:35 |
| mtreinish | fungi: also pick a random language from https://github.com/mqtt/mqtt.github.io/wiki/libraries I'll try to add an example for that too :) | 14:36 |
| SamYaple | mtreinish: +2 thats what i did | 14:36 |
| mtreinish | SamYaple: unless you want to push the doc patch? | 14:36 |
| *** ggnel_t has quit IRC | 14:36 | |
| * mtreinish hopes fungi doesn't pick LotusScript | 14:37 | |
| *** alex_xu has quit IRC | 14:37 | |
| *** hongbin has joined #openstack-infra | 14:37 | |
| mordred | fungi: please god pick LotusScript | 14:38 |
| SamYaple | mtreinish: SamYaple... docs patch.... does not compute | 14:38 |
| fungi | mtreinish: lua? ada? forth? | 14:38 |
| jeblair | SamYaple: whatcha building? | 14:38 |
| * fungi can pick all sorts of relatively inapplicable languages | 14:38 | |
| *** electrofelix has quit IRC | 14:38 | |
| *** alex_xu has joined #openstack-infra | 14:38 | |
| fungi | mtreinish: oh, you said _from_ not _for_ | 14:38 |
| *** electrofelix has joined #openstack-infra | 14:38 | |
| *** jascott1_ has joined #openstack-infra | 14:38 | |
| SamYaple | jeblair: repo changes to keystone/nova/etc trigger new container builds for my new project. so thing that live on quay and dockerhub will always be up-to-date | 14:39 |
| SamYaple | mtreinish: i can do that docs patch, sure. its pretty straight forward | 14:39 |
| mtreinish | SamYaple: ok cool, thanks | 14:39 |
| fungi | mtreinish: yikes, they do have lua in there (also prolog, qt and tcl...) | 14:40 |
| fungi | mtreinish: add haskell and it'll make clarkb warm and tingly | 14:40 |
| fungi | surprised rust is conspicuously absent | 14:41 |
| *** electrofelix has quit IRC | 14:41 | |
| *** ggnel_t has joined #openstack-infra | 14:41 | |
| *** electrofelix has joined #openstack-infra | 14:41 | |
| jeblair | SamYaple: thanks; what's the project? | 14:41 |
| *** marst has quit IRC | 14:41 | |
| mordred | SamYaple: fwiw, publishing container builds to quay and dockerhub also sounds like a thing that would be nice to do directly from infra - just in case you think helping with that at some point sounds like fun | 14:42 |
| mordred | SamYaple: mostly didn't want you to think that was a thing that we'd be opposed to round here | 14:42 |
| mtreinish | fungi: heh, well google says there are at least 3 rust mqtt libs (maybe 4 if mqtt-rs and mqtt_rs are different things) | 14:42 |
| mtreinish | but my guess is that no one bothered to link to any of them | 14:42 |
| mordred | infra-root: pushing release 1.9.0 of glean | 14:43 |
| AJaeger | jeblair: we have created a stable/ocata branch for openstack-manuals and now publishing is somehow screwed up. https://docs.openstack.org/draft/config-reference/.root-marker is the branch-point but https://docs.openstack.org/.root-marker is current and pushed later. Could it be that we somehow got te root-marker there - and should not? | 14:43 |
| jeblair | clarkb, mordred, pabelanger: i guess we're happy with https://review.openstack.org/449366 even though http://logs.openstack.org/66/449366/1/check/gate-dsvm-nodepool-debian-src-nv/d65fd44/console.html timed out right before the xenial test should have finished? maybe we need to bump the timeout on that job since it builds 3 images? | 14:43 |
| *** jascott1_ has quit IRC | 14:43 | |
| *** spzala has joined #openstack-infra | 14:43 | |
| mordred | jeblair: yah - we probably should bump the timeout on that | 14:44 |
| SamYaple | jeblair: the project as it exists today, its https://github.com/yaodu/ . Here is patch to pull it into openstack https://review.openstack.org/#/c/449224/ under the name LOCI | 14:44 |
| pabelanger | jeblair: ya, it was a job timeout. I would be okay with a bump | 14:44 |
| SamYaple | mordred: sure thing. i would like to see infra doing it. ive had the conversation a few times, but i dont know infra enough to push the work forward :/ its a failing on my part | 14:44 |
| rm_work | anyone have any idea when it is planned to cut the next DIB release? | 14:45 |
| pabelanger | I did ssh in to the test node last night an watch things | 14:45 |
| *** amotoki has joined #openstack-infra | 14:45 | |
| * mtreinish starts reading mqtt-hs code | 14:45 | |
| AJaeger | jeblair: do we need to delete https://docs.openstack.org/draft/config-reference/.root-marker to be able to publish there? | 14:45 |
| pabelanger | IIRC, we did boot ubuntu-xenail after logs were collected | 14:45 |
| pabelanger | rm_work: they usually happen every week or so | 14:45 |
| rm_work | hmmmmm | 14:46 |
| *** ggnel_t has quit IRC | 14:46 | |
| jeblair | AJaeger: looking | 14:46 |
| mordred | SamYaple: well, we'll figure it out eventually | 14:46 |
| sdague | mtreinish / fungi what was the mqtt scaling issue you were seeing? | 14:46 |
| sdague | was that written up somewhere? | 14:47 |
| mordred | SamYaple: btw - your scripts look good - if you haven't played with consuming the bindep.txt files yet, you might want to - could save you some ongoing maint in your dockerfiles | 14:47 |
| *** amotoki has quit IRC | 14:47 | |
| fungi | sdague: crashes in libwebsockets https://github.com/eclipse/mosquitto/issues/278 | 14:47 |
| jeblair | SamYaple, mordred: https://review.openstack.org/401003 is a WIP to add a local docker server. however, just publishing to dockerhub is likely easier; can probably just write a post job and put it on the repos in question. | 14:47 |
| SamYaple | mordred: ohh.. thats a good idea. | 14:48 |
| mtreinish | sdague: it wasn't really a scaling issue. Just websockets not working | 14:48 |
| *** alexchadin has quit IRC | 14:48 | |
| mordred | jeblair, SamYaple: I agree - it's publishing to a local dockerhub that's hard - pushing to _actual_ dockerhub is easier | 14:49 |
| AJaeger | jeblair: if my theory is true, we need to delete https://docs.openstack.org/draft/install-guide-obs/.root-marker and others as well (so, install-guide-*/.root-marker). I'm currently puzzled how the docs job created this at all | 14:49 |
| mtreinish | sdague: I was investigating other brokers (which I'll need to pick up again at some point) because we wanted a better ha/clustering story than what mosquitto provides | 14:49 |
| *** marst has joined #openstack-infra | 14:49 | |
| jamespage | sbezverk_: not forgot about your question | 14:49 |
| mordred | in the fullness of time being able to manage the creation of things in the openstack/ namespace on dockerhub also needs automation, but I don't think we'd need to block on that for this | 14:49 |
| mtreinish | sdague: some of the other brokers out there have that concept baked into them | 14:49 |
| * AJaeger needs to step out for a bit | 14:49 | |
| jamespage | sbezverk_: anything funky about how libvirt is being run in this deployment? | 14:49 |
| jeblair | AJaeger: okay, i won't be able to work on this without your help, so let me know when you're back | 14:50 |
| jeblair | AJaeger: for reference, here is the build log: http://logs.openstack.org/e3/e32d413f74160615cd7b17930f1f9224e391b223/post/openstack-manuals-tox-doc-publishdocs/92934b9/console.html | 14:50 |
| mtreinish | sdague: but that was more something for later, when we started depending on it for things | 14:50 |
| mtreinish | although if it got us around the websockets issue it might be a reason to potentially switch sooner | 14:50 |
| SamYaple | mordred: internal to infra docker push would be nice. i think when we discussed this in the past that was kind of vetoed in favor of infra-hosted docker registry | 14:51 |
| *** matrohon has joined #openstack-infra | 14:51 | |
| mordred | SamYaple: ah - sorry, probably a misunderstanding or just miscommunication or something. the local infra-hosted docker registry is about solving gate stability | 14:52 |
| mordred | like, pulling from docker hub is going to be as bad as cloning from github in gate jobs | 14:53 |
| fungi | SamYaple: i believe mnaser is writing up a new spec for what you're talking about | 14:53 |
| SamYaple | oh I see | 14:53 |
| SamYaple | all good info to have | 14:53 |
| mnaser | SamYaple let me link you | 14:53 |
| mordred | but publishing artifacts to the place where consumes expect to consume them is totally in-game | 14:53 |
| mnaser | https://review.openstack.org/#/c/447524/ | 14:53 |
| * SamYaple is reading | 14:53 | |
| mnaser | i mention kolla there but i guess it can work for any project | 14:53 |
| mnaser | we'd have a bit pusher for dockerhub | 14:53 |
| mordred | sort of like how we run pypi mirrors for the gate, but also publish to actual pypi so that people can pip install things | 14:54 |
| sbezverk_ | jamespage: I hope so, as am running out of ideas.. fyi. it is containers env. libvirtd runs in the container same as in case of centos, and I checked that libvirtd socket is available on the host.. | 14:54 |
| fungi | (and the mirroring pypi and publishing to public pypi are two completely unrelated systems/processes) | 14:54 |
| jamespage | sbezverk_: yeah the error is actually coming from the qemu process that libvirt is trying to spawn | 14:54 |
| SamYaple | mnaser: nice. yea thats nice | 14:54 |
| jamespage | which makes this a little tricker | 14:54 |
| mnaser | SamYaple havent gotten around to republishing and addressing the concerns | 14:55 |
| SamYaple | mordred: in the short term, we can publish *a* tarball to tarballs.openstack.org for the single thing we would need to fetch from outside infra | 14:55 |
| *** jbadiapa has quit IRC | 14:55 | |
| SamYaple | mordred: so from the start there shouldn't be extra load on infra | 14:56 |
| SamYaple | at least from outside infra | 14:56 |
| mordred | SamYaple: totes - that souds great | 14:56 |
| *** jbadiapa has joined #openstack-infra | 14:56 | |
| mordred | SamYaple: and in the fullness of time, we'll have all of this supported and everyone will be happy! | 14:56 |
| mordred | jeblair: don't know if you saw in the swarm of scrollback, but https://review.openstack.org/#/c/449647 is the stack in shade that makes that raw=raw bit of the patch in nodepool go away | 14:57 |
| *** jianghuaw-m has joined #openstack-infra | 14:57 | |
| *** HeOS has quit IRC | 14:57 | |
| *** jkilpatr has joined #openstack-infra | 14:58 | |
| *** prateek has joined #openstack-infra | 14:58 | |
| *** Vivek__ is now known as Vivek | 14:58 | |
| *** Vivek has quit IRC | 14:58 | |
| *** Vivek has joined #openstack-infra | 14:58 | |
| jeblair | mordred: the post goes through the TM right? | 14:59 |
| mordred | jeblair: yup | 14:59 |
| *** xarses_ has joined #openstack-infra | 15:00 | |
| *** eharney has quit IRC | 15:00 | |
| *** xarses_ has quit IRC | 15:00 | |
| *** eharney has joined #openstack-infra | 15:00 | |
| sdague | mtreinish: so... if people interface to the mqtt side and not the websocket side, it was fine? | 15:00 |
| *** xarses_ has joined #openstack-infra | 15:00 | |
| *** dhajare has quit IRC | 15:01 | |
| fungi | sdague: correct | 15:01 |
| *** prateek has quit IRC | 15:01 | |
| *** prateek has joined #openstack-infra | 15:02 | |
| *** jianghuaw-m has quit IRC | 15:05 | |
| *** kien-ha has joined #openstack-infra | 15:06 | |
| *** admcleod_afk has quit IRC | 15:06 | |
| *** admcleod has joined #openstack-infra | 15:06 | |
| *** admcleod has quit IRC | 15:07 | |
| jamespage | sbezverk_: the qemu instance logs from /var/log/libvirt/qemu might have more | 15:07 |
| *** jianghuaw has joined #openstack-infra | 15:07 | |
| *** admcleod has joined #openstack-infra | 15:08 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: puppet-vitrage: run integration scenario001 job https://review.openstack.org/449673 | 15:11 |
| *** dangers_away is now known as danger_fo | 15:11 | |
| sbezverk_ | jamespage: http://logs.openstack.org/50/441550/119/experimental/gate-kolla-kubernetes-deploy-ubuntu-source-3-ironic-nv/b1e8c22/logs/libvirt-vm-logs.txt | 15:12 |
| *** skelso has quit IRC | 15:14 | |
| *** skelso has joined #openstack-infra | 15:14 | |
| *** skelso has quit IRC | 15:14 | |
| *** david-lyle has quit IRC | 15:15 | |
| *** vikrant has quit IRC | 15:16 | |
| *** nadya has joined #openstack-infra | 15:16 | |
| pabelanger | Wow, pushing 19 hours for tripleo change queue in gate. What is going on there | 15:17 |
| pabelanger | EmilienM: ^is there a summary of the issue(s) some place? | 15:17 |
| EmilienM | pabelanger: | 15:19 |
| EmilienM | https://bugs.launchpad.net/tripleo/+bugs?field.searchtext=&orderby=-importance&field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_commenter=&field.subscriber=&field.structural_subscriber=&field.tag=alert+&field.tags_combinator=ANY&field.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_me.used=&field.has_pa | 15:19 |
| EmilienM | tch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&field.has_no_blueprints.used=&field.has_no_blueprints=on&search=Search | 15:19 |
| EmilienM | pabelanger: https://goo.gl/Lxn1Eq | 15:19 |
| *** tellesnobrega has quit IRC | 15:20 | |
| pabelanger | EmilienM: is anything merging at this point? | 15:20 |
| EmilienM | pabelanger: very hard | 15:21 |
| EmilienM | pabelanger: we keep rechecking | 15:21 |
| EmilienM | pabelanger: this week was terrible as you can see | 15:21 |
| EmilienM | pabelanger: /join #tripleo and you'll see | 15:21 |
| pabelanger | EmilienM: right, it doesn't seem like rechecking is helping. Just slowing things down | 15:21 |
| jbernard | mriedem: is it expected that graphite and logstash do not agree? graphite indicates a high failure rate, but a logstash query for recent ceph failures seems not to show any... am I doing it wrong? | 15:22 |
| EmilienM | pabelanger: go say it to my team :) I'm not doing all the rechecks | 15:22 |
| AJaeger | jeblair: I'm back, let me double check that build log... | 15:22 |
| EmilienM | pabelanger: what would you do for helping the situation? | 15:23 |
| pabelanger | EmilienM: Is there any patches up confirmed to fix any of the issues you linked? Or still debugging at this point | 15:24 |
| EmilienM | pabelanger: all patches are in gate | 15:24 |
| openstackgerrit | Andrey Kurilin proposed openstack-infra/project-config master: [rally] Add support for rally-branching https://review.openstack.org/449683 | 15:24 |
| pabelanger | EmilienM: at top or bottom? | 15:24 |
| mriedem | jbernard: elasticsearch is backed up | 15:25 |
| EmilienM | pabelanger: I don't know | 15:25 |
| mriedem | so logstash isn't showing the most recent failures | 15:25 |
| AJaeger | jeblair: that's this change - on master, isn't it? http://git.openstack.org/cgit/openstack/openstack-manuals/commit/?id=e32d413f74160615cd7b17930f1f9224e391b223 | 15:25 |
| jbernard | mriedem: ok thanks | 15:25 |
| mriedem | jbernard: or your logstash query is wrong | 15:25 |
| AJaeger | but this shows stable/ocata : http://logs.openstack.org/e3/e32d413f74160615cd7b17930f1f9224e391b223/post/openstack-manuals-tox-doc-publishdocs/92934b9/_zuul_ansible/vars.yaml | 15:25 |
| *** burgerk has joined #openstack-infra | 15:25 | |
| *** rossella_s has joined #openstack-infra | 15:25 | |
| AJaeger | jeblair: but we tagged 15.0.0 and branch on that id, so did a post job run on that with stable/ocata ? | 15:26 |
| EmilienM | pabelanger: I can verify | 15:26 |
| pabelanger | EmilienM: well, if you can figure out which patch is a fix. We can promote to the top of the change queue, stopping zuul from kicking out broken jobs. It is currently 18hrs deep now, and going to take about that time to clear 20 patches | 15:26 |
| EmilienM | pabelanger: let me give you the IDs of patches we need merged ASAP | 15:26 |
| EmilienM | pabelanger: gimme 2 min | 15:26 |
| *** nadya has quit IRC | 15:27 | |
| pabelanger | EmilienM: well, won't merge, but promote | 15:27 |
| andreykurilin | AJaeger: hi! Can you look at https://review.openstack.org/449683 ? I do not have experience in such changes, so I'm unsure that it is made in right way :) | 15:28 |
| EmilienM | pabelanger: 448187 448541 448512 to start | 15:29 |
| rm_work | anyone notice an issue with scrolling through code on gerrit reviews, where it jumps around like crazy? I think it might be related to natural scrolling behavior on OSX maybe? (I am on OSX) | 15:30 |
| *** rbrndt has quit IRC | 15:30 | |
| *** jaosorior has quit IRC | 15:30 | |
| *** rbrndt has joined #openstack-infra | 15:30 | |
| *** rbrndt has quit IRC | 15:30 | |
| pabelanger | EmilienM: 448187 448541 are near top, 40mins out from merging. So we can keep them | 15:31 |
| pabelanger | EmilienM: 448512 is kicked out of the gate, but behind the others | 15:31 |
| EmilienM | pabelanger: 448036 also | 15:31 |
| EmilienM | pabelanger: and that's it I guess | 15:32 |
| *** slaweq_ has quit IRC | 15:32 | |
| AJaeger | jeblair: any idea why that job is run on stable/ocata? http://logs.openstack.org/e3/e32d413f74160615cd7b17930f1f9224e391b223/post/openstack-manuals-tox-doc-publishdocs/92934b9/_zuul_ansible/vars.yaml I now know what happened and can avoid this for the future | 15:32 |
| EmilienM | pabelanger: thanks... | 15:32 |
| *** kien-ha has quit IRC | 15:32 | |
| AJaeger | jeblair: but we need to delete those root-marker files | 15:32 |
| pabelanger | EmilienM: 448036 is 1h15mins out too | 15:33 |
| *** martinkopec has quit IRC | 15:33 | |
| pabelanger | EmilienM: okay, I'll watch those 3 | 15:33 |
| *** adarazs has joined #openstack-infra | 15:33 | |
| mtreinish | fungi: there's your haskell, I'll add it to the docs: http://paste.openstack.org/show/604058/ | 15:33 |
| EmilienM | pabelanger: thx | 15:33 |
| mtreinish | although I don't really know how it works :) | 15:34 |
| jeblair | fungi: any chance you could boot up a new lists server? i should be able to try upgrading it in a bit | 15:34 |
| AJaeger | andreykurilin: that's ugly - and I cannot help with that today, hope others can | 15:34 |
| jeblair | AJaeger: trying to catch up :) | 15:34 |
| AJaeger | ;) | 15:34 |
| andreykurilin | AJaeger: heh. ok, thanks | 15:34 |
| andreykurilin | jeblair: hi hi! Can you help me with making https://review.openstack.org/449683 looks good ?) | 15:35 |
| jeblair | AJaeger: that job ran on branch creation | 15:35 |
| *** rossella_s has quit IRC | 15:35 | |
| jeblair | AJaeger: the "oldrev 000000000..." tells us that | 15:35 |
| *** jkilpatr has quit IRC | 15:35 | |
| fungi | jeblair: oh, sure i'll get it booted in moments and let you know | 15:36 |
| AJaeger | jeblair: ah.. Ok, then I know what went wrong and can avoid that. | 15:36 |
| AJaeger | jeblair: thanks. Could you remvoe those root-markers, then, please? | 15:36 |
| jeblair | AJaeger: i think someone added some things to zuul pipelines to let us say don't match on branch creation or deletion, if we want to adjust the pipeline config to not run jobs in those cases | 15:37 |
| jeblair | AJaeger: which file needs deleting? | 15:37 |
| openstackgerrit | Ken Dreyer proposed openstack-infra/jenkins-job-builder master: scm: handle "bitbucketweb" browser for mercurial https://review.openstack.org/449692 | 15:42 |
| *** prateek has quit IRC | 15:42 | |
| *** knangia has joined #openstack-infra | 15:42 | |
| *** pvaneck has joined #openstack-infra | 15:43 | |
| *** ggnel_t has joined #openstack-infra | 15:45 | |
| *** kien-ha has joined #openstack-infra | 15:45 | |
| *** gyee has joined #openstack-infra | 15:45 | |
| AJaeger | jeblair: these files draft/install-guide-*/.root-marker draft/config-reference/.root-marker draft/networking-guide/.root-marker need deletion | 15:45 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool feature/zuulv3: Remove SSH support from nodepool https://review.openstack.org/446683 | 15:46 |
| jamespage | sbezverk_: the pidfile function in qemu is only triggered if libvirt passes it down to the qemu process - using the template you provided, I can get a kvm instance running OK: | 15:46 |
| jamespage | https://www.irccloud.com/pastebin/kXhy6bqt/ | 15:46 |
| jamespage | but as you can see no -pidfile argument... | 15:47 |
| *** vhosakot has joined #openstack-infra | 15:48 | |
| jamespage | sbezverk_: what goes on here | 15:48 |
| jamespage | 2017-03-23 04:04:42.964378 | changed: [localhost] => (item={u'dest': u'libvirtd.conf', u'src': u'libvirtd.conf.j2'}) | 15:48 |
| *** yamahata has joined #openstack-infra | 15:49 | |
| *** flepied has quit IRC | 15:49 | |
| sbezverk_ | jamespage: it is just ansible, prepare libvirtd.conf from a temaplte | 15:50 |
| sbezverk_ | jamespage: I can show you what we get in the end | 15:50 |
| sbezverk_ | jamespage: http://logs.openstack.org/50/441550/119/experimental/gate-kolla-kubernetes-deploy-ubuntu-source-3-ironic-nv/b1e8c22/logs/configmaps.yaml, do a search for libvirtd.conf | 15:51 |
| jamespage | sbezverk_: ah | 15:51 |
| *** d0ugal has quit IRC | 15:52 | |
| *** amoralej|lunch is now known as amoralej | 15:52 | |
| jamespage | sbezverk_: pondering whether is something todo with https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova/templates/qemu.conf.j2#L3 | 15:53 |
| jamespage | sbezverk_: might be a mismatch between the files directly created via libvirt and those created via nova perhaps? | 15:54 |
| EmilienM | adarazs, trown: please review the list of outstanding patches we have right now to make tripleo ci more stable: https://review.openstack.org/#/q/topic:tripleo/outstanding | 15:54 |
| *** annegentle has joined #openstack-infra | 15:54 | |
| EmilienM | adarazs, trown: please add patches if I missed some of them | 15:54 |
| sbezverk_ | jamespage: hm, looks really interesting thank you very much I will investigate it | 15:55 |
| *** amotoki has joined #openstack-infra | 15:56 | |
| fungi | jeblair: it' | 15:56 |
| *** markus_z has quit IRC | 15:56 | |
| jamespage | sbezverk_: well hope that helps - good luck :-) | 15:57 |
| fungi | s been in a "building" state for about 20 minutes | 15:57 |
| *** hasharAway is now known as hashar | 15:57 | |
| adarazs | EmilienM: I don't see anything missing. | 15:57 |
| fungi | will be lists-upgrade-test.openstack.org 88727f73-052b-4800-b974-c1429128b59d when it eventually completes | 15:57 |
| EmilienM | adarazs: ack | 15:57 |
| *** esikachev has joined #openstack-infra | 15:58 | |
| *** krtaylor has joined #openstack-infra | 15:59 | |
| *** rbrndt has joined #openstack-infra | 16:01 | |
| *** pcaruana has quit IRC | 16:03 | |
| pabelanger | fungi: so, question. What is our policy on force merging things for projects? In this case, EmilienM asked the question to get some CI fixes merged. It seems there is more then once issue they are fighting. I know promotes are usually done for devstack, but not sure force merge of code | 16:08 |
| openstackgerrit | Lubosz Kosnik (diltram) proposed openstack-infra/project-config master: Update nova-lxd gate to use local_conf https://review.openstack.org/449701 | 16:08 |
| EmilienM | fungi: the tl;dr is this week was terrible and different things broke our CI and the fixes fail to land because fixes are applied to different repos & projects, etc | 16:10 |
| *** trown is now known as trown|lunch | 16:10 | |
| EmilienM | I dressed a list of things we need merged asap: https://review.openstack.org/#/q/topic:tripleo/outstanding | 16:10 |
| EmilienM | all jobs passed the check at least once (or more) | 16:10 |
| *** andreww has joined #openstack-infra | 16:11 | |
| EmilienM | pabelanger: it seems like all patches from the list have jobs running and nothing is in queue for now | 16:11 |
| *** e0ne has quit IRC | 16:11 | |
| EmilienM | pabelanger: let's wait them to finish and take a decision on what we do if they still fail to merge | 16:12 |
| jeblair | AJaeger: removed those 5 files. | 16:13 |
| AJaeger | thanks, jeblair . let me merge something ;) | 16:13 |
| *** tjones has joined #openstack-infra | 16:14 | |
| *** d0ugal has joined #openstack-infra | 16:14 | |
| *** d0ugal has quit IRC | 16:14 | |
| *** d0ugal has joined #openstack-infra | 16:14 | |
| *** xarses_ has quit IRC | 16:14 | |
| *** vhosakot_ has joined #openstack-infra | 16:14 | |
| *** vhosakot has quit IRC | 16:14 | |
| *** xarses_ has joined #openstack-infra | 16:15 | |
| *** Goneri has quit IRC | 16:15 | |
| *** andreww has quit IRC | 16:15 | |
| fungi | EmilienM: pabelanger: generally, if there are fixes approved which will significantly reduce job failures for multiple projects and have more than a few changes ahead of them in the gate pipeline, we're pretty okay with promoting them forward to the front to get them merged more quickly | 16:16 |
| fungi | jeblair: still "building" but has been assigned 166.78.27.191 and 2001:4800:7811:513:be76:4eff:fe04:d8bf which i'll get into forward/reverse dns | 16:17 |
| jeblair | fungi: cool, do you have that etherpad handy? | 16:18 |
| fungi | jeblair: i don't remember the url, no | 16:18 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool feature/zuulv3: Remove ipv6-preferred and rely on interface_ip https://review.openstack.org/449705 | 16:18 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Consume interface_ip from nodepool node https://review.openstack.org/449706 | 16:18 |
| fungi | i can dig it up in a sec tho | 16:18 |
| pabelanger | fungi: right, that's basically what I have thought | 16:18 |
| jeblair | fungi: https://etherpad.openstack.org/p/lists.o.o-trusty-upgrade | 16:18 |
| EmilienM | pabelanger: let's wait end of oooq-multinode job results and see how it goes | 16:19 |
| EmilienM | pabelanger: in ~50 min we'll take decision - /me afk 30 min now for lunch | 16:19 |
| *** sshnaidm|off has joined #openstack-infra | 16:19 | |
| *** d0ugal has quit IRC | 16:22 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/system-config master: Add an example not using websockets https://review.openstack.org/449711 | 16:23 |
| *** gouthamr has quit IRC | 16:23 | |
| *** mattmceuen has joined #openstack-infra | 16:24 | |
| openstackgerrit | Khai Do proposed openstack-infra/gerrit master: Replace submodule url reference from relative to absolute https://review.openstack.org/449712 | 16:24 |
| fungi | jeblair: still in "building" state, though the rackspace dashboard has been listing it as "40% complete" for the past 10 minutes or so (whatever that means) | 16:25 |
| fungi | it was "10% complete" there for quite a while | 16:26 |
| *** tjones1 has joined #openstack-infra | 16:27 | |
| *** tjones has quit IRC | 16:27 | |
| *** hashar has quit IRC | 16:29 | |
| jeblair | fungi: that's way more complete! | 16:31 |
| *** gouthamr has joined #openstack-infra | 16:32 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool feature/zuulv3: Remove api-timeout and provider.image-type https://review.openstack.org/449342 | 16:32 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool feature/zuulv3: Remove deprecated networks syntax https://review.openstack.org/449354 | 16:32 |
| *** rlandy is now known as rlandy|brb | 16:32 | |
| *** david-lyle has joined #openstack-infra | 16:33 | |
| *** jascott1_ has joined #openstack-infra | 16:36 | |
| *** jascott1_ has quit IRC | 16:36 | |
| *** bogdando has quit IRC | 16:36 | |
| openstackgerrit | Adam Harwell proposed openstack/diskimage-builder master: Use DIB_PYTHON_VERSION to run commands https://review.openstack.org/449721 | 16:39 |
| rm_work | ^^ still testing a theory | 16:40 |
| fungi | jeblair: it's up! | 16:40 |
| fungi | bup cronjob looks to still be correctly commented out | 16:41 |
| *** jascott1_ has joined #openstack-infra | 16:42 | |
| *** Apoorva has joined #openstack-infra | 16:42 | |
| *** Apoorva has quit IRC | 16:42 | |
| fungi | mailman initscript start symlinks still absent at all runlevels | 16:42 |
| fungi | looking good | 16:42 |
| *** Apoorva has joined #openstack-infra | 16:42 | |
| fungi | no mailman daemon in the process list | 16:42 |
| AJaeger | fungi, pabelanger codesearch is not changed as far as I can see | 16:43 |
| *** beslemon has joined #openstack-infra | 16:43 | |
| *** beslemon has quit IRC | 16:44 | |
| *** kien-ha has quit IRC | 16:45 | |
| *** MasterOfBugs has joined #openstack-infra | 16:46 | |
| *** erikmwilson has quit IRC | 16:47 | |
| *** erikmwilson has joined #openstack-infra | 16:47 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Refactor out Changeish https://review.openstack.org/448913 | 16:49 |
| *** aarefiev is now known as aarefiev_afk | 16:49 | |
| *** tjones1 has left #openstack-infra | 16:52 | |
| *** matrohon has quit IRC | 16:52 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Document signing key cross-signing transition https://review.openstack.org/449729 | 16:55 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Silence hieraedit output during key replacement https://review.openstack.org/449730 | 16:55 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Stop saying we announce signing key transitions https://review.openstack.org/449731 | 16:55 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Correct signing node double-check example https://review.openstack.org/449732 | 16:55 |
| *** psachin has joined #openstack-infra | 16:55 | |
| *** rbrndt has quit IRC | 16:55 | |
| *** e0ne has joined #openstack-infra | 16:56 | |
| *** beslemon has joined #openstack-infra | 16:57 | |
| fungi | AJaeger: yeah, the only other thing i can think of is that the changes we merged aren't actually valid... i can take a look after lunch | 16:57 |
| *** rwsu has quit IRC | 16:57 | |
| *** pvaneck has quit IRC | 16:59 | |
| *** makowals has quit IRC | 17:00 | |
| *** yamahata has quit IRC | 17:00 | |
| *** rwsu has joined #openstack-infra | 17:00 | |
| *** nadya has joined #openstack-infra | 17:01 | |
| *** gouthamr has quit IRC | 17:01 | |
| clarkb | mordred: when I wake up is apparently really late today. sorry, but catching up on glean things | 17:01 |
| *** beslemon has quit IRC | 17:02 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: WIP: Add lvm management to diskimage-builder https://review.openstack.org/444403 | 17:03 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool feature/zuulv3: Remove ipv6-preferred and rely on interface_ip https://review.openstack.org/449705 | 17:04 |
| AJaeger | fungi, that would be sad - but an explanation. | 17:04 |
| *** gouthamr has joined #openstack-infra | 17:05 | |
| AJaeger | fungi, yeah, indeed ;( | 17:07 |
| AJaeger | fungi: I'm fixing... | 17:07 |
| *** tjones has joined #openstack-infra | 17:09 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/jeepyb master: Fix startswith usage https://review.openstack.org/449734 | 17:10 |
| AJaeger | fungi, clarkb, could you +2A this, please? ^ | 17:10 |
| AJaeger | with this change, I now get a config.json file. Sorry for introducing that ;( | 17:10 |
| *** rlandy|brb is now known as rlandy | 17:11 | |
| clarkb | sure enough there is an exception when you do that yay python | 17:11 |
| rajinir | nodepool upgrade causes sql error. How can I upgrade the db? | 17:11 |
| clarkb | rajinir: there hasn't been a nodepool db schema change in quite some time, but all of the details were sent to the infra mailing list the last time in novemeber iirc | 17:12 |
| *** electrofelix has quit IRC | 17:13 | |
| fungi | AJaeger: approved. i'm popping out to lunch but i'll remove codesearch.o.o from the emergency disable list temporarily so that will get applied | 17:14 |
| fungi | oh, it's still removed, i haven't put it back yet | 17:15 |
| fungi | good | 17:15 |
| *** d0ugal has joined #openstack-infra | 17:17 | |
| fungi | okay, heading to grab some lunch. back in a while | 17:17 |
| *** makowals has joined #openstack-infra | 17:17 | |
| *** ralonsoh_ has quit IRC | 17:17 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool feature/zuulv3: Exercise statsd in tests and fix https://review.openstack.org/449737 | 17:18 |
| openstackgerrit | Khai Do proposed openstack-infra/project-config master: Cleanup Gerrit plugin builds https://review.openstack.org/449739 | 17:18 |
| *** yamamoto has quit IRC | 17:19 | |
| openstackgerrit | Merged openstack-infra/jeepyb master: Fix startswith usage https://review.openstack.org/449734 | 17:20 |
| openstackgerrit | Lubosz Kosnik (diltram) proposed openstack-infra/project-config master: Enable Octavia gate with RHEL amphora https://review.openstack.org/449741 | 17:21 |
| *** dtantsur is now known as dtantsur|pto | 17:22 | |
| openstackgerrit | Lubosz Kosnik (diltram) proposed openstack-infra/project-config master: Enable Octavia gate with Centos amphora https://review.openstack.org/449741 | 17:24 |
| openstackgerrit | Lubosz Kosnik (diltram) proposed openstack-infra/project-config master: Enable Octavia gate with Centos amphora https://review.openstack.org/449741 | 17:26 |
| openstackgerrit | Khai Do proposed openstack-infra/project-config master: Build gerrit plugins against 2.13.7 release https://review.openstack.org/449745 | 17:27 |
| *** flepied has joined #openstack-infra | 17:28 | |
| AJaeger | fungi, thank | 17:28 |
| *** nadya has quit IRC | 17:30 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Add a dequeue command to zuul client https://review.openstack.org/95035 | 17:32 |
| *** tellesnobrega has joined #openstack-infra | 17:32 | |
| clarkb | mordred: have you seen prometheanfires comment on https://review.openstack.org/#/c/279818/6/glean/install.py ? I kinda feel liek we shouldn't be expected to accomodate every little distro idiosynchrasity/rule when doing this. I'm sure we violate all the ubuntu and red hat rules for example | 17:33 |
| *** rossella_s has joined #openstack-infra | 17:33 | |
| clarkb | rather we want our service to function on the distros and this is functional aiui | 17:33 |
| mordred | looking | 17:33 |
| *** trown|lunch is now known as trown | 17:34 | |
| mordred | clarkb: hrm. well - I don't care about policy per-se - but I do think we should not surprise people - if openrc starts things, but then someone does systemctl stop $something - does that work and would that user expect it to? | 17:36 |
| *** annegentle has quit IRC | 17:36 | |
| openstackgerrit | Khai Do proposed openstack-infra/project-config master: Cleanup Gerrit plugin builds https://review.openstack.org/449739 | 17:36 |
| clarkb | mordred: my reading of that is that the user would have to opt into using systemd (at build time or later) in order to get the systemd bits in order for systemctl to be there and work | 17:37 |
| *** yamahata has joined #openstack-infra | 17:37 | |
| clarkb | mordred: but that their policy is to accoodate users potentially doing that by writing down the unit files for both | 17:37 |
| *** marst has quit IRC | 17:37 | |
| clarkb | the problem with that is the udev rules will fire always | 17:37 |
| clarkb | so its not so easy to have glean in a half installed state for systemd if you don't have a systemd | 17:37 |
| *** diltram has joined #openstack-infra | 17:38 | |
| diltram | hey | 17:38 |
| diltram | I have a question about project-config | 17:38 |
| *** dimtruck is now known as zz_dimtruck | 17:38 | |
| clarkb | that said we could change it to an if and everyone else should be fine so would only affect gentoo that way | 17:38 |
| diltram | https://review.openstack.org/#/c/449741/3/jenkins/jobs/octavia.yaml | 17:38 |
| clarkb | mordred: ^ so maybe thats the easieest thing to do | 17:38 |
| diltram | here I'm adding a new gate check | 17:38 |
| diltram | and in gate name I'm adding {hypervisor} and {amphora_os} | 17:39 |
| openstackgerrit | Khai Do proposed openstack-infra/project-config master: Build gerrit plugins against 2.13.7 release https://review.openstack.org/449745 | 17:39 |
| diltram | can I somehow don anything based on those values? | 17:39 |
| diltram | when amphora_os=centos7 I would like to add special options into local_conf | 17:39 |
| clarkb | diltram: use {amphora_os} in your job body and it will get text substituted | 17:40 |
| clarkb | you can see the octaviatest and octaviaversion params are already used | 17:40 |
| AJaeger | diltram: might be better to do in your devstack plugin | 17:40 |
| *** derekh has quit IRC | 17:40 | |
| *** martinda has joined #openstack-infra | 17:41 | |
| mordred | clarkb: yah - I think changing to an if isn't terrible | 17:41 |
| *** nadya has joined #openstack-infra | 17:41 | |
| clarkb | mordred: I'm just grumpy because I don't think what gentoo wants is actually a sane thing for people to do (for reasons of the udev rule) | 17:41 |
| diltram | clarkb, AJaeger: ok, thx | 17:42 |
| clarkb | its even worse than what we have to do supporting upstart and openrc and systemd in the same tool because apparently now you ahve to support them on the same host at the same time | 17:42 |
| diltram | I'm gonna make this as static name for now | 17:42 |
| *** mtanino has quit IRC | 17:43 | |
| diltram | we need first make those working then I can try to standarize them and try to minimize the amount of configs | 17:43 |
| *** rossella_s has quit IRC | 17:44 | |
| *** lucasagomes is now known as lucas-afk | 17:44 | |
| *** scarab_ has joined #openstack-infra | 17:44 | |
| SamYaple | fungi: mtreinish thanks for hte firehose help. got my little kicker-offer script going in full swing | 17:44 |
| jeblair | clarkb, fungi: i think the io on the list test upgrade server is *very* slow :/ | 17:44 |
| *** EricGonczer_ has quit IRC | 17:44 | |
| *** spzala has quit IRC | 17:45 | |
| jeblair | clarkb, fungi: also, we have an old bounces directory we saved from some earlier mishap i think we can delete | 17:46 |
| clarkb | jeblair: you think the slow io is just luck of the vm boot draw or some bigger issue that we'll need to address? | 17:47 |
| *** spzala has joined #openstack-infra | 17:47 | |
| jeblair | clarkb: well... we *might* want to try an upgrade to ext4. :) | 17:47 |
| martinda | is gearman able to prioritize jobs, or suspend and resume jobs ? | 17:48 |
| clarkb | oh nice ext3 lives on | 17:48 |
| clarkb | martinda: it can prioritize but only at 3 levels, low medium high. high priority jobs will all be services before medium and so on | 17:48 |
| clarkb | martinda: suspend and resume would need to be implemented on top of gearman | 17:48 |
| martinda | thanks clarkb. Is it able to run interactive jobs? | 17:50 |
| *** tesseract has quit IRC | 17:50 | |
| martinda | the background here is that I am looking for an alternative to LSF. | 17:50 |
| *** spzala has quit IRC | 17:52 | |
| *** spzala has joined #openstack-infra | 17:52 | |
| jeblair | clarkb, fungi: i bet if we had deleted the bounces.old directory, snapshot and server creation would have been much faster. :| | 17:52 |
| *** zz_dimtruck is now known as dimtruck | 17:54 | |
| clarkb | martinda: I'm not sure what you mean by interactive jobs. My hunch is no. Gearman itself is an incredibly simple binary protocol for doing simple things and the assumption seems to be you'll build your own need specific protocol atop it to do what you end up needing | 17:54 |
| clarkb | martinda: but SpamapS is now a maintainer of gearman so may have better input that I have | 17:55 |
| *** yamamoto has joined #openstack-infra | 17:56 | |
| *** baoli has quit IRC | 17:56 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Only force http for repos that allow it https://review.openstack.org/448187 | 17:57 |
| *** jcoufal has joined #openstack-infra | 17:58 | |
| *** beslemon has joined #openstack-infra | 17:59 | |
| *** jcoufal__ has joined #openstack-infra | 18:01 | |
| *** jianghuaw has quit IRC | 18:01 | |
| *** marst has joined #openstack-infra | 18:01 | |
| *** jcoufal_ has quit IRC | 18:01 | |
| *** annegentle has joined #openstack-infra | 18:01 | |
| jeblair | clarkb, fungi: i'd running do-release-upgrade on the test list server now | 18:01 |
| clarkb | jeblair: awesome, did we want to plan for an ext4 upgrade afterwards as well? | 18:02 |
| clarkb | or maybe just a mount ext3 as ext4? | 18:02 |
| clarkb | (that isn't as beneficial but does help) | 18:02 |
| jeblair | clarkb: i was thinking of doing the upgrade | 18:02 |
| jeblair | should be easy | 18:03 |
| clarkb | kk | 18:03 |
| jeblair | (just a couple of tune2fs option iirc) | 18:03 |
| openstackgerrit | Lubosz Kosnik (diltram) proposed openstack-infra/project-config master: Enable Octavia gate with Centos amphora https://review.openstack.org/449741 | 18:03 |
| jeblair | (heh, it's been a while) | 18:03 |
| *** jcoufal has quit IRC | 18:03 | |
| clarkb | ya its pretty minor but trying to rmember if you have to unmount which might be the tricky part | 18:03 |
| *** EricGonczer_ has joined #openstack-infra | 18:04 | |
| *** sshnaidm|off has quit IRC | 18:06 | |
| mtreinish | SamYaple: cool | 18:06 |
| jeblair | clarkb, fungi: we currently have en,fr,it,ko,ru,vi,zh_TW enabled in mailman | 18:07 |
| jeblair | i don't know how we got to that set. should we keep that set for the upgrade, or throw all of the languages it supports in? | 18:07 |
| *** baoli has joined #openstack-infra | 18:07 | |
| *** beslemon has quit IRC | 18:08 | |
| clarkb | jeblair: I believe that when you add a new ml in a non default lang you can specify the lang and it adds it? | 18:08 |
| jeblair | that makes sense. that probably roughly corresponds to our regional mailing lists | 18:08 |
| jeblair | so it's probably okay to just copy that list | 18:08 |
| *** evgenyl has quit IRC | 18:08 | |
| clarkb | ya we have fr and it and so on. So guessing moderators set that in their config and it does the correct install required/setup | 18:08 |
| jeblair | i think they're all installed on the system, and then just copied to /etc as needed | 18:09 |
| jeblair | they are the same filesystem for us, so it doesn't matter. | 18:09 |
| *** thiagolib has joined #openstack-infra | 18:09 | |
| jeblair | i'll just preserve the existing list for now. | 18:09 |
| *** evgenyl has joined #openstack-infra | 18:09 | |
| SpamapS | martinda: what's LSF? | 18:11 |
| jeblair | clarkb: i'm keeping a log of what i'm doing at https://etherpad.openstack.org/p/lists.o.o-trusty-upgrade | 18:12 |
| *** beslemon has joined #openstack-infra | 18:13 | |
| *** mtanino has joined #openstack-infra | 18:13 | |
| *** pvaneck has joined #openstack-infra | 18:14 | |
| *** pvaneck has quit IRC | 18:14 | |
| *** pvaneck has joined #openstack-infra | 18:14 | |
| *** jkilpatr has joined #openstack-infra | 18:16 | |
| *** beslemon_ has joined #openstack-infra | 18:17 | |
| *** d0ugal has quit IRC | 18:17 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config master: Enable firehose reporting on subunit workers https://review.openstack.org/449630 | 18:19 |
| *** e0ne has quit IRC | 18:19 | |
| *** beslemon has quit IRC | 18:19 | |
| *** esikachev has quit IRC | 18:19 | |
| *** yamamoto has quit IRC | 18:19 | |
| jeblair | clarkb: https://debian-administration.org/article/643/Migrating_a_live_system_from_ext3_to_ext4_filesystem seems like good instructions which correspond favorably with https://ext4.wiki.kernel.org/index.php/UpgradeToExt4 | 18:22 |
| clarkb | single user mode! does that even exist anymore? (but cool there are steps to do this) | 18:23 |
| jeblair | clarkb: i might just try it over ssh. :) | 18:24 |
| *** chlong has quit IRC | 18:24 | |
| *** e0ne has joined #openstack-infra | 18:25 | |
| *** chem is now known as chem_off | 18:25 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config master: Start firehose doc section for client on port 1883 https://review.openstack.org/449773 | 18:27 |
| mtreinish | clarkb, fungi: ^^^ | 18:27 |
| mtreinish | SamYaple: ^^^ if you wanted to write up a patch to add the python example to that section | 18:27 |
| *** nadya has quit IRC | 18:27 | |
| * mtreinish wonders if that's the first haskell in an openstack repo | 18:28 | |
| *** MasterOfBugs has quit IRC | 18:28 | |
| *** ociuhandu has quit IRC | 18:29 | |
| *** jcoufal has joined #openstack-infra | 18:29 | |
| openstackgerrit | Thiago da Silva proposed openstack-infra/project-config master: Test pyeclib against libec master https://review.openstack.org/449774 | 18:30 |
| *** esikachev has joined #openstack-infra | 18:30 | |
| tdasilva | AJaeger, jeblair ^^^ just gave a shot with the above, but i'm not sure if it's 100% correct, any help would be appreciated | 18:31 |
| *** jcoufal__ has quit IRC | 18:31 | |
| clarkb | mtreinish: you are writing haskell now? I had to fix my xmonad config recently and it made me want to deal with more haskell. Then I read the C to Rust compiler written in hasekll and really wanted more haskell | 18:32 |
| mtreinish | clarkb: heh, that's the first haskell I've ever written | 18:33 |
| clarkb | mtreinish: qosGranted is necessary? | 18:33 |
| mtreinish | I was playing reduce the code from the in repo example, it might not be needed | 18:33 |
| clarkb | mtreinish: cool! | 18:33 |
| *** sshnaidm|off has joined #openstack-infra | 18:34 | |
| clarkb | mtreinish: you might need ti for a side effect, but I think typically you'd make the result anonymous if that was the case for readbility but its been many years sicne I wrote any real haskell | 18:34 |
| *** esikachev has quit IRC | 18:34 | |
| SamYaple | mtreinish: https://review.openstack.org/#/c/449711/ i had that <-- | 18:35 |
| SamYaple | but it failed the gate soooo | 18:35 |
| *** e0ne has quit IRC | 18:35 | |
| *** ldnunes has quit IRC | 18:35 | |
| mtreinish | SamYaple: oh, I'll rebase my patch on that | 18:36 |
| *** e0ne has joined #openstack-infra | 18:37 | |
| SamYaple | mtreinish: SamYaple doesn't do docs. so you tell me what to do and ill do it | 18:37 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config master: Start firehose doc section for client on port 1883 https://review.openstack.org/449773 | 18:40 |
| mtreinish | SamYaple: it looks like the jessie puppet apply job doesn't work | 18:41 |
| mtreinish | it's failing the same way on another one of my patches | 18:41 |
| *** tosky has quit IRC | 18:43 | |
| AJaeger | fungi, the jeepby change merged 80 mins ago but I still see all deb- repos on codesearch. | 18:44 |
| *** MasterOfBugs has joined #openstack-infra | 18:46 | |
| *** ldnunes has joined #openstack-infra | 18:49 | |
| AJaeger | tdasilva: commented. You used the shell snippet for setting up a devstack-gate job but do not use the other builders for setting up the rest of a devstack-gate job. | 18:50 |
| *** esikachev has joined #openstack-infra | 18:52 | |
| tdasilva | AJaeger: how could I go about it *without* using a devstack setup? | 18:54 |
| tdasilva | AJaeger: any pointers I can look at? | 18:54 |
| AJaeger | tdasilva: so, what exactly do you want to do? Which job is your base and which parameter to change? | 18:57 |
| *** esikachev has quit IRC | 18:57 | |
| *** jcoufal_ has joined #openstack-infra | 18:57 | |
| tdasilva | AJaeger: I'd like to test pyeclib. if you look at pyeclib repo, in the bindep.txt we add a dependency on libec, so we get whatever version of libec that nodes' OS has packaged | 18:58 |
| tdasilva | we would like to test against libec master, so I need to fist checkout libec, build it and then run pyeclib tests | 18:58 |
| *** psachin has quit IRC | 18:58 | |
| *** jcoufal has quit IRC | 18:58 | |
| openstackgerrit | Dave McCowan proposed openstack-infra/project-config master: Adding Python3 Functional Test Gate for Barbican https://review.openstack.org/449343 | 18:59 |
| *** e0ne has quit IRC | 18:59 | |
| *** sshnaidm|off has quit IRC | 18:59 | |
| *** zxiiro has quit IRC | 19:00 | |
| *** petems has quit IRC | 19:00 | |
| *** esikachev has joined #openstack-infra | 19:00 | |
| AJaeger | tdasilva: so, in the shell section, write up the commands you want to run. | 19:01 |
| AJaeger | And for checking out the code: see gate-hacking-integration-{target-project} | 19:01 |
| AJaeger | http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/hacking.yaml#n12 | 19:01 |
| AJaeger | That zuul-cloner invocation checks out two repos - hacking as well as {target-org}/{target-project} . Follow that example to check out the two repos you have. | 19:02 |
| AJaeger | and then do your shell code to build the first and test the other | 19:02 |
| fungi | okay, i'm sort of back from lunch but working from a car over wireless modem so it's taking me time to catch up | 19:02 |
| *** zxiiro has joined #openstack-infra | 19:03 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool feature/zuulv3: Remove mention of non-clouds.yaml from docs https://review.openstack.org/449781 | 19:03 |
| tdasilva | AJaeger: oh, ok, so I will have to explicitly checkout the two repos? but how can I chech against a specific patch of pyeclib? | 19:05 |
| *** EricGonczer_ has quit IRC | 19:05 | |
| AJaeger | tdasilva: if you use zuul-cloner like in that example, it does all the magic for you ;) | 19:05 |
| AJaeger | tdasilva: even depends-on will work. | 19:05 |
| *** jordanP has joined #openstack-infra | 19:05 | |
| fungi | jeblair: slow i/o might be the result of wherever they schedule the rootfs volumes for clones of legacy instances | 19:05 |
| tdasilva | neat! | 19:05 |
| fungi | but who knows really | 19:06 |
| fungi | ext4 migration sounds worthwile | 19:06 |
| clarkb | mordred: did you want to switch the elif to an if in https://review.openstack.org/#/c/279818/6 and get that in then push a release | 19:06 |
| clarkb | mordred: its friday though may not be best time for glean users? | 19:06 |
| clarkb | happy to get that change in though | 19:07 |
| *** ilpianista has left #openstack-infra | 19:07 | |
| fungi | jeblair: i'm okay with increasing the language support list too. guessing the "old" list was a complete list of what was supported by some (perhaps much) older mm release | 19:08 |
| *** ilpianista has joined #openstack-infra | 19:08 | |
| fungi | or clarkb's theory makes sense too | 19:08 |
| *** esikachev has quit IRC | 19:15 | |
| openstackgerrit | Thiago da Silva proposed openstack-infra/project-config master: Test pyeclib against libec master https://review.openstack.org/449774 | 19:17 |
| *** erikmwilson has quit IRC | 19:17 | |
| *** yamamoto has joined #openstack-infra | 19:17 | |
| *** reed has quit IRC | 19:17 | |
| *** YorikSar has quit IRC | 19:18 | |
| *** YorikSar has joined #openstack-infra | 19:18 | |
| *** erikmwilson has joined #openstack-infra | 19:19 | |
| *** reed has joined #openstack-infra | 19:19 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool feature/zuulv3: WIP Add ability to select flavor by name or id https://review.openstack.org/449784 | 19:24 |
| tdasilva | AJaeger: does zuul-cloner take care of installing requirements defined in bindep.txt? | 19:24 |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci master: Combine HA and non-ha features https://review.openstack.org/449785 | 19:24 |
| *** jkilpatr has quit IRC | 19:24 | |
| *** yamamoto has quit IRC | 19:25 | |
| tdasilva | AJaeger: from your comment, I understood that zuul-git-prep would take care of the pyeclib bindep.txt, but I'm concerned about the libec bindep.txt | 19:25 |
| clarkb | tdasilva: no, bindep is a separate step independent of zuul-cloner | 19:26 |
| mordred | clarkb: I can - although I released this morning already (we can release again of course) | 19:26 |
| clarkb | mordred: oh I didn't catch that, nevermind then | 19:26 |
| mordred | clarkb: yah - we need to trigger an image rebuild - but also I suppose at this point we can just wait until tomorrow | 19:27 |
| *** jcoufal_ has quit IRC | 19:28 | |
| AJaeger | tdasilva: no, zuul-cloner is like "git clone" but knows about your patch | 19:29 |
| *** tosky has joined #openstack-infra | 19:29 | |
| *** jcoufal has joined #openstack-infra | 19:29 | |
| AJaeger | tdasilva: what you currently do is: zuul-git-prep checks out your repo; then install-distro-packages installs packages and uses - if you have one - your bindep.txt for it - then you call zuul-cloner to check out two repos - again the first repo and another one. So, I wonder whether we can avoid the double check out | 19:30 |
| tdasilva | AJaeger: yeah, i was going to try removing the second checkout of pyeclib, but also will need to call bindep for libec... | 19:31 |
| mtreinish | clarkb, jeblair, mordred, fungi, AJaeger: any idea what's up with the puppet apply jessie job? It seems to be failing all the system-config patches I've been working on today | 19:32 |
| openstackgerrit | Thiago da Silva proposed openstack-infra/project-config master: Test pyeclib against libec master https://review.openstack.org/449774 | 19:32 |
| clarkb | mtreinish: no, have a link? | 19:33 |
| mtreinish | clarkb: http://logs.openstack.org/73/449773/2/check/gate-infra-puppet-apply-3-debian-jessie/fd03380/puppetapplytest59.final.out.FAILED | 19:33 |
| mtreinish | which is from my doc patch: https://review.openstack.org/#/c/449773/ | 19:33 |
| openstackgerrit | Thiago da Silva proposed openstack-infra/project-config master: Test pyeclib against libec master https://review.openstack.org/449774 | 19:34 |
| openstackgerrit | Dan Prince proposed openstack-infra/project-config master: Combine the TripleO nonha and ha jobs https://review.openstack.org/449791 | 19:35 |
| scarab_ | someone gotta second? to spare i gave a question about the periodic pipeline.....The periodic pipeline doesn’t have any notification triggers on it. Is there a way to modify the pipeline to where it notifies the PTL on failures, either through email or irc channels? | 19:35 |
| mtreinish | scarab_: you can get an rss feed of failures via openstack-health | 19:36 |
| clarkb | you can also have emails but we removed them because iirc they were ignored | 19:36 |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/zuul feature/zuulv3: Add github reporter status_url config option https://review.openstack.org/449794 | 19:38 |
| mtreinish | for example: http://health.openstack.org/runs/key/build_name/periodic-tempest-dsvm-neutron-full-ssh-master/recent/rss | 19:38 |
| clarkb | mtreinish: we've had that jessie job running since at least decemeber looks like | 19:39 |
| mtreinish | heh, well it's not working now | 19:40 |
| mtreinish | do we actually deploy anything on jessie? | 19:40 |
| pabelanger | yes | 19:40 |
| pabelanger | I wonder if puppet module path has changed | 19:41 |
| clarkb | and thats the only place we run jessie apply tests | 19:42 |
| clarkb | (was just checking if it worked anywhere else) | 19:42 |
| *** esikachev has joined #openstack-infra | 19:42 | |
| pabelanger | I can hold a node and see what is going on | 19:42 |
| *** kzaitsev_mb has joined #openstack-infra | 19:43 | |
| scarab_ | mtreinish_: I was looking there and the openstack-ansible feed is not appearing so they have no way to check the upgrade-ansible-aio periodic test. | 19:43 |
| clarkb | its weird that its only jessie | 19:44 |
| *** dprince has quit IRC | 19:44 | |
| openstackgerrit | Mikhail S Medvedev proposed openstack/diskimage-builder master: Create PReP boot partition for PPC https://review.openstack.org/447739 | 19:44 |
| *** jcoufal_ has joined #openstack-infra | 19:46 | |
| mtreinish | scarab_: I was talking to odyssey4me about that a couple of weeks ago. The jobs need to emit subunit for it to be picked up by openstack-health | 19:46 |
| *** jcoufal has quit IRC | 19:47 | |
| *** gcb has quit IRC | 19:49 | |
| *** florianf has quit IRC | 19:49 | |
| clarkb | pabelanger: we don't seem to set the module path explicitly anywhere that I can find so it may be relying on ansible defaults? | 19:49 |
| openstackgerrit | Eric Harney proposed openstack-infra/project-config master: Enable Barbican for Cinder LIO job https://review.openstack.org/449799 | 19:50 |
| pabelanger | $ puppet config print modulepath | cut -d ':' -f 1 | 19:51 |
| pabelanger | /home/jenkins/.puppet/modules | 19:51 |
| pabelanger | that is jessie for some reason | 19:51 |
| pabelanger | let me see where we installed them | 19:51 |
| clarkb | the main modules are all in /etc/puppet/modules but its failing at finding system-config/modules not sure yet where that is expected to be | 19:52 |
| clarkb | home/jenkins/workspace/gate-infra-puppet-apply-3-debian-jessie/openstack-infra/system-config/modules is where they should be based on the job log | 19:52 |
| pabelanger | that is right | 19:53 |
| pabelanger | where we zuul-cloned the job | 19:53 |
| *** annegentle has quit IRC | 19:53 | |
| clarkb | pabelanger: http://logs.openstack.org/73/449773/2/check/gate-infra-puppet-apply-3-debian-jessie/fd03380/console.html#_2017-03-24_18_46_04_637817 and then later http://logs.openstack.org/73/449773/2/check/gate-infra-puppet-apply-3-debian-jessie/fd03380/console.html#_2017-03-24_18_46_06_949971 | 19:53 |
| pabelanger | ya | 19:54 |
| *** petems has joined #openstack-infra | 19:54 | |
| pabelanger | let me see what ansible-role-puppet is doing | 19:54 |
| clarkb | so I think puppet is ignoring MODULE_PATH | 19:54 |
| clarkb | pabelanger: well thats happening outside of ansible | 19:54 |
| pabelanger | Hmm | 19:55 |
| pabelanger | we install our modules into /etc/puppet/modules | 19:55 |
| pabelanger | but system-config isn't | 19:56 |
| clarkb | ya we use the module path to point at the cloned repo | 19:56 |
| clarkb | see tools/pre-apply-test.sh | 19:56 |
| clarkb | er tools/prep-apply.sh | 19:56 |
| pabelanger | right, but then we run ansible | 19:57 |
| clarkb | pabelanger: lines 76-80 appear to be where we do the thing that attempts to tie ansible to the system-config modules | 19:59 |
| clarkb | we do a symlink to /opt/system-config/production | 19:59 |
| clarkb | http://logs.openstack.org/73/449773/2/check/gate-infra-puppet-apply-3-debian-jessie/fd03380/console.html#_2017-03-24_18_47_03_437454 shows that happening in the log | 20:00 |
| pabelanger | ya | 20:00 |
| pabelanger | that exists | 20:00 |
| pabelanger | I'm going to run ansible again | 20:01 |
| pabelanger | and enable some verbose | 20:01 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool feature/zuulv3: Encode exclusivity into voluptuous https://review.openstack.org/449801 | 20:01 |
| *** hashar has joined #openstack-infra | 20:03 | |
| *** dprince has joined #openstack-infra | 20:03 | |
| jeblair | clarkb, fungi: it took 2 hours to do-release-upgrade | 20:03 |
| jeblair | i think most of that was spent dealing with all the installed kernel packages | 20:04 |
| jeblair | so if we remove them all before starting, it may go much faster. | 20:04 |
| clarkb | pabelanger puppet_basemodulepath: '$confdir/modules' is the ansible module default | 20:04 |
| clarkb | jeblair: are you wanting to test that on another boot off the snapshot? | 20:05 |
| fungi | jeblair: oh, absolutely we should remove all but the current kernel package | 20:05 |
| clarkb | jeblair: but ya thats a pretty easy apt-get autoremove command | 20:05 |
| jeblair | it did that n! thing where it removes one package, then does a bunch of grub stuff. | 20:05 |
| pabelanger | clarkb: is what the jobs defaults too modulepath = '$confdir/modules:/opt/system-config/production/modules' | 20:05 |
| fungi | jeblair: it's all thanks to initrd generation in postinst triggers | 20:05 |
| clarkb | pabelanger: where do you see that? | 20:05 |
| jeblair | fungi: yep that's it | 20:05 |
| *** burgerk has quit IRC | 20:05 | |
| pabelanger | clarkb: /etc/puppet/environments/production/environment.conf | 20:05 |
| *** spzala has quit IRC | 20:05 | |
| pabelanger | ansible-role-puppet sets that up | 20:06 |
| jeblair | clarkb, fungi: anything i should check before i reboot? | 20:06 |
| clarkb | pabelanger: ok that should be the correct path then /etc/puppet/modules is $confir/modules | 20:06 |
| *** spzala has joined #openstack-infra | 20:06 | |
| clarkb | jeblair: I've never had a problem with autoremove... | 20:06 |
| fungi | jeblair: double-check that things you didn't want reenabled (mailman daemon?) are still disabled i guess. though the spool cleanup should make that non-dangerous | 20:07 |
| jeblair | fungi: yep, that looks good | 20:07 |
| jeblair | exim/mailman/bug are all still disabled | 20:07 |
| jeblair | bup even | 20:07 |
| clarkb | pabelanger: so that makes it even more confusing, what version of puppet and is it new enough to respect environments? | 20:08 |
| fungi | clarkb: i've had trouble with autoremove in old systems which have been iteratively upgraded and somehow some packages i was relying on ended up as auto-installed so got marked for autoremoval when something else that depended on them got removed | 20:08 |
| fungi | but it's rare | 20:08 |
| jeblair | rebooting now | 20:08 |
| fungi | and i think there was an apt bug at one point which was responsible for the majority of that | 20:09 |
| fungi | so probably no longer an issue | 20:09 |
| mordred | fungi: yup. I remember a similar thing happening to me ... but I think it was maybe 8 or 10 years ago? | 20:09 |
| jeblair | it's back up | 20:09 |
| fungi | mordred: possibly. also actually looking at the package list autoremove gives you avoids pretty much all the problem | 20:09 |
| clarkb | pabelanger: I'm gonna go grab lunch now but let me know if you find anything | 20:09 |
| clarkb | jeblair: that was fast | 20:10 |
| fungi | clouds | 20:10 |
| *** spzala has quit IRC | 20:10 | |
| pabelanger | clarkb: ack | 20:10 |
| jeblair | Welcome to Ubuntu 14.04.5 LTS (GNU/Linux 3.2.0-123-virtual x86_64) | 20:10 |
| clarkb | jeblair: nice! | 20:11 |
| clarkb | though that kernel seems old | 20:11 |
| clarkb | 3.13 is trusty | 20:11 |
| jeblair | yeah, that was the kernel it was running before | 20:11 |
| jeblair | The following packages have been kept back: linux-headers-virtual linux-image-virtual | 20:12 |
| *** jcoufal_ has quit IRC | 20:12 | |
| mordred | I'm impressed it operates on the old kernel | 20:12 |
| jeblair | okay, so i *thought* i selected en for the default language: http://lists-upgrade-test.openstack.org/cgi-bin/mailman/listinfo | 20:12 |
| *** eharney has quit IRC | 20:12 | |
| mordred | jeblair: that's the new english | 20:13 |
| *** rbuzatu_ has joined #openstack-infra | 20:13 | |
| fungi | probably need apt-get dist-upgrade to get those kernel packages? | 20:13 |
| jeblair | i'm going to apt-get dist-upgrade now | 20:14 |
| jeblair | yeah | 20:14 |
| fungi | indeed, that | 20:14 |
| jeblair | it says it will install them | 20:14 |
| clarkb | odd it kept them back | 20:14 |
| jeblair | now i will dpkg-reconfigure mailman | 20:15 |
| scarab_ | mtreinish: The subunit stream is genereated its just moved to a tmp folder on another host. Is there a way to email the PTL of a project without notifying every other PTL of the respective projects? | 20:15 |
| *** spzala has joined #openstack-infra | 20:15 | |
| *** annegentle has joined #openstack-infra | 20:16 | |
| fungi | scarab_: zuul's e-mail reporter uses a fixed e-mail address per pipeline | 20:16 |
| fungi | so we traditionally set that to a specific mailing list (e.g. quality assurance or stable branch management) | 20:16 |
| *** rbuzatu has quit IRC | 20:16 | |
| jeblair | http://lists-upgrade-test.openstack.org/cgi-bin/mailman/listinfo looks more english now. | 20:17 |
| jeblair | maybe i missed the target in the magenta-on-hot-pink dialog box. | 20:17 |
| jeblair | i'm going to reboot and see if the kernel is happy. | 20:17 |
| jeblair | Welcome to Ubuntu 14.04.5 LTS (GNU/Linux 3.13.0-113-generic x86_64) | 20:18 |
| *** annegent_ has joined #openstack-infra | 20:19 | |
| pabelanger | clarkb: okay, so if I use --modulepath on the CLI, it works | 20:19 |
| pabelanger | but we don't set that in ansible-role-puppet | 20:19 |
| *** luzC has joined #openstack-infra | 20:19 | |
| pabelanger | so, something has changed with loading the /etc/puppet/environments/production/environment.conf on jessie | 20:20 |
| *** thiagolib has quit IRC | 20:21 | |
| *** HeOS has joined #openstack-infra | 20:21 | |
| *** hashar has quit IRC | 20:22 | |
| fungi | that kernel lgtm | 20:22 |
| clarkb | ya is puppet new enough to support environments? | 20:22 |
| *** annegentle has quit IRC | 20:22 | |
| *** gouthamr has quit IRC | 20:23 | |
| pabelanger | 3.7.2 | 20:23 |
| *** yamamoto has joined #openstack-infra | 20:23 | |
| clarkb | pretty sure thats new enough | 20:24 |
| pabelanger | we can try rolling back nodepool image for jessie | 20:25 |
| pabelanger | currently looking at DIB differences | 20:25 |
| *** amoralej is now known as amoralej|off | 20:25 | |
| *** hashar has joined #openstack-infra | 20:25 | |
| luzC | fungi: so about scarab_ question, can e-mail reporter be used on a single job instead of the pipeline? | 20:26 |
| fungi | luzC: no, zuul reporters are pipeline-specific | 20:26 |
| pabelanger | Hmm | 20:26 |
| pabelanger | jessie-backports/main puppet all 3.8.5-2~bpo8+1 [26.3 kB] | 20:27 |
| pabelanger | was yesterday | 20:27 |
| pabelanger | jessie/main puppet all 3.7.2-4 [25.7 kB] | 20:28 |
| pabelanger | today | 20:28 |
| pabelanger | we some how changed puppet version | 20:28 |
| pabelanger | backwards at that | 20:28 |
| fungi | scarab_: luzC: this is what the failure reporter for the periodic-stable pipeline looks like, for example: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n184 | 20:28 |
| *** armax has joined #openstack-infra | 20:29 | |
| pabelanger | clarkb: Oh, so this is likely our issue | 20:29 |
| pabelanger | https://packages.debian.org/jessie-backports/puppet | 20:29 |
| mtreinish | scarab_: just put the subunit in the log dir and the problem solves itself. I don't understand what's so difficult about that | 20:29 |
| fungi | pabelanger: that version doesn't match any of the packages in debian anyway | 20:29 |
| pabelanger | clarkb: puppet 4 is now in jessie-backports | 20:29 |
| pabelanger | and I think we pin to puppet 3 | 20:29 |
| luzC | fungi yes, we look at it... so scarab_ is trying to notify a ptl when a osa periodic job fails... via email, irc bot, something... from pipeline itself it won't be possible since reporter expect a fixed email address... and unable to do it through the job itself | 20:29 |
| pabelanger | and are now skipping backports | 20:29 |
| fungi | pabelanger: yep, 4.8.2-3~bpo8+1 | 20:29 |
| mtreinish | scarab_: why would you generate a subunit stream and not put it in the job artifacts? | 20:30 |
| luzC | mtreinish I'm doing that, moving the subunit file to the right place (WIP) | 20:31 |
| pabelanger | fungi: clarkb: so, we should roll back jessie for now, and pause builds, until we can find where to get puppet 3 now | 20:31 |
| luzC | mtreinish still it needs someone to look at it and we want just a notification to let them know when job is broken | 20:31 |
| mtreinish | luzC: when that's done subunit2sql will pick up periodic jobs and you can get rss feeds for all your periodic jobs | 20:31 |
| clarkb | pabelanger: arent we getting 3 though? | 20:31 |
| mtreinish | luzC: well that's the problem with periodic jobs, someone has to look at them | 20:31 |
| clarkb | is it just not new enough for environments? | 20:32 |
| mtreinish | which rarely happens... | 20:32 |
| fungi | pabelanger: yeah, or we could pin the puppet packages from stable instead of stable-backports (installing everything from backports unconditionally is generally a terrible idea anyway) | 20:32 |
| pabelanger | clarkb: ya, I think we have rolled back to a version that doesn't work with ansible-role-puppet | 20:32 |
| clarkb | gotcha we could also use 4 on jessie like we do on fedora | 20:32 |
| clarkb | it shouod work since it works on fedora | 20:33 |
| pabelanger | in fact, I don't know why we have backports enabled on jessie | 20:33 |
| pabelanger | I think this is what dib does by default | 20:33 |
| pabelanger | we could just not enable it | 20:33 |
| fungi | pabelanger: that sounds broken to me | 20:33 |
| clarkb | if we dont enable backports we continue to get broken 3 though right? | 20:33 |
| *** yamamoto has quit IRC | 20:33 | |
| clarkb | so thats orthogonal I think | 20:33 |
| fungi | pabelanger: odds are some things are needed from bpo, but those should be job-specific and added selectively. maybe zigo knows why that's happening | 20:34 |
| pabelanger | fungi: ya, for some reason debian-minimal defaults to that | 20:34 |
| jeblair | clarkb, fungi: if set an entry for lists.openstack.org with the new ip in my local hosts file, mailman web ui seems to work. | 20:34 |
| pabelanger | clarkb: the odd thing is, we do install the puppetlabs apt repo | 20:34 |
| jeblair | clarkb, fungi: (i verified i'm accessing the right server by tailing logs) | 20:34 |
| fungi | jeblair: testing that myself | 20:34 |
| pabelanger | and we are still pulling from debian | 20:34 |
| pabelanger | clarkb: I'll have to debug more, but not sure today is the day | 20:34 |
| luzC | mtreinish yes but a remainder (only if broken) would be nice to have... :) we will drop the notification and see how it goes with just the subunit part first | 20:34 |
| clarkb | pabelanger: ya lets make job nonvoting | 20:35 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/system-config master: Correct Logstash Query Example https://review.openstack.org/257501 | 20:35 |
| mtreinish | luzC: that's what openstack health's rss feed gives you... | 20:35 |
| pabelanger | clarkb: or that | 20:36 |
| clarkb | then we xan iterate more easily | 20:36 |
| pabelanger | ya, we can keep an eye on nodepool-builders too | 20:37 |
| luzC | mtreinish: ahhh, good to know, thank you... scarab_: will keep you posted | 20:37 |
| pabelanger | so far, it built | 20:37 |
| clarkb | pabelanger: it doesnt use ansible or rely on environments iirc | 20:37 |
| clarkb | it just directly puppet applies so is simpler | 20:38 |
| pabelanger | ya | 20:38 |
| pabelanger | so a little different process for builds | 20:38 |
| *** aNuposic has joined #openstack-infra | 20:39 | |
| *** trown is now known as trown|outtypewww | 20:41 | |
| openstackgerrit | Dan Prince proposed openstack-infra/project-config master: Combine the TripleO nonha and ha jobs https://review.openstack.org/449791 | 20:43 |
| fungi | jeblair: yep, after solving some editor default stupidity on my workstation, i too have successful browsing of archives from the test upgrade (confirmed i'm hitting the right server using tcpdump) | 20:44 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Switch puppet-apply-3 for debian-jessie non voting https://review.openstack.org/449812 | 20:44 |
| fungi | also my list preference logins still work on it | 20:45 |
| fungi | as does admin/moderator access to lists where i have it | 20:45 |
| fungi | and all confirmed over ipv6, not that i expected any issue there | 20:46 |
| jeblair | fungi: do we want to make a new test list, start up exim/mailman and send it a message, or do we want to trust all that will work? | 20:46 |
| fungi | i'm pretty comfortable already, but if others are interested in a more in-depth test drive i won't object | 20:47 |
| *** erikmwilson has quit IRC | 20:48 | |
| *** erikmwilson has joined #openstack-infra | 20:48 | |
| fungi | it seems to have kept/migrated what was in the archive trees and config picklefiles just fine, so the only unconfirmed bit really is exim mailrouting integration | 20:48 |
| jeblair | i'm comfortable with it too, but if someone wants, i'll be happy to do the test. | 20:49 |
| jeblair | clarkb, pabelanger, mordred: ^ ? | 20:49 |
| fungi | unless you modified exim's configuration, new versions are extremely great at config backward-compatibility | 20:49 |
| *** jcoufal has joined #openstack-infra | 20:49 | |
| clarkb | I'm definitely the noob when it comes to mailman and exim. I'm happy to defer to your judgement there. Generally though testing as mucha s possible is nice | 20:50 |
| *** electrical has quit IRC | 20:50 | |
| *** rfolco has quit IRC | 20:51 | |
| *** electrical has joined #openstack-infra | 20:51 | |
| jeblair | yeah, i didn't change the exim config. and we were running exim4 before. | 20:52 |
| pabelanger | same, my email administration skills are pretty low | 20:52 |
| *** matrohon has joined #openstack-infra | 20:52 | |
| jeblair | i think in testing we'd probably run into some hostname mismatch issues, which is part of my reluctance. | 20:53 |
| jeblair | (though i'm not certain :) | 20:53 |
| jeblair | i could try and see how far we get | 20:53 |
| *** yolanda has quit IRC | 20:54 | |
| fungi | yeah, it might be as simple as just recursively grepping for lists.openstack.org in /etc and switching those out for its real fqdn, then restarting a few services | 20:55 |
| fungi | but it is extra work and definitely involves modifying configuration | 20:55 |
| *** baoli has quit IRC | 20:55 | |
| jeblair | fungi: well, mailman gets picky about it; the list database files have the hostnames encoded in them | 20:55 |
| fungi | oh, eep. so would need to tweak the picklefiles too i guess (using the mm editor cli/lib) | 20:56 |
| jeblair | so if we get to the point where we have to update those to make it work, then i'd say we stop because the test is bonkers. :) | 20:56 |
| *** yolanda has joined #openstack-infra | 20:56 | |
| *** baoli has joined #openstack-infra | 20:58 | |
| jeblair | newlist --urlhost=lists.openstack.org --emailhost=lists.openstack.org testlist | 20:58 |
| jeblair | i ran that to create the new list | 20:59 |
| jeblair | and i got email notification of its creation (since i'm the owner) | 20:59 |
| fungi | your mta presumably is okay that reverse dns doesn't match the helo/ehlo | 21:00 |
| fungi | which, granted, would be unusually picky | 21:00 |
| jeblair | fungi: well, actually it does. it helo'd as lists-test-upgrade | 21:00 |
| fungi | aha, thanks exim! | 21:01 |
| *** dave-mccowan has quit IRC | 21:01 | |
| fungi | so maybe it's on the inbound side this testing would get tricky | 21:01 |
| *** jamesdenton has quit IRC | 21:01 | |
| jeblair | echo "corvus@inaugust.com" | add_members -r - testlist | 21:01 |
| jeblair | Subscribed: corvus@inaugust.com | 21:01 |
| jeblair | anyone else? | 21:01 |
| clarkb | you can add me if you want | 21:02 |
| jeblair | i just got the subscription notification | 21:02 |
| fungi | i'll test subscribing via webui | 21:02 |
| clarkb | or I can edit /etc/hosts and sign myself up I guess /em does this | 21:02 |
| fungi | though confirming might get tricky | 21:02 |
| * jeblair refrains from adding anyone | 21:02 | |
| *** dimtruck is now known as zz_dimtruck | 21:03 | |
| jeblair | fungi: well, with a hosts entry, should just work i think? | 21:03 |
| fungi | e-mail replying to the confirmation e-mail i mean | 21:03 |
| jeblair | fungi: yeah, though you can web-confirm | 21:04 |
| fungi | oh, right | 21:04 |
| *** e0ne has joined #openstack-infra | 21:04 | |
| clarkb | there are not public lists (but I managed to hit testlist direclty0 | 21:04 |
| *** yolanda has quit IRC | 21:04 | |
| jeblair | clarkb: it works better with a local /etc/hosts entry | 21:04 |
| clarkb | jeblair: for lists.o.o? | 21:04 |
| jeblair | clarkb: yes | 21:05 |
| jeblair | fungi: you have mail :) | 21:05 |
| *** dprince has quit IRC | 21:05 | |
| *** yolanda has joined #openstack-infra | 21:05 | |
| *** links has joined #openstack-infra | 21:05 | |
| fungi | and confirmed via url (didn't feel like tweaking my mailrouting) | 21:06 |
| fungi | so that all worked | 21:06 |
| clarkb | my fastmail webclient is going to respond to lists.o.o so confirming confirmation via email won't work | 21:06 |
| jeblair | clarkb: let me know when you have confirmed, then i will send a hand-crafted test message | 21:06 |
| fungi | i suppose i could have pretended to be an mta and confirmed that way | 21:06 |
| clarkb | well I didn't have /etc/hosts mangled when I tried confirming the first time so the redirect after confirming isn't working. Going to edit /etc/hosts | 21:08 |
| fungi | oh, inbound 25/tcp is timing out for me | 21:08 |
| jeblair | fungi: wfm | 21:09 |
| jeblair | i sent a test message | 21:09 |
| clarkb | I'm subscribed | 21:10 |
| clarkb | I got the test message | 21:10 |
| fungi | oho, i wonder if my isp is blocking me. i'll try from the cloud | 21:10 |
| *** kzaitsev_mb has quit IRC | 21:11 | |
| fungi | yep, that seems to have been the issue | 21:13 |
| fungi | manually spoke smtp back to the test server to confirm my subscription and that worked fine | 21:13 |
| fungi | so the control address routing seems to still be working too | 21:14 |
| *** ldnunes has quit IRC | 21:14 | |
| jeblair | i found my test message, fungi you should have a copy too | 21:14 |
| fungi | yep, received | 21:14 |
| *** scarab_ has quit IRC | 21:15 | |
| jeblair | that wasn't so bad after all :) | 21:15 |
| jeblair | mostly because fungi set up rdns. :) | 21:15 |
| * fungi has learnt that lesson many times over | 21:15 | |
| clarkb | heh the test message got sniped by spam filters after I initially raed it | 21:15 |
| fungi | it shows up in the list archives too http://lists.openstack.org/pipermail/testlist/2017-March/thread.html | 21:16 |
| jeblair | clarkb: ha, they have async spam filtering? neat. i guess that's "faster". :) | 21:16 |
| fungi | assuming you have an /etc/hosts entry to go to the test server for that fqdn | 21:16 |
| *** jcoufal_ has joined #openstack-infra | 21:16 | |
| *** pramodrj07 has joined #openstack-infra | 21:16 | |
| *** yolanda has quit IRC | 21:17 | |
| *** jcoufal has quit IRC | 21:17 | |
| *** baoli has quit IRC | 21:17 | |
| jeblair | should we try an ext4 upgrade now? | 21:17 |
| fungi | i think this is very well-tested at this point. do we want a second snapshot run (maybe after addressing the cleanup noted in the etherpad) to see what the upgrade outage timeframe is looking like before scheduling, or just give it a broad enough window we've okay with some fudge factor? | 21:18 |
| *** mattmceuen has quit IRC | 21:18 | |
| fungi | oh, right, filesystem upgrade | 21:18 |
| fungi | yes please! | 21:18 |
| clarkb | ++ | 21:18 |
| jeblair | okay, stopped exim/mm/apache | 21:19 |
| *** yolanda has joined #openstack-infra | 21:19 | |
| jeblair | i'm in screen if anyone wants to join | 21:19 |
| *** rbuzatu_ has quit IRC | 21:19 | |
| *** rfolco has joined #openstack-infra | 21:19 | |
| *** rfolco has quit IRC | 21:19 | |
| clarkb | do we think its safer if there are fewer connections? | 21:20 |
| fungi | attached | 21:20 |
| clarkb | (I'd join but the directions said single user :P ) | 21:20 |
| *** MasterOfBugs has quit IRC | 21:20 | |
| fungi | clarkb: i don't think so | 21:20 |
| *** rbuzatu has joined #openstack-infra | 21:20 | |
| clarkb | fwiw pretty sure upstart killed the concept of single user | 21:20 |
| jeblair | yeah, i don't think it will make it worse | 21:20 |
| *** jascott1 has quit IRC | 21:21 | |
| *** vgadiraj has quit IRC | 21:21 | |
| *** dharinic- has quit IRC | 21:21 | |
| *** aunnam has quit IRC | 21:21 | |
| fungi | my guess is that was either 1. to reduce what's running, or 2. so / can be mounted read-only (and the latter won't be possible over ssh regardless) | 21:21 |
| * clarkb watches | 21:21 | |
| jeblair | i guess we're about to find out that it can't. | 21:21 |
| *** aysyd has quit IRC | 21:21 | |
| fungi | so several people in a root screen session or one is going to be roughly the same impact | 21:21 |
| clarkb | ya | 21:21 |
| *** ihrachys has quit IRC | 21:21 | |
| jeblair | can we force that? | 21:22 |
| *** ihrachys has joined #openstack-infra | 21:22 | |
| clarkb | apparently you might be able to do it at the block device layer | 21:22 |
| fungi | oh, and i guess we can't set it to mount ext4 in fstab because xen is just gonna xen and ignore that? | 21:22 |
| clarkb | but then I don't think you can update the fs itself | 21:22 |
| jeblair | fungi: i don't understand the xen comment | 21:23 |
| fungi | i don't think xen is looking at the fstab before it mounts the rootfs (unless maybe it's actually using our initrd which has a copy of it)? | 21:24 |
| *** scarab_ has joined #openstack-infra | 21:24 | |
| *** gordc has quit IRC | 21:24 | |
| clarkb | it uses grub doesn't it? | 21:24 |
| clarkb | which should use our fstab I think | 21:24 |
| jeblair | yeah, i think that all changed with the pvm stuff a while back, now it's real grub | 21:24 |
| fungi | oh, good. i'm still remembering the bad old days | 21:24 |
| clarkb | back when calamity ganon first struck | 21:25 |
| *** rlandy has quit IRC | 21:25 | |
| fungi | xelda will be our next hypervisor | 21:25 |
| *** kzaitsev_mb has joined #openstack-infra | 21:25 | |
| *** yolanda has quit IRC | 21:26 | |
| fungi | so it is possible to do this with a "rescue mode" boot in rackspace's dashboard, since that boots from a separate block device | 21:26 |
| clarkb | fungi: yes booting off separate media and modifying the / should work | 21:26 |
| fungi | or booting into single-user mode from the virtual console | 21:26 |
| fungi | with / mounted read-only | 21:27 |
| clarkb | the other alternative is just accept that it won't be true ext4 but mount it ext4 | 21:27 |
| *** askb has joined #openstack-infra | 21:27 | |
| jeblair | the only remaining process with a writable file handle open on / is pid1 | 21:27 |
| fungi | can you convert it to real ext4 live after mounting an ext3 filesystem ext4? | 21:28 |
| *** ihrachys has quit IRC | 21:28 | |
| clarkb | fungi: I don't think so | 21:28 |
| clarkb | not without making it ro | 21:28 |
| jeblair | you know, pid1 didn't used to have open log file handles. | 21:28 |
| jeblair | just saying. | 21:28 |
| fungi | that would have been too convenient | 21:28 |
| jeblair | init 1 root 18w REG 202,1 380 90941 /var/log/upstart/acpid.log | 21:29 |
| jeblair | init 1 root 20w REG 202,1 244 91214 /var/log/upstart/systemd-logind.log | 21:29 |
| jeblair | any chance we can convince it to close those? | 21:29 |
| clarkb | I was going to say before anyone blames systemd its upstart but its using systemd logind :) | 21:29 |
| fungi | logind should be a separate daemon, as should acpid | 21:30 |
| clarkb | jeblair: acpi should be safe unless we get a really important please shut down nicely message from xen | 21:30 |
| clarkb | killing login might force us to reboot if we get kicked out? not sure if pam funnels ssh users through there | 21:30 |
| jeblair | oh hey, stopping acpid caused pid1 to close the file. neat. | 21:30 |
| *** zz_dimtruck is now known as dimtruck | 21:30 | |
| clarkb | jeblair: ya I think tahts upstart logging stdout/stderr for daemons | 21:30 |
| jeblair | so just that one process now... | 21:31 |
| fungi | logind i _think_ is mainly for taking care of adding permissions when users log on from a local console | 21:31 |
| jeblair | not quite sure how to stop it. | 21:31 |
| jeblair | service systemd-logind stop | 21:31 |
| jeblair | that did indeed work | 21:31 |
| jeblair | and we're read-only now :) | 21:31 |
| jeblair | let me catch up the etherpad | 21:31 |
| fungi | woo! | 21:32 |
| *** jcoufal has joined #openstack-infra | 21:32 | |
| *** yolanda has joined #openstack-infra | 21:32 | |
| clarkb | neat | 21:32 |
| *** jcoufal_ has quit IRC | 21:32 | |
| fungi | the fun things you do on remote headless systems during an upgrade | 21:32 |
| *** e0ne has quit IRC | 21:32 | |
| jeblair | that command look ok? | 21:33 |
| fungi | i don't think i've ever remounted / read-only without a reboot or telinit 1 or similar | 21:33 |
| *** kzaitsev_mb has quit IRC | 21:34 | |
| clarkb | jeblair: yes thats the correct filesystem according to fstab | 21:34 |
| fungi | yeah, lgtm. preen and force even if clean | 21:34 |
| clarkb | and -pf looks like the flags we want | 21:34 |
| clarkb | fix and force | 21:34 |
| jeblair | running | 21:34 |
| jeblair | i wonder if -v would have been a good addition. | 21:35 |
| *** pradk has quit IRC | 21:36 | |
| clarkb | ya probably, oh well | 21:36 |
| *** yolanda has quit IRC | 21:39 | |
| *** yolanda has joined #openstack-infra | 21:39 | |
| jeblair | done | 21:42 |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder master: Use stevedore for plugin config of block device https://review.openstack.org/447090 | 21:42 |
| jeblair | i'm going to skip the "it may be a good idea" reboot step and just continue | 21:43 |
| clarkb | ++ | 21:43 |
| fungi | er, yeah | 21:43 |
| clarkb | doing the ro dance more than once seems like an unnecessary pain | 21:44 |
| jeblair | tune2fs -O extents,uninit_bg,dir_index /dev/xvda1 look good? | 21:44 |
| fungi | and tempting fate ;) | 21:44 |
| clarkb | jeblair: its what debian says to do :) I haven't checked on what uninit_bg and dir_index are | 21:44 |
| clarkb | but extents are important aiui | 21:45 |
| fungi | i'm still reading manpage | 21:45 |
| clarkb | uninit_bg is a fsck optimization. | 21:45 |
| fungi | indeed, a nice looking speedup at that | 21:46 |
| clarkb | dir_index makes lookups faster in large dirs | 21:46 |
| jeblair | the uninit_bg, from what i've read, is basically 'deactivate the ext3 datastructures as they are no longer used'. adding it requireds that we run that fsck step which is next. | 21:46 |
| clarkb | I have a tune2fs manpage on the internets that says its "extent" not "extents" | 21:46 |
| *** jcoufal_ has joined #openstack-infra | 21:47 | |
| fungi | dir_index only seems to tage effect for new directories, unless you use tune2fs -D on existing ones | 21:47 |
| clarkb | also local man page | 21:47 |
| jeblair | hrm, the debian ad kernel wikis both says extents. | 21:47 |
| *** jcoufal has quit IRC | 21:47 | |
| fungi | my manpage doesn't even have extents | 21:48 |
| clarkb | fungi: does it have "extent" | 21:48 |
| clarkb | I'm guessing the debian page is correct | 21:48 |
| clarkb | since its first google result | 21:48 |
| fungi | yeah, manpage on the server and one of my systems both call that -O option "extent" (singular) | 21:48 |
| *** ssbarnea has quit IRC | 21:49 | |
| fungi | so either the manpage is wrong or multiple someones are wrong on the internet | 21:49 |
| fungi | i'd give it 50/50 odds | 21:50 |
| jeblair | i've never bet against a manpage before, but i want to this time. | 21:50 |
| fungi | can't hurt to try it | 21:50 |
| jeblair | it's both | 21:52 |
| fungi | whee! | 21:53 |
| jeblair | https://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git/tree/lib/e2p/feature.c#n84 | 21:53 |
| *** scarab_ has quit IRC | 21:53 | |
| fungi | it's a floor wax! it's a dessert topping! stop, you're both right! | 21:53 |
| clarkb | I wonder if they did that because the docs did it all one way and the manpage the other | 21:53 |
| jeblair | okay that's done. | 21:53 |
| clarkb | easier to just add another option | 21:53 |
| *** ssbarnea has joined #openstack-infra | 21:54 | |
| clarkb | jeblair: don't forget the -v on the second fsck | 21:54 |
| jeblair | should i stick a 'v' on there | 21:54 |
| jeblair | yeah :) | 21:54 |
| jeblair | fsck.ext4 -yfDv /dev/xvda1 | 21:54 |
| fungi | definitely nicer | 21:55 |
| clarkb | maybe drop the -y and answer manually each time? though guessing its safe otherwise it wouldn't be suggested | 21:55 |
| *** annegent_ has quit IRC | 21:56 | |
| jeblair | done in etherpad for next time :) | 21:57 |
| clarkb | fungi: jeblair looks like the -D will optimize existing dirs | 21:57 |
| clarkb | so we should get the dir_index goodness | 21:57 |
| *** esikachev has quit IRC | 22:00 | |
| *** kzaitsev_mb has joined #openstack-infra | 22:00 | |
| *** sshnaidm|off has joined #openstack-infra | 22:02 | |
| *** hashar has quit IRC | 22:03 | |
| *** pramodrj07 has quit IRC | 22:03 | |
| *** pramodrj07 has joined #openstack-infra | 22:03 | |
| jeblair | Pass 3A: Optimizing directories | 22:03 |
| *** knikolla has left #openstack-infra | 22:03 | |
| jeblair | there's a pass 3A. :) | 22:03 |
| *** mdrabe has quit IRC | 22:03 | |
| fungi | yep, that's what i was looking for | 22:04 |
| fungi | certainly takes a while | 22:10 |
| jeblair | done | 22:12 |
| *** kzaitsev_mb has quit IRC | 22:12 | |
| jeblair | 10363 symbolic links (9586 fast symbolic links) | 22:12 |
| jeblair | later i'm going to learn what a "fast symbolic link" is. | 22:12 |
| jeblair | fungi, clarkb: reboot now? | 22:12 |
| fungi | go for it | 22:13 |
| clarkb | ya sorry was reading things | 22:13 |
| fungi | fast symlinks? ;) | 22:14 |
| jeblair | /dev/xvda1 on / type ext4 (rw,noatime,errors=remount-ro,barrier=0) | 22:14 |
| fungi | excellent | 22:14 |
| clarkb | well reading that now, but no coreos' enterprise product supports openstack now | 22:14 |
| fungi | jeblair: that roofs seem any faster now? | 22:15 |
| jeblair | yes actually | 22:16 |
| clarkb | fast symbolic links store their target path in the data structures used for storing inode info. slow symlinks store it in a regular file | 22:16 |
| clarkb | apparently the contraint here is you can have paths taht exceed the amount of space for inode info storage | 22:16 |
| fungi | so fast symlinks need only one read | 22:16 |
| clarkb | TIL | 22:16 |
| jeblair | neat | 22:16 |
| jeblair | okay, so i think that's the end of the road; any other testing we should do? | 22:17 |
| *** xarses_ has quit IRC | 22:17 | |
| fungi | none that i can think of, probably ready to pick a date for the outage | 22:17 |
| clarkb | I can't think of anything either | 22:17 |
| jeblair | i want to say that the elapsed time is about 2.5 hours. i'm pretty sure we can expect 1 hour shorter with the kernel cleanup. | 22:17 |
| jeblair | (~2 hours for upgrade, 30 minus for ext4) | 22:18 |
| jeblair | mins | 22:18 |
| fungi | alternatively, we just do it because almost nobody seemed to notice the other day when mailman was stopped for 10+ hours ;) | 22:18 |
| *** ricardoas has quit IRC | 22:18 | |
| jeblair | heh :) | 22:18 |
| jeblair | smtp is nice that way | 22:19 |
| *** matrohon has quit IRC | 22:19 | |
| fungi | but yeah, announcing a 2-hour outage a few days in advance might be nice of us | 22:20 |
| jeblair | fungi: at least 3 hours? (1.5 x 2) :) | 22:20 |
| fungi | fair enough | 22:20 |
| clarkb | we can even point out "you probably won't notice if the outage we had was any indication :) " | 22:20 |
| jeblair | so maybe e just schedule a window on friday afternoon where we say 'expect list traffic to queue'. | 22:20 |
| *** baoli has joined #openstack-infra | 22:21 | |
| fungi | wfm | 22:21 |
| clarkb | sounds good | 22:21 |
| *** esikachev has joined #openstack-infra | 22:22 | |
| *** emerson has joined #openstack-infra | 22:24 | |
| *** dimtruck is now known as zz_dimtruck | 22:25 | |
| *** spzala has quit IRC | 22:30 | |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/zuul feature/zuulv3: Add github reporter status_url config option https://review.openstack.org/449794 | 22:30 |
| *** danger_fo is now known as dangers_away | 22:31 | |
| *** ricardoas has joined #openstack-infra | 22:34 | |
| *** yamahata has quit IRC | 22:36 | |
| *** kzaitsev_mb has joined #openstack-infra | 22:38 | |
| *** xyang1 has quit IRC | 22:38 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/project-config master: Skip puppet integration tests for test-requirements.txt https://review.openstack.org/449834 | 22:40 |
| *** kzaitsev_mb has quit IRC | 22:43 | |
| openstackgerrit | Alex Schultz proposed openstack-infra/project-config master: Skip puppet integration tests for test-requirements.txt https://review.openstack.org/449834 | 22:44 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Fix timeout for quickstart jobs https://review.openstack.org/449023 | 22:45 |
| openstackgerrit | Khai Do proposed openstack-infra/puppet-gerrit master: Update bouncy castle libraries for Gerrit 2.13 https://review.openstack.org/449835 | 22:48 |
| *** mat128 has quit IRC | 22:53 | |
| *** mtanino has quit IRC | 22:57 | |
| *** scarab_ has joined #openstack-infra | 23:01 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Switch trunk/cbs/buildlogs to use https https://review.openstack.org/448041 | 23:04 |
| *** jascott1 has joined #openstack-infra | 23:04 | |
| *** mat128 has joined #openstack-infra | 23:04 | |
| *** ssbarnea has quit IRC | 23:05 | |
| *** scarab_ has quit IRC | 23:05 | |
| *** scarab__ has joined #openstack-infra | 23:05 | |
| *** scarab__ has quit IRC | 23:09 | |
| clarkb | zaro: I have approved the change to build gerrit plugins against new branch. Do we need anything else before we can build everything at the new version? | 23:12 |
| *** ihrachys has joined #openstack-infra | 23:14 | |
| clarkb | zaro: also it looks like we may have to upgrade through 2.12 for online indexer stuff? can we avoid that by doing an offline index update on 2.13 from scratch? | 23:15 |
| *** esikachev has quit IRC | 23:16 | |
| *** yolanda has quit IRC | 23:17 | |
| *** ihrachys has quit IRC | 23:18 | |
| clarkb | oh https://gerrit.googlesource.com/gerrit/+/v2.13-rc0/ReleaseNotes/ReleaseNotes-2.13.txt#27 that next little bit clears it up | 23:18 |
| clarkb | so we are fine but we have to take a longer downtime (which I think is worth it) | 23:18 |
| *** vhosakot_ has quit IRC | 23:19 | |
| openstackgerrit | Khai Do proposed openstack-infra/system-config master: Remove Gerrit delete-project plugin from review-dev.o.o https://review.openstack.org/449840 | 23:20 |
| *** baoli_ has joined #openstack-infra | 23:20 | |
| zaro | clarkb: yeah, upside is one hop instead two, offline index is recommended. | 23:23 |
| *** ijw has joined #openstack-infra | 23:23 | |
| *** markvoelker has quit IRC | 23:24 | |
| *** baoli has quit IRC | 23:24 | |
| clarkb | zaro: reviewing the change you just psuhed now as well (would be good for you to look at comments before heading off to weekend) | 23:24 |
| *** links has quit IRC | 23:24 | |
| zaro | clarkb: you’ll need to approve this series, https://review.openstack.org/#/c/449716/ | 23:24 |
| clarkb | zaro: comment posted on 449835 | 23:25 |
| clarkb | can another infra-root quickly review series at 449716 so we can get gerrit builds going? | 23:26 |
| clarkb | zaro: do my comments on 449835 make sense? basically we need to be careful shifting those comparisons in order to keep support 2.11 while we are on 2.11 too | 23:28 |
| *** hongbin has quit IRC | 23:28 | |
| clarkb | zaro: so we may need if >2.12 else if > 2.10 else (2.9) or just make the 2.11 and 2.13 libs coexist | 23:29 |
| zigo | fungi: Are you saying that DIB adds jessie-bpo by default? I have no clue why it's doing that... | 23:29 |
| pabelanger | -3 | 23:29 |
| clarkb | pabelanger: the -2 code review and -W ? /me assumes its a tyop | 23:29 |
| zigo | pabelanger: I've been trying and trying again to get a Stretch image generated, but I still fail... :/ | 23:30 |
| pabelanger | clarkb: which is that? | 23:30 |
| clarkb | pabelanger: you said -3 :) | 23:30 |
| pabelanger | clarkb: Ah | 23:30 |
| pabelanger | :) | 23:30 |
| pabelanger | +3 | 23:30 |
| pabelanger | zigo: I was going to look at it over the weekend | 23:30 |
| pabelanger | zigo: but jessie is broken for us now too | 23:31 |
| zigo | That'd be great, thanks. | 23:31 |
| zigo | Oh ... :( | 23:31 |
| clarkb | pabelanger: the git review patch too ? | 23:31 |
| zigo | In what way? | 23:31 |
| clarkb | pabelanger: 449715 | 23:31 |
| zaro | clarkb: this one is blocking gerrit plugin build one, https://review.openstack.org/#/c/449739/ | 23:31 |
| clarkb | zaro: ok reviewing that one now | 23:31 |
| zaro | clarkb: i think it’s safe to leave comparison as is. | 23:31 |
| clarkb | zaro: but won't that break 2.11 beacuse the 2.9 code will match 2.11 now too? | 23:32 |
| clarkb | zaro: so we'll use distro libs on 2.11 instead of maven hosted libs like we do currently and we'll tidy the wrong set of filse? | 23:33 |
| *** jamesden_ has joined #openstack-infra | 23:35 | |
| *** jordanP has quit IRC | 23:39 | |
| openstackgerrit | Merged openstack-infra/project-config master: Cleanup Gerrit plugin builds https://review.openstack.org/449739 | 23:40 |
| zaro | clarkb: i think i did that all wrong. let me redo. basically i think we only need to consider 2.11 and 2.13 now correct? | 23:40 |
| openstackgerrit | Merged openstack-infra/project-config master: Build gerrit plugins against 2.13.7 release https://review.openstack.org/449745 | 23:40 |
| *** kzaitsev_mb has joined #openstack-infra | 23:41 | |
| clarkb | zaro: for us yes, its probably worth being extra nice continuing to support older too just in case peolpe use the pupept module | 23:41 |
| clarkb | but I don't think thats super required (guessing not many users use it?) asselin do you know? | 23:42 |
| *** sdague has quit IRC | 23:43 | |
| *** jcoufal has joined #openstack-infra | 23:46 | |
| clarkb | *not many users use it with old gerrit if any. | 23:47 |
| *** spzala has joined #openstack-infra | 23:47 | |
| *** jcoufal_ has quit IRC | 23:49 | |
| *** jcoufal has quit IRC | 23:50 | |
| *** baoli_ has quit IRC | 23:51 | |
| *** scarab_ has joined #openstack-infra | 23:51 | |
| *** jamesden_ has quit IRC | 23:53 | |
| *** scarab_ has quit IRC | 23:55 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!