*** heyongli has quit IRC | 00:02 | |
*** heyongli has joined #openstack-infra | 00:02 | |
*** alex_xu has joined #openstack-infra | 00:04 | |
*** Nil_ has quit IRC | 00:07 | |
*** rpioso is now known as rpioso|afk | 00:08 | |
openstackgerrit | Merged openstack-infra/zuul master: gerrit: cast change and patchset numbers to str https://review.openstack.org/577154 | 00:09 |
---|---|---|
*** Nil_ has joined #openstack-infra | 00:10 | |
*** heyongli has quit IRC | 00:12 | |
*** heyongli has joined #openstack-infra | 00:12 | |
*** felipemonteiro_ has joined #openstack-infra | 00:12 | |
*** jamesmcarthur has joined #openstack-infra | 00:22 | |
*** dingyichen has joined #openstack-infra | 00:22 | |
*** heyongli has quit IRC | 00:22 | |
*** heyongli has joined #openstack-infra | 00:23 | |
*** felipemonteiro_ has quit IRC | 00:26 | |
*** jamesmcarthur has quit IRC | 00:27 | |
*** eharney has joined #openstack-infra | 00:29 | |
*** heyongli has quit IRC | 00:32 | |
*** heyongli has joined #openstack-infra | 00:33 | |
*** austinsun[m] has joined #openstack-infra | 00:33 | |
*** hemna_ has quit IRC | 00:36 | |
*** gyee has quit IRC | 00:39 | |
*** kzaitsev_pi has joined #openstack-infra | 00:42 | |
*** heyongli has quit IRC | 00:43 | |
*** heyongli has joined #openstack-infra | 00:43 | |
*** kzaitsev1pi has quit IRC | 00:45 | |
*** heyongli has quit IRC | 00:53 | |
*** heyongli has joined #openstack-infra | 00:53 | |
*** kzaitsev1pi has joined #openstack-infra | 00:54 | |
*** rh-jelabarre has joined #openstack-infra | 00:55 | |
*** kzaitsev_pi has quit IRC | 00:55 | |
openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Move localloop to exec_sudo https://review.openstack.org/578616 | 00:58 |
*** Dinesh_Bhor has joined #openstack-infra | 01:01 | |
*** heyongli has quit IRC | 01:03 | |
*** heyongli has joined #openstack-infra | 01:04 | |
*** spotz has quit IRC | 01:04 | |
ianw | gema: re my previous msg, linaro keystone seems back now, so glitch in the matrix i guess | 01:04 |
ianw | i don't know what's happened to nb03.o.o however, non-responsive. i'm trying a reboot | 01:04 |
*** yolanda_ has joined #openstack-infra | 01:05 | |
*** yamahata has quit IRC | 01:07 | |
openstackgerrit | Merged openstack-infra/zuul-jobs master: ara-report: use a different fact name for ara_report_path https://review.openstack.org/577676 | 01:08 |
*** yolanda has quit IRC | 01:08 | |
*** yolanda has joined #openstack-infra | 01:09 | |
ianw | hard reboot has it up again | 01:10 |
*** yolanda_ has quit IRC | 01:11 | |
*** jesslampe has joined #openstack-infra | 01:11 | |
*** yolanda_ has joined #openstack-infra | 01:13 | |
*** heyongli has quit IRC | 01:13 | |
*** heyongli has joined #openstack-infra | 01:14 | |
*** edmondsw has joined #openstack-infra | 01:16 | |
*** yolanda has quit IRC | 01:16 | |
*** yolanda__ has joined #openstack-infra | 01:19 | |
*** edmondsw has quit IRC | 01:20 | |
*** spotz has joined #openstack-infra | 01:20 | |
*** jesslampe has quit IRC | 01:22 | |
*** yolanda_ has quit IRC | 01:22 | |
*** heyongli has quit IRC | 01:24 | |
*** heyongli has joined #openstack-infra | 01:24 | |
*** jesslampe has joined #openstack-infra | 01:25 | |
*** jesslampe has quit IRC | 01:26 | |
openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Move localloop to exec_sudo https://review.openstack.org/578616 | 01:26 |
*** linshuicheng[m] has joined #openstack-infra | 01:31 | |
*** rlandy has quit IRC | 01:33 | |
*** heyongli has quit IRC | 01:34 | |
*** heyongli has joined #openstack-infra | 01:34 | |
*** gongysh has joined #openstack-infra | 01:35 | |
*** heyongli has quit IRC | 01:44 | |
*** heyongli has joined #openstack-infra | 01:45 | |
*** heyongli has quit IRC | 01:54 | |
*** heyongli has joined #openstack-infra | 01:55 | |
*** Shrews has quit IRC | 02:00 | |
*** gfidente has joined #openstack-infra | 02:05 | |
*** heyongli has quit IRC | 02:05 | |
*** heyongli has joined #openstack-infra | 02:05 | |
*** hongbin has joined #openstack-infra | 02:06 | |
openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Move localloop to exec_sudo https://review.openstack.org/578616 | 02:09 |
ianw | https://bootstrap.pypa.io/get-pip.py appears unreachable from nb03 | 02:15 |
*** heyongli has quit IRC | 02:15 | |
*** heyongli has joined #openstack-infra | 02:15 | |
*** heyongli has quit IRC | 02:25 | |
*** heyongli has joined #openstack-infra | 02:26 | |
*** links has joined #openstack-infra | 02:30 | |
*** links has quit IRC | 02:30 | |
*** kzaitsev1pi has quit IRC | 02:32 | |
*** sthussey has quit IRC | 02:32 | |
*** kzaitsev_pi has joined #openstack-infra | 02:32 | |
*** dpawlik has joined #openstack-infra | 02:34 | |
*** heyongli has quit IRC | 02:35 | |
*** heyongli has joined #openstack-infra | 02:36 | |
*** dpawlik has quit IRC | 02:38 | |
*** heyongli has quit IRC | 02:46 | |
*** heyongli has joined #openstack-infra | 02:46 | |
*** vivsoni has quit IRC | 02:49 | |
*** heyongli has quit IRC | 02:56 | |
*** heyongli has joined #openstack-infra | 02:56 | |
*** edmondsw has joined #openstack-infra | 03:03 | |
*** roman_g has quit IRC | 03:06 | |
*** heyongli has quit IRC | 03:06 | |
*** heyongli has joined #openstack-infra | 03:07 | |
*** tonytan4ever has quit IRC | 03:07 | |
*** edmondsw has quit IRC | 03:08 | |
*** zhangfei has joined #openstack-infra | 03:10 | |
*** heyongli has quit IRC | 03:16 | |
*** heyongli has joined #openstack-infra | 03:17 | |
*** zhangfei has quit IRC | 03:26 | |
*** zhangfei has joined #openstack-infra | 03:27 | |
*** heyongli has quit IRC | 03:27 | |
*** heyongli has joined #openstack-infra | 03:27 | |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: builder: do not configure provider that doesn't manage images https://review.openstack.org/578642 | 03:35 |
*** heyongli has quit IRC | 03:37 | |
*** heyongli has joined #openstack-infra | 03:37 | |
*** tonytan4ever has joined #openstack-infra | 03:46 | |
*** heyongli has quit IRC | 03:47 | |
*** heyongli has joined #openstack-infra | 03:48 | |
*** lpetrut has joined #openstack-infra | 03:49 | |
*** tonytan4ever has quit IRC | 03:50 | |
*** vivsoni has joined #openstack-infra | 03:51 | |
*** rh-jelabarre has quit IRC | 03:52 | |
*** udesale has joined #openstack-infra | 03:53 | |
*** slaweq has quit IRC | 03:53 | |
*** heyongli has quit IRC | 03:57 | |
*** heyongli has joined #openstack-infra | 03:58 | |
*** heyongli has quit IRC | 04:08 | |
*** heyongli has joined #openstack-infra | 04:08 | |
*** yamahata has joined #openstack-infra | 04:08 | |
*** slaweq has joined #openstack-infra | 04:11 | |
*** slaweq has quit IRC | 04:16 | |
*** heyongli has quit IRC | 04:18 | |
*** heyongli has joined #openstack-infra | 04:18 | |
*** boden has joined #openstack-infra | 04:22 | |
*** tonytan4ever has joined #openstack-infra | 04:24 | |
*** boden has quit IRC | 04:26 | |
*** sree_ has joined #openstack-infra | 04:27 | |
*** sree_ has quit IRC | 04:27 | |
*** hongbin has quit IRC | 04:27 | |
*** heyongli has quit IRC | 04:28 | |
*** heyongli has joined #openstack-infra | 04:29 | |
*** lpetrut has quit IRC | 04:32 | |
*** ramishra has joined #openstack-infra | 04:34 | |
*** heyongli has quit IRC | 04:38 | |
*** heyongli has joined #openstack-infra | 04:39 | |
AJaeger | ianw, frickler, jhesketh with https://review.openstack.org/#/c/577307/ merged, can we merge the docs update as well, please? That's https://review.openstack.org/#/c/577310 | 04:48 |
jhesketh | AJaeger: looking | 04:49 |
*** heyongli has quit IRC | 04:49 | |
*** heyongli has joined #openstack-infra | 04:49 | |
jhesketh | +W | 04:51 |
AJaeger | thanks! | 04:51 |
*** edmondsw has joined #openstack-infra | 04:52 | |
AJaeger | smcginnis: https://review.openstack.org/#/c/578574/ should be fine, if you don't define jobs, no need t olist the repo in projects.yaml | 04:57 |
*** edmondsw has quit IRC | 04:57 | |
openstackgerrit | Merged openstack-infra/system-config master: Update to openstackdocstheme https://review.openstack.org/577678 | 04:58 |
*** heyongli has quit IRC | 04:59 | |
*** heyongli has joined #openstack-infra | 04:59 | |
*** ykarel has joined #openstack-infra | 05:00 | |
*** janki has joined #openstack-infra | 05:00 | |
openstackgerrit | Merged openstack-infra/infra-manual master: Remove system-required instructions for creators https://review.openstack.org/577310 | 05:05 |
openstackgerrit | Merged openstack-infra/project-config master: Readd openstack/operations-guide repo https://review.openstack.org/578441 | 05:05 |
*** gongysh has quit IRC | 05:06 | |
*** heyongli has quit IRC | 05:09 | |
*** heyongli has joined #openstack-infra | 05:10 | |
openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Call kpartx remove in umount, not cleanup https://review.openstack.org/578657 | 05:10 |
*** slaweq has joined #openstack-infra | 05:11 | |
openstackgerrit | Merged openstack-infra/system-config master: Add statusbot to #openstack-helm https://review.openstack.org/565887 | 05:13 |
openstackgerrit | Merged openstack-infra/system-config master: Add statusbot to #openstack-security https://review.openstack.org/565888 | 05:14 |
*** slaweq has quit IRC | 05:15 | |
openstackgerrit | Merged openstack-infra/system-config master: Blackhole messages to ambassadors-owner on lists https://review.openstack.org/576539 | 05:16 |
*** jbadiapa has quit IRC | 05:17 | |
*** dhajare has joined #openstack-infra | 05:19 | |
*** heyongli has quit IRC | 05:19 | |
*** heyongli has joined #openstack-infra | 05:20 | |
*** heyongli has quit IRC | 05:30 | |
*** heyongli has joined #openstack-infra | 05:30 | |
*** gongysh has joined #openstack-infra | 05:31 | |
*** openstackgerrit has quit IRC | 05:34 | |
*** jesslampe has joined #openstack-infra | 05:35 | |
*** pgadiya has joined #openstack-infra | 05:36 | |
*** pgadiya has quit IRC | 05:36 | |
*** quiquell|off is now known as quiquell|rover | 05:37 | |
*** cshastri has joined #openstack-infra | 05:38 | |
*** heyongli has quit IRC | 05:40 | |
*** heyongli has joined #openstack-infra | 05:40 | |
*** jogo has quit IRC | 05:42 | |
*** iranzo has joined #openstack-infra | 05:44 | |
*** quiquell|rover is now known as quique|rover|bbl | 05:45 | |
*** bdodd has quit IRC | 05:46 | |
*** heyongli has quit IRC | 05:50 | |
*** heyongli has joined #openstack-infra | 05:51 | |
*** openstackstatus has quit IRC | 05:51 | |
*** pahuang has quit IRC | 05:51 | |
*** openstackstatus has joined #openstack-infra | 05:52 | |
*** ChanServ sets mode: +v openstackstatus | 05:52 | |
*** Bhujay has joined #openstack-infra | 05:54 | |
*** yolanda__ is now known as yolanda | 05:59 | |
*** heyongli has quit IRC | 06:00 | |
*** heyongli has joined #openstack-infra | 06:01 | |
*** zhangfei has quit IRC | 06:01 | |
*** andreas_s has joined #openstack-infra | 06:08 | |
*** quique|rover|bbl is now known as quiquell|rover | 06:10 | |
*** ramishra has quit IRC | 06:10 | |
*** heyongli has quit IRC | 06:11 | |
*** slaweq has joined #openstack-infra | 06:11 | |
*** heyongli has joined #openstack-infra | 06:11 | |
*** ramishra has joined #openstack-infra | 06:12 | |
*** zhangfei has joined #openstack-infra | 06:13 | |
*** slaweq has quit IRC | 06:15 | |
*** jogo has joined #openstack-infra | 06:17 | |
*** quiquell|rover is now known as quique|rover|afk | 06:17 | |
*** heyongli has quit IRC | 06:21 | |
*** heyongli has joined #openstack-infra | 06:21 | |
*** jamesmcarthur has joined #openstack-infra | 06:23 | |
*** jamesmcarthur has quit IRC | 06:27 | |
*** lpetrut has joined #openstack-infra | 06:28 | |
*** ccamacho has quit IRC | 06:29 | |
*** bdodd has joined #openstack-infra | 06:30 | |
*** heyongli has quit IRC | 06:31 | |
*** heyongli has joined #openstack-infra | 06:32 | |
*** ramishra has quit IRC | 06:35 | |
*** openstackgerrit has joined #openstack-infra | 06:35 | |
openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Test for system-required https://review.openstack.org/578701 | 06:35 |
AJaeger | clarkb: added a test so that nobody readds system-required ^ | 06:36 |
*** ramishra has joined #openstack-infra | 06:37 | |
openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Test for system-required https://review.openstack.org/578701 | 06:39 |
*** aspiers[m] has quit IRC | 06:40 | |
*** dpawlik has joined #openstack-infra | 06:40 | |
*** edmondsw has joined #openstack-infra | 06:40 | |
*** aspiers[m] has joined #openstack-infra | 06:41 | |
*** smcginnis[m] has quit IRC | 06:41 | |
*** heyongli has quit IRC | 06:41 | |
*** heyongli has joined #openstack-infra | 06:42 | |
*** smcginnis[m] has joined #openstack-infra | 06:43 | |
*** nicolasbock has joined #openstack-infra | 06:43 | |
*** slaweq has joined #openstack-infra | 06:44 | |
*** edmondsw has quit IRC | 06:46 | |
*** slaweq has quit IRC | 06:48 | |
*** heyongli has quit IRC | 06:52 | |
*** heyongli has joined #openstack-infra | 06:52 | |
*** caphrim007_ has quit IRC | 06:52 | |
*** caphrim007 has joined #openstack-infra | 06:53 | |
*** quique|rover|afk is now known as quiquell|rover | 06:53 | |
*** slaweq has joined #openstack-infra | 06:54 | |
*** ramishra has quit IRC | 06:55 | |
*** agopi has quit IRC | 06:56 | |
*** ramishra has joined #openstack-infra | 07:00 | |
*** jcoufal has joined #openstack-infra | 07:01 | |
*** heyongli has quit IRC | 07:02 | |
*** cfriesen has quit IRC | 07:02 | |
*** heyongli has joined #openstack-infra | 07:02 | |
*** shardy has joined #openstack-infra | 07:05 | |
openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: executor: add executing job INFO log https://review.openstack.org/578704 | 07:08 |
*** peereb has joined #openstack-infra | 07:09 | |
*** stakeda has quit IRC | 07:10 | |
*** pcaruana has joined #openstack-infra | 07:10 | |
*** Adri2000 has quit IRC | 07:11 | |
*** tesseract has joined #openstack-infra | 07:11 | |
*** heyongli has quit IRC | 07:12 | |
*** heyongli has joined #openstack-infra | 07:13 | |
*** Adri2000 has joined #openstack-infra | 07:15 | |
openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: block-device lvm: fix umount phase https://review.openstack.org/503958 | 07:17 |
openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Move localloop to exec_sudo https://review.openstack.org/578616 | 07:17 |
openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Call kpartx remove in umount, not cleanup https://review.openstack.org/578657 | 07:17 |
openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [WIP] LVM testing for xenial-build-succeeds https://review.openstack.org/578708 | 07:17 |
*** jbadiapa has joined #openstack-infra | 07:17 | |
*** heyongli has quit IRC | 07:22 | |
*** heyongli has joined #openstack-infra | 07:23 | |
*** ramishra has quit IRC | 07:23 | |
*** ramishra has joined #openstack-infra | 07:24 | |
*** amoralej|off is now known as amoralej | 07:26 | |
*** ramishra has quit IRC | 07:29 | |
*** ramishra has joined #openstack-infra | 07:30 | |
*** andreas_s has quit IRC | 07:31 | |
*** sungn has joined #openstack-infra | 07:31 | |
*** andreas_s has joined #openstack-infra | 07:32 | |
*** heyongli has quit IRC | 07:33 | |
*** heyongli has joined #openstack-infra | 07:33 | |
*** jiapei has joined #openstack-infra | 07:34 | |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-nodepool master: Fix name of libxslt dev package https://review.openstack.org/577519 | 07:34 |
gema | ianw: yeah, we had a 10 min down time yesterday, whilst rebooting a switch to get the jumbo frames configured | 07:34 |
gema | ianw: sorry about that, didn't expect it to take so long to reboot :S | 07:34 |
ianw | gema: ok ... you can see from the builds it is trying to do @ https://nb03.openstack.org/ that it's not going well | 07:35 |
*** yolanda_ has joined #openstack-infra | 07:35 | |
gema | ianw: ok, we'll pick that up today | 07:35 |
gema | ianw: we were trying to get the cloud to have sane MTUs for VMs/containers | 07:36 |
*** agopi has joined #openstack-infra | 07:36 | |
gema | ianw: what is the time of those logs? UTC? | 07:37 |
*** yolanda is now known as Guest63448 | 07:37 | |
*** yolanda_ is now known as yolanda | 07:37 | |
sungn | Hello, everyone. I'm now building a third party CI environment. Should I use zuul and molteniron to build it? If so should I also use jenkins? Can anybody here have experience help me please? Thank you! | 07:38 |
*** Guest63448 has quit IRC | 07:38 | |
ianw | gema: yes, utc | 07:39 |
*** ramishra has quit IRC | 07:39 | |
gema | ack, so things are not going great yet, anyway, keep an eye on that ticket we have open | 07:40 |
gema | we'll get there | 07:40 |
gema | ianw: feel free to add any concerns to it also | 07:40 |
ianw | gema: ok, yeah you can mostly see that it's just random network failures trying to grab things during the build | 07:40 |
gema | yup | 07:40 |
gema | will add that | 07:40 |
ianw | https://bootstrap.pypa.io/get-pip.py was a good one to test, sometimes it would be ok, other times not | 07:41 |
ianw | that is behind some sort of cdn, but best i could tell it was always to same ip | 07:41 |
ianw | 151.101.76.175 | 07:41 |
*** ccamacho has joined #openstack-infra | 07:41 | |
ianw | http://paste.openstack.org/show/724499/ -- two runs from just a second ago. first one was fine, second one didn't work | 07:42 |
*** heyongli has quit IRC | 07:43 | |
*** heyongli has joined #openstack-infra | 07:43 | |
*** jpena|off is now known as jpena | 07:44 | |
*** hashar has joined #openstack-infra | 07:44 | |
gema | ianw: cool, thanks for the info, we'll look into that | 07:45 |
openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: Only detach device if all partitions have been cleaned https://review.openstack.org/576876 | 07:52 |
*** heyongli has quit IRC | 07:53 | |
*** heyongli has joined #openstack-infra | 07:54 | |
*** numans has joined #openstack-infra | 07:55 | |
*** numans_ has quit IRC | 07:56 | |
*** tosky has joined #openstack-infra | 07:58 | |
*** jesslampe has quit IRC | 07:59 | |
*** jesslampe has joined #openstack-infra | 07:59 | |
*** andreas_s has quit IRC | 07:59 | |
*** andreas_s_ has joined #openstack-infra | 08:03 | |
*** rpittau has quit IRC | 08:03 | |
*** rpittau has joined #openstack-infra | 08:03 | |
*** andreas_s_ has quit IRC | 08:03 | |
*** ykarel is now known as ykarel|lunch | 08:03 | |
*** heyongli has quit IRC | 08:03 | |
*** heyongli has joined #openstack-infra | 08:04 | |
*** jesslampe has quit IRC | 08:04 | |
*** threestrands has joined #openstack-infra | 08:06 | |
*** threestrands has quit IRC | 08:06 | |
*** threestrands has joined #openstack-infra | 08:06 | |
*** jpich has joined #openstack-infra | 08:07 | |
*** d0ugal has joined #openstack-infra | 08:08 | |
*** sshnaidm is now known as sshnaidm|afk | 08:09 | |
*** threestrands has quit IRC | 08:13 | |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-nodepool master: Fix name of libxslt dev package https://review.openstack.org/577519 | 08:13 |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-openstackci master: Add beaker-rspec bindep dependencies https://review.openstack.org/578559 | 08:13 |
*** heyongli has quit IRC | 08:14 | |
*** heyongli has joined #openstack-infra | 08:14 | |
*** electrofelix has joined #openstack-infra | 08:15 | |
*** tonytan4ever_brb has joined #openstack-infra | 08:15 | |
*** tonytan4ever has quit IRC | 08:19 | |
*** tonytan4ever_brb has quit IRC | 08:20 | |
*** tonytan4ever has joined #openstack-infra | 08:20 | |
*** dansmith has quit IRC | 08:20 | |
*** florianf|afk is now known as florianf | 08:21 | |
*** dansmith has joined #openstack-infra | 08:21 | |
*** dansmith is now known as Guest88823 | 08:21 | |
*** roman_g has joined #openstack-infra | 08:22 | |
*** heyongli has quit IRC | 08:24 | |
*** heyongli has joined #openstack-infra | 08:24 | |
*** ramishra has joined #openstack-infra | 08:25 | |
*** edmondsw has joined #openstack-infra | 08:29 | |
*** numans has quit IRC | 08:30 | |
*** jcoufal has quit IRC | 08:31 | |
*** ethfci has joined #openstack-infra | 08:32 | |
*** edmondsw has quit IRC | 08:34 | |
*** heyongli has quit IRC | 08:34 | |
*** heyongli has joined #openstack-infra | 08:35 | |
*** derekh has joined #openstack-infra | 08:40 | |
*** heyongli has quit IRC | 08:44 | |
*** heyongli has joined #openstack-infra | 08:45 | |
*** dingyichen has quit IRC | 08:47 | |
*** andreas_s has joined #openstack-infra | 08:54 | |
*** andreas_s has quit IRC | 08:54 | |
*** andreas_s has joined #openstack-infra | 08:54 | |
*** heyongli has quit IRC | 08:55 | |
*** andreas_s has quit IRC | 08:55 | |
*** heyongli has joined #openstack-infra | 08:55 | |
*** andreas_s has joined #openstack-infra | 08:59 | |
*** andreas_s has quit IRC | 09:01 | |
*** andreas_s has joined #openstack-infra | 09:01 | |
*** andreas_s has quit IRC | 09:02 | |
*** heyongli has quit IRC | 09:05 | |
*** heyongli has joined #openstack-infra | 09:05 | |
*** andreas__ has joined #openstack-infra | 09:06 | |
*** andreas__ has quit IRC | 09:06 | |
*** andreas_s_ has joined #openstack-infra | 09:07 | |
*** heyongli has quit IRC | 09:15 | |
*** d0ugal has quit IRC | 09:16 | |
*** heyongli has joined #openstack-infra | 09:16 | |
*** d0ugal has joined #openstack-infra | 09:16 | |
*** armaan has joined #openstack-infra | 09:17 | |
*** andreas_s_ has quit IRC | 09:22 | |
*** ykarel|lunch is now known as ykarel | 09:23 | |
*** heyongli has quit IRC | 09:25 | |
*** heyongli has joined #openstack-infra | 09:26 | |
*** andreas_s has joined #openstack-infra | 09:27 | |
*** andreas_s has quit IRC | 09:29 | |
*** andreas_s has joined #openstack-infra | 09:29 | |
*** pcichy has joined #openstack-infra | 09:30 | |
*** zoli is now known as zoli|lunch | 09:34 | |
*** Dinesh_Bhor has quit IRC | 09:34 | |
*** evrardjp has quit IRC | 09:35 | |
*** pcichy has quit IRC | 09:35 | |
*** pcichy has joined #openstack-infra | 09:35 | |
*** heyongli has quit IRC | 09:36 | |
*** heyongli has joined #openstack-infra | 09:36 | |
*** gfidente has quit IRC | 09:37 | |
*** evrardjp has joined #openstack-infra | 09:41 | |
*** heyongli has quit IRC | 09:46 | |
*** heyongli has joined #openstack-infra | 09:46 | |
*** dtantsur|afk is now known as dtantsur | 09:49 | |
openstackgerrit | Chason Chan proposed openstack-infra/project-config master: Add api-ref-jobs to Watcher templates list https://review.openstack.org/578750 | 09:51 |
*** rcernin has quit IRC | 09:56 | |
*** heyongli has quit IRC | 09:56 | |
*** heyongli has joined #openstack-infra | 09:57 | |
*** armaan has quit IRC | 10:02 | |
*** armaan has joined #openstack-infra | 10:02 | |
*** armaan has quit IRC | 10:03 | |
*** armaan has joined #openstack-infra | 10:04 | |
*** heyongli has quit IRC | 10:06 | |
*** heyongli has joined #openstack-infra | 10:07 | |
*** jcoufal has joined #openstack-infra | 10:11 | |
*** andreas_s has quit IRC | 10:12 | |
*** jcoufal has quit IRC | 10:14 | |
*** jcoufal has joined #openstack-infra | 10:15 | |
*** andreas_s has joined #openstack-infra | 10:16 | |
*** heyongli has quit IRC | 10:17 | |
*** heyongli has joined #openstack-infra | 10:17 | |
*** edmondsw has joined #openstack-infra | 10:17 | |
*** edmondsw has quit IRC | 10:22 | |
*** boden has joined #openstack-infra | 10:24 | |
*** armaan has quit IRC | 10:26 | |
*** armaan has joined #openstack-infra | 10:27 | |
*** heyongli has quit IRC | 10:27 | |
*** heyongli has joined #openstack-infra | 10:28 | |
*** boden has quit IRC | 10:29 | |
*** apetrich has quit IRC | 10:30 | |
*** armaan has quit IRC | 10:31 | |
*** udesale has quit IRC | 10:32 | |
*** boden has joined #openstack-infra | 10:33 | |
*** heyongli has quit IRC | 10:37 | |
*** heyongli has joined #openstack-infra | 10:38 | |
*** peereb has quit IRC | 10:43 | |
*** ramishra has quit IRC | 10:44 | |
*** salv-orlando has joined #openstack-infra | 10:44 | |
*** salv-orlando has quit IRC | 10:45 | |
*** heyongli has quit IRC | 10:47 | |
*** heyongli has joined #openstack-infra | 10:48 | |
*** gongysh has quit IRC | 10:52 | |
*** zoli|lunch is now known as zoli | 10:55 | |
*** heyongli has quit IRC | 10:58 | |
*** heyongli has joined #openstack-infra | 10:58 | |
*** sshnaidm|afk is now known as sshnaidm | 10:58 | |
*** jpena is now known as jpena|lunch | 10:59 | |
*** andreas_s has quit IRC | 11:04 | |
*** andreas_s has joined #openstack-infra | 11:05 | |
*** heyongli has quit IRC | 11:08 | |
*** heyongli has joined #openstack-infra | 11:08 | |
*** andreas_s has quit IRC | 11:10 | |
*** zhangfei has quit IRC | 11:10 | |
*** andreas_s has joined #openstack-infra | 11:11 | |
*** apetrich has joined #openstack-infra | 11:11 | |
*** dpawlik has quit IRC | 11:17 | |
*** heyongli has quit IRC | 11:18 | |
*** heyongli has joined #openstack-infra | 11:19 | |
*** andreas_s has quit IRC | 11:19 | |
*** dpawlik has joined #openstack-infra | 11:20 | |
*** andreas_s has joined #openstack-infra | 11:20 | |
*** numans_ has joined #openstack-infra | 11:27 | |
*** heyongli has quit IRC | 11:28 | |
*** heyongli has joined #openstack-infra | 11:29 | |
*** gfidente has joined #openstack-infra | 11:29 | |
*** gfidente has quit IRC | 11:29 | |
*** gfidente has joined #openstack-infra | 11:29 | |
*** ldnunes has joined #openstack-infra | 11:29 | |
openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: Only detach device if all partitions have been cleaned https://review.openstack.org/576876 | 11:29 |
*** dtantsur is now known as dtantsur|brb | 11:32 | |
*** camunoz has joined #openstack-infra | 11:34 | |
*** agopi has quit IRC | 11:34 | |
*** gongysh has joined #openstack-infra | 11:36 | |
*** numans_ has quit IRC | 11:36 | |
*** zul has joined #openstack-infra | 11:39 | |
openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Add tenant yaml validation option to scheduler https://review.openstack.org/574265 | 11:39 |
*** heyongli has quit IRC | 11:39 | |
*** heyongli has joined #openstack-infra | 11:39 | |
*** pcichy has quit IRC | 11:43 | |
*** salv-orlando has joined #openstack-infra | 11:45 | |
*** jcoufal has quit IRC | 11:48 | |
*** zul has quit IRC | 11:49 | |
*** heyongli has quit IRC | 11:49 | |
*** heyongli has joined #openstack-infra | 11:49 | |
*** ramishra has joined #openstack-infra | 11:50 | |
openstackgerrit | Ankit Kumar proposed openstack/diskimage-builder master: Adding new dib element https://review.openstack.org/578773 | 11:50 |
*** jiapei has quit IRC | 11:50 | |
*** ramishra has quit IRC | 11:51 | |
*** salv-orlando has quit IRC | 11:51 | |
*** ramishra has joined #openstack-infra | 11:51 | |
*** amoralej is now known as amoralej|lunch | 11:54 | |
openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: Add new modprobe element https://review.openstack.org/577170 | 11:56 |
*** pcaruana has quit IRC | 11:56 | |
*** jpena|lunch is now known as jpena | 11:56 | |
*** heyongli has quit IRC | 11:59 | |
*** heyongli has joined #openstack-infra | 12:00 | |
*** andreas_s has quit IRC | 12:03 | |
*** andreas_s has joined #openstack-infra | 12:04 | |
*** edmondsw has joined #openstack-infra | 12:06 | |
*** andreas_s has quit IRC | 12:08 | |
*** andreas_s has joined #openstack-infra | 12:08 | |
fungi | #status log lists.openstack.org has been removed from the emergency disable list now that https://review.openstack.org/576539 has merged | 12:09 |
openstackstatus | fungi: finished logging | 12:09 |
*** heyongli has quit IRC | 12:09 | |
*** heyongli has joined #openstack-infra | 12:10 | |
*** edmondsw has quit IRC | 12:11 | |
*** zhangfei has joined #openstack-infra | 12:11 | |
*** Bhujay has quit IRC | 12:12 | |
*** rosmaita has joined #openstack-infra | 12:13 | |
*** rlandy has joined #openstack-infra | 12:13 | |
*** sthussey has joined #openstack-infra | 12:19 | |
*** heyongli has quit IRC | 12:20 | |
*** heyongli has joined #openstack-infra | 12:20 | |
*** rosmaita has quit IRC | 12:21 | |
*** trown|outtypewww is now known as trown | 12:28 | |
*** rosmaita has joined #openstack-infra | 12:29 | |
*** panda|bike is now known as panda | 12:30 | |
*** heyongli has quit IRC | 12:30 | |
*** heyongli has joined #openstack-infra | 12:30 | |
*** zhangfei has quit IRC | 12:30 | |
*** eharney has quit IRC | 12:35 | |
*** zul has joined #openstack-infra | 12:37 | |
*** agopi has joined #openstack-infra | 12:37 | |
*** tpsilva has joined #openstack-infra | 12:37 | |
*** apetrich has quit IRC | 12:38 | |
*** heyongli has quit IRC | 12:40 | |
*** rlandy is now known as rlandy|afk | 12:40 | |
*** heyongli has joined #openstack-infra | 12:41 | |
mordred | sungn: I don't know about molteniron, but yes, you should use zuul. if you use zuul, you do not need jenkins | 12:41 |
cmurphy | is there a standard workaround for this pip 10 weirdness? http://logs.openstack.org/22/577522/2/check/openstackci-beaker-ubuntu-trusty/de1a314/job-output.txt.gz#_2018-06-28_08_45_58_485083 i'm only seeing it on trusty | 12:42 |
odyssey4me | is it possible for projects to use status bot (or another bot) similarly to the way infra uses it to record 'log' events? We'd like to have it as a nice way of recording noted events for project members which can then easily be shared/reviewed/discussed at meetings. | 12:43 |
*** pcaruana has joined #openstack-infra | 12:44 | |
fungi | cmurphy: the pip 10 maintainers don't consider it weirdness. some potential workarounds are enumerated in https://github.com/pypa/pip/issues/4805 | 12:50 |
*** heyongli has quit IRC | 12:50 | |
*** heyongli has joined #openstack-infra | 12:51 | |
*** alexchadin has joined #openstack-infra | 12:51 | |
fungi | odyssey4me: you could run one yourself, the source code is available: https://git.openstack.org/cgit/openstack-infra/statusbot | 12:51 |
fungi | odyssey4me: though working with us to design something could be cool too | 12:52 |
*** rcernin has joined #openstack-infra | 12:52 | |
*** dtantsur|brb is now known as dtantsur | 12:55 | |
fungi | cmurphy: if we can't uninstall the six package from the distro, the next easiest thing is to delete its dist-info file from the filesystem so pip doesn't see it | 12:57 |
cmurphy | fungi: thanks, I was wondering if infra had settled on a particular workaround since I'm sure it's caused problems | 13:00 |
cmurphy | I sort of assume python-six can't just be uninstalled but i'll make sure before I get more heavy handed | 13:00 |
*** heyongli has quit IRC | 13:01 | |
openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/project-config master: Add devstack-plugin-ceph to #openstack-qa channel https://review.openstack.org/578790 | 13:01 |
*** heyongli has joined #openstack-infra | 13:01 | |
frickler | gmann: andreaf: ^^ | 13:01 |
fungi | cmurphy: where it came up in devstack, i think we ended up removing the dist-info files for affected distro packages | 13:01 |
*** edmondsw has joined #openstack-infra | 13:01 | |
cmurphy | fungi: ah okay | 13:02 |
AJaeger | config-core, could you review https://review.openstack.org/#/c/578574/ and https://review.openstack.org/#/c/578701 to finalize removing of system-required template from project-config, please? | 13:02 |
andreaf | frickler +1 | 13:02 |
frickler | cmurphy: fungi: current solution in devstack is still to pin pip<10, workarounds for pip>=10 are still in progress | 13:02 |
*** myoung|pto has quit IRC | 13:03 | |
cmurphy | oh | 13:03 |
cmurphy | lovely | 13:03 |
fungi | frickler: interesting. maybe i'm thinking of cases where it came up in diskimage-builder in that case | 13:03 |
frickler | see the stack below https://review.openstack.org/#/c/561597/ | 13:03 |
*** _ari_ has quit IRC | 13:03 | |
*** otherwiseguy has quit IRC | 13:03 | |
*** ihar has joined #openstack-infra | 13:03 | |
*** _ari_ has joined #openstack-infra | 13:04 | |
*** myoung has joined #openstack-infra | 13:04 | |
*** otherwiseguy has joined #openstack-infra | 13:05 | |
fungi | aha, it was the egg-info files and i was thinking of https://review.openstack.org/562884 it seems | 13:05 |
ssbarnea[m] | how can I disable use of upper_constraints use on build-openstack-sphinx-docs? | 13:05 |
*** andreas_s has quit IRC | 13:06 | |
*** andreas_s has joined #openstack-infra | 13:07 | |
*** andreas_s has quit IRC | 13:07 | |
ssbarnea[m] | i found one case where this fails because package has "pyyaml>4" and the constraints are lower, instead of failing pip respects the constraints and installed a version that is not compatible. | 13:07 |
*** andreas_s has joined #openstack-infra | 13:07 | |
smcginnis | ssbarnea[m]: Hmm, that shouldn't happen that way. Do you have a link to the job? | 13:08 |
*** rcernin has quit IRC | 13:08 | |
ssbarnea[m] | smcginnis: check https://review.openstack.org/#/c/578754/ -- look at differences | 13:08 |
*** andreas_s has quit IRC | 13:08 | |
*** rh-jelabarre has joined #openstack-infra | 13:09 | |
ssbarnea[m] | building docs in classic way works, as it does not use any constraints. but on this template it fails. | 13:09 |
*** heyongli has quit IRC | 13:11 | |
*** heyongli has joined #openstack-infra | 13:11 | |
gmann | frickler: thanks. Lgtm | 13:12 |
*** andreas_s_ has joined #openstack-infra | 13:13 | |
*** amoralej|lunch is now known as amoralej | 13:14 | |
smcginnis | ssbarnea[m]: Oh, I read what you said better now. So you are trying to raise the pyyaml version in requirements.txt to be higher than what is in upper-constraints. | 13:15 |
smcginnis | ssbarnea[m]: That's not possible, at least as far as I'm aware of. | 13:15 |
smcginnis | ssbarnea[m]: The upper-constraint for pyyaml would need to be raised first to allow for that. | 13:16 |
odyssey4me | fungi: it seems that with the right extensions, mediawiki could do something like a datatables view - so all data of each different type could be in a single page, then there can be sub-pages which have a subset of that data | 13:17 |
*** andreas_s_ has quit IRC | 13:17 | |
odyssey4me | fungi: so effectively it could be a pretty straightforward implementation - put all 'log' items in a single page, then have filtered views per channel | 13:17 |
*** quiquell|rover is now known as quique|rover|lch | 13:18 | |
odyssey4me | it's probably terribly inefficient, but no worse than it is today I guess | 13:18 |
*** agopi has quit IRC | 13:19 | |
ssbarnea[m] | jjb is not really openstack anymore, and raising pyyaml version on constraints would be very hard. pyyaml broke the API in 4.1. I am inclined to remove the openstack-docs job and rely on one. As long the publish to readthedocs will work we should be fine. | 13:19 |
ssbarnea[m] | anyway , nobody needs two flavours of "docs" building :) | 13:19 |
*** eharney has joined #openstack-infra | 13:19 | |
smcginnis | ssbarnea[m]: That makes sense. If it doesn't need to go to docs.openstack.org then you don't necessarily need that job. | 13:19 |
AJaeger | ssbarnea[m]: jjb is still listed in governance as infra project | 13:20 |
*** udesale has joined #openstack-infra | 13:20 | |
smcginnis | ssbarnea[m]: Another non-openstack centric docs job could probably be added. | 13:20 |
ssbarnea[m] | ... or I could spend 2h and make it work with both versions of pyyaml. | 13:21 |
*** heyongli has quit IRC | 13:21 | |
AJaeger | ssbarnea[m]: do you need pyyaml for docs building at all? | 13:21 |
smcginnis | It doesn't look like many projects use pyyaml from my quick search. Probably good just to propose raising it above 4.1 and see what the repercussions would be. | 13:22 |
AJaeger | ssbarnea[m]: alternative: Create doc/requirements.txt and use that one... | 13:22 |
fungi | you could probably use a job variant which passes in an alternative (empty) constraints file | 13:22 |
*** heyongli has joined #openstack-infra | 13:22 | |
fungi | but yeah, if you don't need pyyaml installed for docs builds, what AJaeger suggests is easiest | 13:22 |
smcginnis | AJaeger: Wouldn't doc/requirements.txt still be subject to constraints checks? | 13:22 |
AJaeger | smcginnis: it would be - but it would not include pyyaml ;) | 13:22 |
smcginnis | AJaeger: Ah, so if it's not really needed just for docs, use that different file to get by. Gotcha. | 13:23 |
AJaeger | smcginnis: exactly | 13:23 |
fungi | right, the recommendation is to use a doc/requirements.txt which omits pyyaml and anything else not needed for sphinx-build to run successfully | 13:23 |
AJaeger | ssbarnea[m]: that should be the quickest way ^ | 13:23 |
ssbarnea[m] | i am afraid that the failure is caused by an inline code example in the docs. | 13:23 |
AJaeger | ;( | 13:24 |
ssbarnea[m] | but I am considering removing or updating it as I do not find it useful at all. | 13:24 |
fungi | is there some sphinx extension which is importing pyyaml? | 13:24 |
ssbarnea[m] | to me it looks more like documenting the code not the usage, so it should not be in the docs. let me try few approaches. | 13:25 |
ssbarnea[m] | and guys, thanks for the help here, i clearly have few paths that I can take. | 13:25 |
*** andreas_s has joined #openstack-infra | 13:27 | |
*** numans_ has joined #openstack-infra | 13:27 | |
*** caphrim007 has quit IRC | 13:28 | |
AJaeger | ssbarnea[m]: tested locally and worked directly - feel free to take over (change https://review.openstack.org/578798 ) | 13:28 |
*** caphrim007 has joined #openstack-infra | 13:29 | |
AJaeger | ssbarnea[m]: Oh, that one installed "PyYAML==4.1", so might break with constraints in the gate - let's see | 13:29 |
*** heyongli has quit IRC | 13:31 | |
*** heyongli has joined #openstack-infra | 13:32 | |
*** numans_ has quit IRC | 13:32 | |
mordred | ssbarnea[m], fungi: it's jjb - even as an infra project it doesn' tneed build-openstack-sphinx-docs - just use build-sphinx-docs | 13:33 |
*** caphrim007 has quit IRC | 13:33 | |
fungi | agreed. i see no particular reason to force jjb to use the constranits-based jobs | 13:34 |
ssbarnea[m] | i will fix it | 13:34 |
mordred | build-openstack-sphinx-docs is just a wrapper around build-sphinx-docs which adds openstack constraints enforcement | 13:34 |
*** Shrews has joined #openstack-infra | 13:34 | |
*** mriedem has joined #openstack-infra | 13:37 | |
*** apetrich has joined #openstack-infra | 13:39 | |
*** andreas_s has quit IRC | 13:40 | |
*** heyongli has quit IRC | 13:42 | |
*** heyongli has joined #openstack-infra | 13:42 | |
ssbarnea[m] | mordred: i see that jjb docs are published on docs.openstack.org too, so the use of "publish-openstack-sphinx-docs-infra" template sounds correct. and I suspect that this one is also introducing the constrains. | 13:43 |
*** quique|rover|lch is now known as quiquell|rover | 13:43 | |
*** alexchadin has quit IRC | 13:44 | |
openstackgerrit | Sorin Sbarnea proposed openstack-infra/project-config master: jenkins-job-builder change docs job https://review.openstack.org/578802 | 13:45 |
openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Remove credentials from urls in merger log https://review.openstack.org/578803 | 13:45 |
mordred | ssbarnea[m]: I believe you're right - and I think that's a bug in publish-openstack-sphinx-docs-infra | 13:47 |
*** andreas_s has joined #openstack-infra | 13:49 | |
*** hongbin has joined #openstack-infra | 13:50 | |
*** rlandy|afk is now known as rlandy | 13:50 | |
openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Don't use requirements/constraints for infra docs https://review.openstack.org/578807 | 13:52 |
mordred | fungi, AJaeger, ssbarnea[m]: ^^ something like that | 13:52 |
*** heyongli has quit IRC | 13:52 | |
*** heyongli has joined #openstack-infra | 13:52 | |
*** alexchadin has joined #openstack-infra | 13:54 | |
openstackgerrit | Sorin Sbarnea proposed openstack-infra/project-config master: Don't use requirements/constraints for infra docs https://review.openstack.org/578807 | 13:57 |
*** andreas_s has quit IRC | 13:58 | |
*** kgiusti has joined #openstack-infra | 13:58 | |
*** andreas_s has joined #openstack-infra | 13:59 | |
*** ykarel is now known as ykarel|afk | 13:59 | |
*** bobh has joined #openstack-infra | 13:59 | |
*** agopi has joined #openstack-infra | 14:00 | |
*** cshastri has quit IRC | 14:00 | |
*** Guest88823 is now known as dansmith | 14:01 | |
*** e0ne has joined #openstack-infra | 14:02 | |
*** annp_ has joined #openstack-infra | 14:02 | |
*** heyongli has quit IRC | 14:02 | |
*** lpetrut has quit IRC | 14:03 | |
*** heyongli has joined #openstack-infra | 14:03 | |
*** ykarel|afk has quit IRC | 14:04 | |
*** esarault has joined #openstack-infra | 14:05 | |
*** esarault_ has joined #openstack-infra | 14:06 | |
*** andreas_s has quit IRC | 14:08 | |
*** esarault has quit IRC | 14:09 | |
*** andreas_s has joined #openstack-infra | 14:10 | |
*** andreas_s has quit IRC | 14:10 | |
*** lbragstad has quit IRC | 14:10 | |
*** andreas_s has joined #openstack-infra | 14:11 | |
fungi | anybody happen to know why we use the term "core developer" instead of "core reviewer" in our zuul pipeline descriptions? | 14:12 |
ssbarnea[m] | mordred: see https://review.openstack.org/#/c/578807/2/zuul.d/jobs.yaml@253 | 14:12 |
*** heyongli has quit IRC | 14:12 | |
*** heyongli has joined #openstack-infra | 14:13 | |
*** apetrich has quit IRC | 14:16 | |
*** andreas_s has quit IRC | 14:17 | |
*** dhajare has quit IRC | 14:17 | |
*** zhangfei has joined #openstack-infra | 14:17 | |
mordred | fungi: historical accidents? | 14:18 |
*** esarault has joined #openstack-infra | 14:19 | |
*** andreas_s has joined #openstack-infra | 14:20 | |
*** esarault_ has quit IRC | 14:21 | |
*** quiquell|rover is now known as quiquell|off | 14:21 | |
fungi | mordred: i'm tempted to clean up those mentions but they've propagated from project-config to system-config and zuul so i couldn't tell if it was just reuse of some original prose or something more intentional | 14:22 |
*** andreas_s has quit IRC | 14:22 | |
*** andreas_s has joined #openstack-infra | 14:23 | |
*** heyongli has quit IRC | 14:23 | |
*** heyongli has joined #openstack-infra | 14:23 | |
openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Don't use requirements/constraints for infra docs https://review.openstack.org/578807 | 14:25 |
mordred | ssbarnea[m]: good catch - thanks! | 14:25 |
*** hemna_ has joined #openstack-infra | 14:25 | |
*** jamesmcarthur has joined #openstack-infra | 14:25 | |
mordred | fungi: I think it's just propagation | 14:25 |
*** hemna_ has quit IRC | 14:25 | |
mordred | fungi: I happen to have an old copy of the old layout file with this: | 14:26 |
mordred | "description: Changes that have been approved by core developers are enqueued in order in this pipeline, and if they pass tests in Jenkins, will be merged." | 14:26 |
*** haleyb has quit IRC | 14:27 | |
*** rnoriega has quit IRC | 14:27 | |
*** weshay|ruck has quit IRC | 14:27 | |
*** _ari_ has quit IRC | 14:27 | |
*** dpawlik has quit IRC | 14:27 | |
*** _ari_ has joined #openstack-infra | 14:28 | |
fungi | yeah, i'll push some changes to clean it up in that case. just did a similar one for the project-team-guide | 14:28 |
*** haleyb has joined #openstack-infra | 14:28 | |
*** rnoriega has joined #openstack-infra | 14:28 | |
*** weshay has joined #openstack-infra | 14:28 | |
Diabelko | hello again | 14:30 |
Diabelko | might be a little bit weird question, but bear with me | 14:30 |
Diabelko | how many physical nodes does OpenStack's CI have? | 14:30 |
Diabelko | I mean, how big are the clusters that you spawn slaves on | 14:30 |
fungi | we have no idea | 14:30 |
fungi | we don't operate them | 14:30 |
fungi | service providers donate accounts to us | 14:31 |
fungi | all we know is the quotas they've set for us | 14:31 |
Diabelko | mhm | 14:31 |
fungi | in most cases they're just accounts in public "clouds" | 14:31 |
*** e0ne has quit IRC | 14:31 | |
Diabelko | sure, thanks | 14:32 |
fungi | and so the hardware is shared across all their customers | 14:32 |
Diabelko | and the reason I asked it's because someone was interested in having an openstack cluster with 2k nodes | 14:32 |
Diabelko | and supposedly all of that for the CI | 14:32 |
*** CrayZee has joined #openstack-infra | 14:33 | |
Diabelko | and I'm trying to check if Zuul and Nodepool can handle that ;) | 14:33 |
fungi | sounds pretty massive | 14:33 |
mordred | yes | 14:33 |
mordred | zuul and nodepool can definitely handle it :) | 14:33 |
*** heyongli has quit IRC | 14:33 | |
Diabelko | awesome | 14:33 |
Diabelko | I'll print out that quote and hang at the wall :P | 14:34 |
*** heyongli has joined #openstack-infra | 14:34 | |
mordred | but it's a *lot* of capacity - for comparison, 2k concurrent vms is about our high-water mark currently - and those are 8G vms typically with about 4vcpus | 14:34 |
fungi | yeah, the zuul/nodepool control plane are designed to be scaled out horizontally as needed (zuul executors and mergers, nodepool launchers and builders) | 14:34 |
*** annp_ has quit IRC | 14:35 | |
*** Swami has joined #openstack-infra | 14:35 | |
Diabelko | Tungsten Fabric's (OpenContrail's) CI right now has limits for 70 VMs | 14:35 |
Diabelko | and we rarely hit that | 14:35 |
Diabelko | so there was no need to actually increase the limits for now | 14:35 |
Diabelko | totally different scale :) | 14:36 |
Diabelko | again, thank you guys ;) | 14:36 |
*** Swami_ has joined #openstack-infra | 14:37 | |
*** weshay has quit IRC | 14:37 | |
*** myoung has quit IRC | 14:37 | |
*** _ari_ has quit IRC | 14:37 | |
*** pabelanger has quit IRC | 14:37 | |
*** rasca has quit IRC | 14:37 | |
*** weshay has joined #openstack-infra | 14:38 | |
*** pabelanger has joined #openstack-infra | 14:38 | |
*** lbragstad has joined #openstack-infra | 14:38 | |
*** lpetrut has joined #openstack-infra | 14:38 | |
*** _ari_ has joined #openstack-infra | 14:38 | |
*** rasca has joined #openstack-infra | 14:39 | |
*** Swami has quit IRC | 14:39 | |
*** e0ne has joined #openstack-infra | 14:40 | |
*** shardy has quit IRC | 14:42 | |
*** heyongli has quit IRC | 14:43 | |
*** heyongli has joined #openstack-infra | 14:44 | |
*** myoung has joined #openstack-infra | 14:44 | |
openstackgerrit | Merged openstack-infra/zuul master: Remove credentials from urls in merger log https://review.openstack.org/578803 | 14:44 |
*** shardy has joined #openstack-infra | 14:45 | |
*** numans_ has joined #openstack-infra | 14:45 | |
openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Consolidate navigation functions https://review.openstack.org/578385 | 14:46 |
*** kiennt26 has joined #openstack-infra | 14:47 | |
openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: WIP: Fix for pools with diff labels https://review.openstack.org/578831 | 14:49 |
aspiers | anyone know what's up with bugs.launchpad.net? I get a timeout when I try to file a bug | 14:51 |
frickler | aspiers: most of the time that disappears if you retry after a few minutes | 14:51 |
*** Swami_ has quit IRC | 14:51 | |
aspiers | I did retry after a few mins :-/ | 14:52 |
frickler | aspiers: if it persists, try opening the bug with a short subject and edit it later | 14:52 |
frickler | aspiers: if that also doesn't help, take the error-code you get in the timeout and ask on #launchpad | 14:52 |
aspiers | I'm not sure how a short subject would help? I literally can't file a bug at all | 14:53 |
aspiers | I'll try #launchpad, thanks | 14:53 |
frickler | aspiers: launchpad searches for existing bugs that match your subject initially. it may timeout if that search gets too complex | 14:53 |
*** heyongli has quit IRC | 14:53 | |
*** heyongli has joined #openstack-infra | 14:54 | |
*** cfriesen has joined #openstack-infra | 14:54 | |
aspiers | no, that search succeeded | 14:54 |
fungi | aspiers: sometimes it has to do with which project you're trying to create a bugtask against too. sometimes it throws its hands up in disgust at write operations involving projects with a large number of subscribers | 14:54 |
aspiers | it was the submission which failed | 14:54 |
aspiers | fungi: I was submitting to masakari, so ... :-) | 14:54 |
fungi | ahh, yeah masakari is probably not one of those | 14:55 |
aspiers | I've reported in #launchpad | 14:55 |
fungi | i've seen it with neutron and nova bugs | 14:55 |
fungi | projects with somewhere around a thousand subscribers | 14:55 |
*** alex_xu has quit IRC | 14:58 | |
aspiers | got a reply on #launchpad: wait even longer (e.g. 10 mins) | 14:58 |
*** CrayZee is now known as snapiri- | 15:00 | |
*** gongysh has quit IRC | 15:00 | |
*** alex_xu has joined #openstack-infra | 15:00 | |
openstackgerrit | Merged openstack/diskimage-builder master: Fix bootloader for efi on rhel systems https://review.openstack.org/576041 | 15:02 |
*** duke has joined #openstack-infra | 15:02 | |
*** duke has left #openstack-infra | 15:02 | |
*** Swami has joined #openstack-infra | 15:03 | |
*** heyongli has quit IRC | 15:04 | |
*** heyongli has joined #openstack-infra | 15:04 | |
*** alexchadin has quit IRC | 15:06 | |
openstackgerrit | Merged openstack-infra/project-config master: jenkins-job-builder change docs job https://review.openstack.org/578802 | 15:07 |
*** bobh has quit IRC | 15:07 | |
*** rpioso|afk is now known as rpioso | 15:08 | |
*** kiennt26 has quit IRC | 15:10 | |
*** hamzy has quit IRC | 15:11 | |
*** yamamoto has quit IRC | 15:12 | |
*** yamamoto has joined #openstack-infra | 15:12 | |
*** heyongli has quit IRC | 15:14 | |
*** heyongli has joined #openstack-infra | 15:14 | |
AJaeger | mordred: I don't think 578807 will work - if the constraints file is missing, the roles break. Don't they? | 15:15 |
mordred | AJaeger: they shouldn't - build-sphinx-docs does not require a constraints file, since that is an openstack-specific thing | 15:15 |
*** e0ne has quit IRC | 15:15 | |
mriedem | hmm, new pbr release? http://logs.openstack.org/32/560832/11/check/tempest-full-py3/79f5de2/job-output.txt.gz#_2018-06-28_13_29_30_784225 | 15:15 |
mriedem | nope | 15:16 |
mordred | AJaeger: http://git.openstack.org/cgit/openstack-infra/zuul-jobs/tree/roles/install-if-python/tasks/main.yaml#n33 | 15:16 |
AJaeger | mordred: indeed, let's try... | 15:19 |
AJaeger | mordred: I overlooked one bit in my first investigation... | 15:19 |
*** Shrews has quit IRC | 15:20 | |
*** kiennt26 has joined #openstack-infra | 15:23 | |
*** caphrim007 has joined #openstack-infra | 15:23 | |
*** gyee has joined #openstack-infra | 15:24 | |
*** heyongli has quit IRC | 15:24 | |
*** heyongli has joined #openstack-infra | 15:25 | |
*** esarault_ has joined #openstack-infra | 15:25 | |
*** Shrews has joined #openstack-infra | 15:26 | |
*** caphrim007 has quit IRC | 15:26 | |
*** caphrim007 has joined #openstack-infra | 15:26 | |
*** caphrim007 has quit IRC | 15:27 | |
*** caphrim007 has joined #openstack-infra | 15:27 | |
*** esarault has quit IRC | 15:29 | |
*** jamesmcarthur has quit IRC | 15:31 | |
*** heyongli has quit IRC | 15:34 | |
*** heyongli has joined #openstack-infra | 15:35 | |
*** zhangfei has quit IRC | 15:35 | |
*** lpetrut has quit IRC | 15:36 | |
*** lpetrut has joined #openstack-infra | 15:37 | |
*** jamesmcarthur has joined #openstack-infra | 15:38 | |
openstackgerrit | Merged openstack-infra/zuul master: Don't add implied branch matchers to project-pipeline variants https://review.openstack.org/577881 | 15:39 |
*** lxkong has quit IRC | 15:39 | |
*** hamzy has joined #openstack-infra | 15:40 | |
*** andreas_s has quit IRC | 15:42 | |
*** zoli is now known as zoli|gone | 15:43 | |
*** zoli|gone is now known as zoli | 15:44 | |
openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Fix typos in docs https://review.openstack.org/578854 | 15:44 |
*** heyongli has quit IRC | 15:45 | |
*** ykarel has joined #openstack-infra | 15:45 | |
*** heyongli has joined #openstack-infra | 15:45 | |
*** Shrews has quit IRC | 15:45 | |
*** iranzo has quit IRC | 15:46 | |
*** Shrews has joined #openstack-infra | 15:47 | |
*** pbourke has quit IRC | 15:50 | |
*** dpawlik has joined #openstack-infra | 15:51 | |
*** hashar is now known as hasharAway | 15:52 | |
*** pbourke has joined #openstack-infra | 15:52 | |
smcginnis | I've added ACLs and other things for the operations-guide repo, but when I try to submit a patch I get an error that it is read-only. | 15:53 |
smcginnis | Is there somewhere else I need to make a change? | 15:53 |
fungi | smcginnis: i wonder if the new acl doesn't actually turn off the read-only flag and leaves it in whatever state it found it | 15:54 |
fungi | we don't often un-retire repos so this may be slightly uncharted territory | 15:55 |
*** heyongli has quit IRC | 15:55 | |
smcginnis | fungi: Any idea what I can check or where to go from here? | 15:55 |
*** heyongli has joined #openstack-infra | 15:55 | |
fungi | i can manually alter the read-only state of the project in gerrit, but it would be good to figure out how to make the manage-projects script do that on its own | 15:55 |
*** udesale has quit IRC | 15:56 | |
*** dpawlik has quit IRC | 15:56 | |
*** udesale has joined #openstack-infra | 15:56 | |
ssbarnea[m] | can we inject some life in gerritlib? I have one review that is >1 year old, https://review.openstack.org/#/c/451992/ | 15:57 |
openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Remove obsolete zuul and jenkins dev configs https://review.openstack.org/578865 | 15:58 |
openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: There are no core developers, only core reviewers https://review.openstack.org/578866 | 15:58 |
corvus | fungi, smcginnis: i'm guessing it's a matter of explicitly setting the state attribute rather than just removing it from the acl file. | 15:58 |
fungi | corvus: yeah, that's my assumption as well, i just need to free up a minute to look | 16:00 |
*** esarault has joined #openstack-infra | 16:00 | |
smcginnis | It can definitely wait until after TC office hours. | 16:00 |
fungi | oh look, they just ended ;) | 16:00 |
TheJulia | So, how do we go about changing the irc channel topics? | 16:01 |
TheJulia | At least for ones owned by openstackinfra and the openstack bot | 16:02 |
*** amoralej is now known as amoralej|off | 16:02 | |
*** esarault_ has quit IRC | 16:02 | |
fungi | TheJulia: you do it through chanserv | 16:03 |
fungi | TheJulia: if you're on the chanserv access list for the channel and have the +t permission granted, you can /msg chanserv topic #mychannel This is our channel topic | 16:03 |
TheJulia | fungi: looks like I'm not | 16:04 |
TheJulia | :\ | 16:04 |
fungi | TheJulia: which channel? | 16:04 |
openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Don't use requirements/constraints for infra docs https://review.openstack.org/578807 | 16:04 |
TheJulia | #openstack-ironic | 16:04 |
*** lpetrut has quit IRC | 16:05 | |
*** heyongli has quit IRC | 16:05 | |
fungi | TheJulia: seeing as you're the ironic ptl, i'm just going to add you | 16:05 |
TheJulia | fungi: thanks! | 16:05 |
*** heyongli has joined #openstack-infra | 16:06 | |
*** slaweq has quit IRC | 16:06 | |
fungi | TheJulia: done, you now have the same access as your predecessors | 16:06 |
TheJulia | \o/ | 16:06 |
fungi | we can also add anyone else you want to be a channel operator | 16:07 |
fungi | just let us know | 16:07 |
openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Consolidate navigation functions https://review.openstack.org/578385 | 16:07 |
fungi | smcginnis: i need to go run some errands, but can look at the repo state as soon as i get back. should be less than an hour | 16:07 |
*** bobh has joined #openstack-infra | 16:08 | |
*** shardy has quit IRC | 16:08 | |
smcginnis | fungi: Thanks, take your time. | 16:08 |
*** ccamacho has quit IRC | 16:09 | |
dtantsur | fungi: oh, fun, I did not know that when I was the PTL :) | 16:09 |
smcginnis | I thought channel topics were periodically reset from something listed in a yaml file somewhere. | 16:10 |
*** florianf has quit IRC | 16:10 | |
TheJulia | smcginnis: I was thinking the same, but at the same time I was also wondering "Wait, I would have likely had seen this changed at some point...." if it was | 16:11 |
openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: web: fix status page flickering https://review.openstack.org/578226 | 16:11 |
mgagne | what's the best way to change my main email address in LP and not break Gerrit and friends? | 16:12 |
*** panda is now known as panda|off | 16:12 | |
AJaeger | config-core, could you review https://review.openstack.org/#/c/578574/ and https://review.openstack.org/#/c/578701 to finalize removing of system-required template from project-config, please? | 16:12 |
*** ccamacho has joined #openstack-infra | 16:14 | |
*** heyongli has quit IRC | 16:15 | |
*** heyongli has joined #openstack-infra | 16:16 | |
*** d0ugal has quit IRC | 16:17 | |
*** kiennt26 has quit IRC | 16:18 | |
clarkb | TheJulia: smcginnis our bot resets topics during outages if we ask it too you then sets them back to what they were on the all clear | 16:19 |
openstackgerrit | Matt Riedemann proposed openstack-infra/project-config master: Remove ironic-tempest-dsvm-ipa-wholedisk-bios-agent_ipmitool-tinyipa for nova https://review.openstack.org/578879 | 16:19 |
clarkb | mgagne: I am not sure there is a way? I want to say changing the primary email address on an lp account results in a new openid which will confuse gerrit. fungi probably remembers best having debugging and understood that all at the time it was causing us much trouble | 16:20 |
mgagne | yes, my concern is the change with openid | 16:21 |
mriedem | mgagne: when i left ibm i had to change launchpad from mriedem@us.ibm.com to mriedem.os@gmail.com | 16:21 |
*** jpich has quit IRC | 16:21 | |
mriedem | i have an email from fungi if i can find it | 16:21 |
*** Shrews has quit IRC | 16:21 | |
mgagne | mriedem: I can't change mine to mriedem.os@gmail.com too :D | 16:22 |
stephenfin | Question: do zuul jobs support nodes and anchors? https://anil.io/blog/symfony/yaml/using-variables-in-yaml-files/ | 16:22 |
smcginnis | Yeah, I think you just need to add the new email address to your LP settings. | 16:22 |
stephenfin | To avoid needing to duplicate stuff | 16:22 |
smcginnis | stephenfin: I played around with that awhile back. I believe so. | 16:22 |
clarkb | stephenfin: they should, both zuul and ansible use pyyaml which supports them | 16:22 |
stephenfin | Excellent. Thanks smcginnis, clarkb :) | 16:22 |
* stephenfin goes to experiment | 16:23 | |
mgagne | I got the new email as an alternative email too. I want to change my main to be more neutral. So I guess if mriedem can find that email, that would be great. :P | 16:23 |
corvus | stephenfin: for zuul.yaml files, note that the scope is only within a single file (you can't reference one defined in another file) | 16:23 |
corvus | but otherwise yes | 16:23 |
*** Shrews has joined #openstack-infra | 16:23 | |
mriedem | stephenfin: i'll make that update | 16:24 |
stephenfin | mriedem: I was just about to ask. Go for it | 16:24 |
mriedem | mgagne: unfortunately i'm not seeing it, probably deleted it and thunderbird only keeps it around for a year i think (and this was over a year ago) | 16:24 |
mriedem | mgagne: but yeah you can add your new email here https://launchpad.net/~mriedem and sign in with it as your primary | 16:25 |
*** slaweq has joined #openstack-infra | 16:25 | |
*** janki has quit IRC | 16:25 | |
mriedem | well, for your profile of course :) | 16:25 |
pabelanger | do we happen to gather the complete output of syslog (on node) from a job? Trying to compare something we are seeing in rdo about multiple connections to a job from zuul-executor | 16:25 |
mgagne | might be the same procedure? https://docs.openstack.org/infra/system-config/gerrit.html#duplicate-accounts-in-gerrit or https://docs.openstack.org/infra/system-config/gerrit.html#combining-gerrit-accounts ? | 16:25 |
pabelanger | if not, I should be able to quickly write something | 16:25 |
mriedem | interestingly, openid loging for launchpad still shows my us.ibm.com email | 16:25 |
mgagne | which will require help from infra team | 16:26 |
*** heyongli has quit IRC | 16:26 | |
*** heyongli has joined #openstack-infra | 16:26 | |
mriedem | https://review.openstack.org/#/settings/web-identities to change emails in gerrit | 16:26 |
mriedem | https://review.openstack.org/#/settings/contact | 16:26 |
*** raopajay has quit IRC | 16:26 | |
mgagne | yep, got that one sorted out. I guess what's left is changing my openid email | 16:28 |
corvus | pabelanger: devstack does, regular jobs don't | 16:28 |
*** Shrews has quit IRC | 16:29 | |
mgagne | I guess I'm just gonna update it and see how bad it breaks | 16:29 |
clarkb | mgagne: let us know either way (as it would be helpful to know fr the future) | 16:29 |
pabelanger | corvus: thanks | 16:30 |
clarkb | pabelanger: corvus: devstack grabs the entire journal too | 16:30 |
*** udesale has quit IRC | 16:31 | |
pabelanger | yup, I found one thanks | 16:33 |
*** auristor has quit IRC | 16:34 | |
*** ramishra has quit IRC | 16:34 | |
stephenfin | Another question for the audience: is there a local linter/client we can use to validate our zuul job definitions locally? | 16:36 |
*** heyongli has quit IRC | 16:36 | |
*** heyongli has joined #openstack-infra | 16:36 | |
*** eharney has quit IRC | 16:37 | |
*** tesseract has quit IRC | 16:38 | |
mgagne | so it appears that you can change your main email address without infra involvement if you do it in the right order | 16:38 |
mgagne | I think that previous users changed it in Ubuntu One/LP and wondered why Gerrit didn't detect it when in fact, they didn't add the new email to Gerrit first. | 16:40 |
clarkb | stephenfin: not currently, proper checking requires loading the entire global state and the zuul service is what has that and we've not sorted out an api to query it for linting locally (or some other method) | 16:40 |
clarkb | mgagne: ah | 16:40 |
stephenfin | clarkb: Makes sense. Thanks for the info | 16:41 |
*** apetrich has joined #openstack-infra | 16:42 | |
*** Shrews has joined #openstack-infra | 16:44 | |
*** jbadiapa has quit IRC | 16:44 | |
*** auristor has joined #openstack-infra | 16:45 | |
*** heyongli has quit IRC | 16:46 | |
*** heyongli has joined #openstack-infra | 16:47 | |
*** jamesmcarthur has quit IRC | 16:48 | |
*** trown is now known as trown|lunch | 16:49 | |
*** eharney has joined #openstack-infra | 16:50 | |
*** salv-orlando has joined #openstack-infra | 16:51 | |
notmyname | where's the right place to ask questions or generally talk about dev work in bindep? | 16:52 |
*** jamesmcarthur has joined #openstack-infra | 16:53 | |
*** Swami has quit IRC | 16:53 | |
*** rh-jelabarre has quit IRC | 16:53 | |
clarkb | notmyname: probably here or on the infra mailing list | 16:54 |
notmyname | clarkb: ok, thanks | 16:54 |
*** yamahata has quit IRC | 16:54 | |
*** salv-orlando has quit IRC | 16:56 | |
*** heyongli has quit IRC | 16:56 | |
*** heyongli has joined #openstack-infra | 16:57 | |
*** baoli has joined #openstack-infra | 16:58 | |
*** ramishra has joined #openstack-infra | 16:59 | |
*** derekh has quit IRC | 17:00 | |
*** eernst has joined #openstack-infra | 17:00 | |
*** Shrews has quit IRC | 17:00 | |
openstackgerrit | Merged openstack-infra/gerritlib master: Updated list of Python versions in tox https://review.openstack.org/451992 | 17:01 |
*** jpena is now known as jpena|off | 17:01 | |
*** baoli has quit IRC | 17:02 | |
openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Translate zuulStartStream into typescript https://review.openstack.org/558618 | 17:02 |
openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Shift log streaming code into StreamComponent https://review.openstack.org/558619 | 17:02 |
*** Shrews has joined #openstack-infra | 17:03 | |
*** dpawlik has joined #openstack-infra | 17:04 | |
*** gfidente is now known as gfidente|afk | 17:06 | |
*** heyongli has quit IRC | 17:07 | |
*** heyongli has joined #openstack-infra | 17:07 | |
*** dpawlik has quit IRC | 17:09 | |
*** Shrews has quit IRC | 17:09 | |
*** jaosorior has quit IRC | 17:10 | |
dhellmann | corvus : do you think you'll have some time to talk about https://etherpad.openstack.org/p/goal-python3-first tomorrow? | 17:10 |
*** electrofelix has quit IRC | 17:10 | |
*** eernst has quit IRC | 17:11 | |
*** Shrews has joined #openstack-infra | 17:11 | |
*** alexchadin has joined #openstack-infra | 17:12 | |
*** yamahata has joined #openstack-infra | 17:13 | |
*** Shrews has quit IRC | 17:16 | |
*** Shrews has joined #openstack-infra | 17:17 | |
*** heyongli has quit IRC | 17:17 | |
*** heyongli has joined #openstack-infra | 17:17 | |
*** ccamacho has quit IRC | 17:18 | |
*** alexchadin has quit IRC | 17:19 | |
*** lpetrut has joined #openstack-infra | 17:20 | |
mgagne | clarkb: the best I could do for documentation: https://gist.github.com/mgagne/f8c42692f83995c9f4b413e8dd4e1767 | 17:20 |
mgagne | I had already done some steps in the past so I'm not sure if ALL are required but it worked for me | 17:20 |
fungi | clarkb: mgagne: changing which e-mail address you use to authenticate to login.ubuntu.com is what i think ends up causing you to get a different openid. updating which address is primary in lp should _probably_ have no impact, as long as that's a separate configuration? | 17:22 |
*** dtantsur is now known as dtantsur|afk | 17:22 | |
mgagne | I think you should add it to Gerrit too so it can found your current account? | 17:22 |
*** bobh_ has joined #openstack-infra | 17:23 | |
*** bobh has quit IRC | 17:23 | |
mgagne | otherwise you just end up with the same old problem where one has a new account in Gerrit and can't find its old changes and you need to merge both accounts. | 17:24 |
*** SumitNaiksatam has joined #openstack-infra | 17:24 | |
fungi | mgagne: _if_ you wind up with a new openid, then yes we have to manually alter the db to associate that with your old account and mark the newly autocreated account inactive so you don't end up with query conflicts in odd places | 17:25 |
mgagne | fungi: you DO end up with a new openid | 17:26 |
*** ramishra has quit IRC | 17:26 | |
mgagne | or not? | 17:26 |
openstackgerrit | Brian Haley proposed openstack-infra/devstack-gate master: Support an IPv6 underlay network https://review.openstack.org/343041 | 17:27 |
fungi | mgagne: if you change the address you use with login.ubuntu.com then i _think_ you end up with a new openid | 17:27 |
mgagne | wut... same openid but different domain: login.ubuntu.com vs login.launchpad.net | 17:27 |
*** heyongli has quit IRC | 17:27 | |
fungi | login.launchpad.net is also login.ubuntu.com, just the former name is deprecated | 17:27 |
clarkb | mgagne: login.ubuntu.com is the canonical (no pun intended) location now aiui | 17:27 |
mgagne | so it's all fine, same openid. just need to make sure Gerrit has your new email prior to change. | 17:27 |
*** alexchadin has joined #openstack-infra | 17:28 | |
fungi | ubuntu sso isn't technically part of launchpad, just used to share a domain name with it | 17:28 |
*** heyongli has joined #openstack-infra | 17:28 | |
*** e0ne has joined #openstack-infra | 17:28 | |
*** e0ne has quit IRC | 17:28 | |
fungi | i've been instructed that they're entirely separate software/services, launchpad just happens to _use_ the sso | 17:29 |
mgagne | because I don't think LP/Ubuntu One shares your alternative emails during the login process. | 17:29 |
mgagne | which could confuse Gerrit otherwise | 17:29 |
mgagne | right | 17:29 |
mgagne | so add new email to openid consumer sites first and then update ubuntu one. | 17:30 |
*** esarault_ has joined #openstack-infra | 17:31 | |
openstackgerrit | Brian Haley proposed openstack-infra/devstack-gate master: Support an IPv6 underlay network https://review.openstack.org/343041 | 17:31 |
*** eharney_ has joined #openstack-infra | 17:32 | |
*** ykarel has quit IRC | 17:32 | |
*** bobh_ has quit IRC | 17:33 | |
*** alexchadin has quit IRC | 17:33 | |
*** bobh has joined #openstack-infra | 17:33 | |
*** esarault has quit IRC | 17:33 | |
openstackgerrit | Merged openstack-infra/zuul master: Revert "Revert "Upgrade from angularjs (v1) to angular (v6)"" https://review.openstack.org/578384 | 17:33 |
openstackgerrit | Merged openstack-infra/zuul master: Use window.location.href for the href not pathname https://review.openstack.org/578243 | 17:33 |
*** bobh_ has joined #openstack-infra | 17:33 | |
*** tonytan4ever_brb has joined #openstack-infra | 17:34 | |
*** eharney has quit IRC | 17:35 | |
*** eharney_ is now known as eharney | 17:35 | |
*** tonytan4ever has quit IRC | 17:37 | |
fungi | mgagne: good to know that works--thanks for testing! | 17:37 |
*** heyongli has quit IRC | 17:37 | |
*** bobh has quit IRC | 17:38 | |
*** heyongli has joined #openstack-infra | 17:38 | |
*** jesslampe has joined #openstack-infra | 17:39 | |
*** jamesmcarthur has quit IRC | 17:40 | |
fungi | smcginnis: so, for starters, i've confirmed that https://review.openstack.org/#/admin/projects/openstack/operations-guide is read-only | 17:43 |
fungi | smcginnis: _however_ i think the fact that https://review.openstack.org/#/admin/projects/openstack/operations-guide,access is empty points to the problem | 17:44 |
smcginnis | fungi: Any automation that flips that switched? | 17:44 |
fungi | specifically, the way manage-projects updates configuration for a project is to push the updated configuration into it | 17:44 |
fungi | and if a project is read-only, it can't do that | 17:44 |
fungi | i'm starting to remember this catch-22 came up at least once in the past | 17:44 |
*** Shrews has quit IRC | 17:45 | |
smcginnis | fungi: Ah, so my ACL changes couldn't actually be applied in gerrit because of that? | 17:45 |
fungi | and i think we decided at the time that it was too complex an issue to work around given the infrequency with which we encounter it | 17:45 |
fungi | so, yes, i think if i manually flip the switch then manage-projects will be able to push a working acl into it | 17:46 |
fungi | but yeah, basically it can make a project read-only by pushing a configuration update into it, but then once it's read-only it can't undo it the same way | 17:46 |
smcginnis | That's too bad it can't just be automated, but at least it does seem to be a simple and infrequent thing. | 17:46 |
fungi | we could automate this, if we wanted, by making calls to the api instead | 17:46 |
smcginnis | Probably more effort than it's worth. | 17:47 |
fungi | yeah, as i said, i think that's the conclusion we came to last time | 17:47 |
smcginnis | Unless that would make it better by not needing to remember these caveats on the rare occasions it happens. | 17:47 |
fungi | right ;) | 17:47 |
*** snapiri- has quit IRC | 17:47 | |
*** jesslampe has quit IRC | 17:47 | |
smcginnis | fungi: Is there anywhere that would make sense for me to push a documentation patch noting this? | 17:47 |
fungi | anyway, i've switched it back from read-only to active state | 17:47 |
fungi | smcginnis: that might be a sufficient compromise | 17:48 |
*** heyongli has quit IRC | 17:48 | |
fungi | maybe at/near the end of the retirement instructions we can have a notebox about it or something | 17:48 |
clarkb | smcginnis: fungi possibly a footnote on retiring a project for unretiring a project | 17:48 |
fungi | yeah | 17:48 |
*** heyongli has joined #openstack-infra | 17:48 | |
*** jamesmcarthur has joined #openstack-infra | 17:49 | |
smcginnis | OK, I'll see if I can draft some kind of breadcrumb at least to leave there to trigger everyone's memory next time it happens. | 17:49 |
smcginnis | fungi: Any next step actions needed from me to move ahead, or is this something you can take care of? | 17:49 |
fungi | so when manage-projects runs again this would correct itself, but seeing as how we haven't merged anything recently to trigger that i'll run it manually instead | 17:49 |
*** trown|lunch has quit IRC | 17:50 | |
smcginnis | Oh good, that was going to be my next question. | 17:50 |
smcginnis | I see state says active now. | 17:50 |
fungi | our mp caching is fighting me on this | 17:51 |
fungi | apparently because nothing with the acl has changed since the last time it ran, it doesn't want to do anything | 17:51 |
fungi | need to figure out a workaround with the cache, just a sec | 17:52 |
*** salv-orlando has joined #openstack-infra | 17:52 | |
fungi | i mangled the sha for it in the cache | 17:53 |
fungi | that seems to have worked | 17:53 |
fungi | https://review.openstack.org/#/admin/projects/openstack/operations-guide,access is now an acl | 17:53 |
openstackgerrit | Merged openstack-infra/project-config master: Don't use requirements/constraints for infra docs https://review.openstack.org/578807 | 17:53 |
fungi | smcginnis: lmk if you need anything else on this, but i think you should be all set now | 17:53 |
smcginnis | fungi: Thanks, it looks correct to me. | 17:54 |
smcginnis | I will try pushing up a patch for review. If you don't hear anything more from me you can assume all is good. | 17:54 |
*** lpetrut has quit IRC | 17:54 | |
*** salv-orlando has quit IRC | 17:56 | |
*** trown has joined #openstack-infra | 17:57 | |
*** Shrews has joined #openstack-infra | 17:58 | |
*** heyongli has quit IRC | 17:58 | |
*** heyongli has joined #openstack-infra | 17:58 | |
*** jamesmcarthur has quit IRC | 18:07 | |
*** alexchadin has joined #openstack-infra | 18:08 | |
*** jamesmcarthur has joined #openstack-infra | 18:08 | |
*** heyongli has quit IRC | 18:08 | |
openstackgerrit | Sean McGinnis proposed openstack-infra/infra-manual master: Add note about project read-only step for unretirement https://review.openstack.org/578954 | 18:08 |
*** heyongli has joined #openstack-infra | 18:09 | |
smcginnis | fungi, clarkb: We'll see if that helps. ^ | 18:09 |
fungi | thanks! | 18:10 |
*** jamesmcarthur has quit IRC | 18:10 | |
*** jamesmcarthur has joined #openstack-infra | 18:11 | |
*** SumitNaiksatam_ has joined #openstack-infra | 18:12 | |
*** alexchadin has quit IRC | 18:13 | |
*** SumitNaiksatam has quit IRC | 18:13 | |
*** SumitNaiksatam_ is now known as SumitNaiksatam | 18:13 | |
*** pcaruana has quit IRC | 18:14 | |
*** jamesmcarthur has quit IRC | 18:15 | |
*** heyongli has quit IRC | 18:18 | |
*** esarault has joined #openstack-infra | 18:19 | |
*** heyongli has joined #openstack-infra | 18:19 | |
*** SumitNaiksatam has quit IRC | 18:20 | |
*** huzefa has joined #openstack-infra | 18:21 | |
huzefa | Is this the right place to ask about DevStack? I'm trying to use osprofiler and having some problems | 18:22 |
clarkb | huzefa: #openstack-qa is probably more appropriate for devstack and osprofiler | 18:22 |
huzefa | thanks | 18:22 |
*** esarault_ has quit IRC | 18:22 | |
*** Shrews has quit IRC | 18:23 | |
*** yolanda_ has joined #openstack-infra | 18:25 | |
corvus | dhellmann: yes; my afternoon would probably be best | 18:26 |
*** jamesmcarthur has joined #openstack-infra | 18:27 | |
*** yolanda has quit IRC | 18:27 | |
openstackgerrit | Merged openstack-infra/zuul master: Consolidate navigation functions https://review.openstack.org/578385 | 18:29 |
*** heyongli has quit IRC | 18:29 | |
*** Shrews has joined #openstack-infra | 18:29 | |
*** jamesmcarthur has quit IRC | 18:29 | |
*** heyongli has joined #openstack-infra | 18:29 | |
dhellmann | corvus : you're west cost of US still, right? | 18:29 |
corvus | ya | 18:29 |
dhellmann | corvus : how about 13:00 your time, then? | 18:30 |
corvus | dhellmann: wfm | 18:30 |
*** Shrews_ has joined #openstack-infra | 18:32 | |
dhellmann | corvus : invitation incoming | 18:32 |
*** esarault has quit IRC | 18:34 | |
fungi | "west cost" is certainly an apt description of the bay area | 18:34 |
*** Shrews_ has quit IRC | 18:34 | |
*** Shrews has quit IRC | 18:35 | |
*** gfidente|afk has quit IRC | 18:37 | |
corvus | dhellmann: i feel like i'm involved in this because i can answer questions about how zuul will run jobs, but it's looking like the proposal may involve changing some of the job content, so it'd be great if we can also get some input from someone with more of a stake in them such as mordred, fungi, or clarkb. | 18:38 |
*** heyongli has quit IRC | 18:39 | |
*** heyongli has joined #openstack-infra | 18:39 | |
*** armaan has joined #openstack-infra | 18:40 | |
corvus | (a bunch of unrelated 20:00 utc meetings just got scheduled, didn't they? :) | 18:40 |
clarkb | corvus: dhellmann what day are we talking about for 2000UTC? | 18:41 |
corvus | tomorrow | 18:41 |
dhellmann | the latest draft of the plan is in https://etherpad.openstack.org/p/goal-python3-first for your review between now and then | 18:41 |
dhellmann | I can just drop by here if that's the easiest place to "meet" | 18:42 |
corvus | yeah. we can move to another channel if it's busy, but we might want to do some voice chatting too. | 18:42 |
dhellmann | oh, good point, I forgot you mentioned that | 18:43 |
dhellmann | are folks ok with a google hangout for that? | 18:43 |
corvus | not really. we do have a pbx server though. | 18:43 |
dhellmann | ok, that works then | 18:43 |
dhellmann | I don't know how to get to that, but I'm sure one of you can tell me :-) | 18:43 |
corvus | https://wiki.openstack.org/wiki/Infrastructure/Conferencing | 18:43 |
*** vivsoni_ has joined #openstack-infra | 18:44 | |
dhellmann | ok | 18:44 |
*** vivsoni has quit IRC | 18:44 | |
mordred | dhellmann: if I wanted to make comments on that plan - should I do them inline? or collect them somewhere else like the end? | 18:44 |
* mordred wants to do the thing that's the least disruptive to folks | 18:44 | |
dhellmann | mordred : inline is OK, that's part of why I moved it to an etherpad for now | 18:45 |
*** huzefa has left #openstack-infra | 18:45 | |
dhellmann | and I wiped the original color from the import so we could tell what mods are being made | 18:45 |
*** armaan has quit IRC | 18:45 | |
mordred | cool | 18:45 |
dhellmann | I tried to bold the actions I still need to take, fwiw | 18:45 |
*** Shrews has joined #openstack-infra | 18:45 | |
*** armaan has joined #openstack-infra | 18:45 | |
*** rfolco is now known as rfolco_afk | 18:46 | |
*** e0ne has joined #openstack-infra | 18:47 | |
*** heyongli has quit IRC | 18:49 | |
*** heyongli has joined #openstack-infra | 18:50 | |
*** e0ne has quit IRC | 18:50 | |
*** armaan has quit IRC | 18:50 | |
*** salv-orlando has joined #openstack-infra | 18:53 | |
mtreinish | corvus, clarkb, pabelanger, mordred: If you get a sec can you take a look at: https://review.openstack.org/#/q/status:open+topic:firehose-cyrus | 18:55 |
mtreinish | I'm keen to get lpmqtt fixed since it hasn't been emitting messges since dec. | 18:56 |
*** tonytan4ever_brb has quit IRC | 18:56 | |
*** tonytan4ever has joined #openstack-infra | 18:56 | |
*** jamesmcarthur has joined #openstack-infra | 18:56 | |
*** salv-orlando has quit IRC | 18:57 | |
*** heyongli has quit IRC | 18:59 | |
*** heyongli has joined #openstack-infra | 19:00 | |
clarkb | corvus: do you want to review mtreinish's ^ exim change? | 19:00 |
clarkb | er I guess fungi wrote it | 19:01 |
mtreinish | clarkb: heh, yeah they're fungi's patches :) | 19:02 |
*** jamesmcarthur has quit IRC | 19:03 | |
corvus | +5 | 19:03 |
clarkb | I got the second +w | 19:04 |
openstackgerrit | Merged openstack-infra/project-config master: Add kata runtime and tests repos to zuul https://review.openstack.org/578562 | 19:05 |
openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Fix for pools with different labels https://review.openstack.org/578831 | 19:05 |
openstackgerrit | Merged openstack-infra/zuul master: Fix typos in docs https://review.openstack.org/578854 | 19:09 |
*** heyongli has quit IRC | 19:10 | |
*** heyongli has joined #openstack-infra | 19:10 | |
*** heyongli has quit IRC | 19:20 | |
*** heyongli has joined #openstack-infra | 19:20 | |
*** rfolco_afk is now known as rfolco | 19:20 | |
*** rh-jelabarre has joined #openstack-infra | 19:23 | |
*** jesslampe has joined #openstack-infra | 19:27 | |
*** dpawlik has joined #openstack-infra | 19:28 | |
*** heyongli has quit IRC | 19:30 | |
*** heyongli has joined #openstack-infra | 19:31 | |
*** jcoufal has joined #openstack-infra | 19:31 | |
*** jcoufal has quit IRC | 19:31 | |
*** dpawlik has quit IRC | 19:33 | |
openstackgerrit | Merged openstack-infra/puppet-exim master: Make localuser router optional https://review.openstack.org/577893 | 19:33 |
*** Shrews has quit IRC | 19:33 | |
*** Shrews has joined #openstack-infra | 19:34 | |
*** Shrews has quit IRC | 19:35 | |
*** Shrews has joined #openstack-infra | 19:35 | |
*** jesslampe has quit IRC | 19:35 | |
openstackgerrit | Merged openstack-infra/system-config master: Check "cyrus" Exim router before "localuser" https://review.openstack.org/577895 | 19:36 |
*** bobh_ has quit IRC | 19:37 | |
*** eharney has quit IRC | 19:39 | |
*** heyongli has quit IRC | 19:40 | |
*** heyongli has joined #openstack-infra | 19:41 | |
weshay | anyone know why I would hit http://logs.openstack.org/13/528213/42/gate/tripleo-ci-centos-7-containerized-undercloud-upgrades/58315c1/logs/undercloud/home/zuul/undercloud_upgrade.log.txt.gz#_2018-06-28_18_27_52 | 19:41 |
weshay | 2018-06-28 18:27:52 | time="2018-06-28T18:27:51Z" level=fatal msg="pinging docker registry returned: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 127.0.0.1:53: no such host" | 19:41 |
weshay | http://logs.openstack.org/13/528213/42/gate/tripleo-ci-centos-7-containerized-undercloud-upgrades/58315c1/logs/undercloud/etc/resolv.conf.txt.gz | 19:41 |
weshay | using unbound | 19:42 |
fungi | does it happen regularly or only occasionally? | 19:42 |
weshay | occasionally.. but I fear infra's wrath re: gate failures | 19:42 |
weshay | :) | 19:42 |
fungi | yeah, one common explanation is that one of the authoritative nameservers for the domain was briefly returning a no-such-domain response for that record and so it got cached by one of the recursive resolvers being used as a forwarder | 19:43 |
pabelanger | does docker have a crazy low TTL on dns? | 19:44 |
*** Shrews has quit IRC | 19:44 | |
weshay | does it make sense to cache no-such-domain responses? | 19:44 |
pabelanger | I want to say 60 seconds | 19:45 |
clarkb | docker doesn't use unbound | 19:45 |
clarkb | iirc it will use google dns | 19:45 |
clarkb | because you can't talk to localhost port 53 on the host from a container | 19:45 |
fungi | pabelanger: it has a 60-second ttl on those records but a 1-day negative cache ttl | 19:45 |
fungi | weshay: the ietf thinks so, yes | 19:45 |
fungi | which is why there's a negative cache ttl field in every soa | 19:45 |
weshay | heh fair.. seems odd to me though | 19:45 |
openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-openstackci master: Add beaker tests for nodepool https://review.openstack.org/577522 | 19:46 |
pabelanger | clarkb: fungi: I thought the last time we discussed this, there was a setting in unbound we could override the 60sec TTL on the domain? | 19:46 |
fungi | it's so that authoritative nameservers don't get overrun with lookups for the same nonexistent record getting accidentally pounded over and over because of a typo somewhere | 19:46 |
pabelanger | to avoid the lookup on forwarders | 19:47 |
weshay | fungi, that makes sense | 19:47 |
fungi | pabelanger: probably, but as clarkb suggested docker ignores your actual nameserver configuration | 19:47 |
clarkb | ianw: good point on https://review.openstack.org/#/c/578153/4 I left some concerns with that approach, curious to know what you think of the laternatives I left | 19:47 |
clarkb | fungi: pabelanger yes docker ebhavior aiui is to use the host /etc/resolv.conf unless it has localhost listed then they just use google | 19:48 |
fungi | aha, and since we run unbound on localhost/loopback... | 19:48 |
pabelanger | maybe we need some firewall rule to rewrite 8.8.8.8 lookup in that case? | 19:48 |
pabelanger | or use public interface for nameserver in those cases | 19:49 |
weshay | code in question is.. http://git.openstack.org/cgit/openstack/tripleo-common/tree/tripleo_common/image/image_uploader.py#n324 | 19:49 |
clarkb | pabelanger: I don't think we should do that, we aren't the police, we provide resources software doesn't have to use them | 19:49 |
fungi | or bind a linklocal address to a virtual interface and have unbound listen on that in jobs which use docker | 19:49 |
clarkb | infra-root https://review.openstack.org/#/c/578556/ is an easy one that will help finish up my ssl work of the last week or so | 19:49 |
fungi | pabelanger: relying on the public interface for doing such things is problematic for a couple reasons | 19:49 |
weshay | so I don't think it's docker cli | 19:50 |
weshay | at least | 19:50 |
pabelanger | clarkb: right, but if jobs are failing because docker is ignoring our configuration, we likely need to workaround it | 19:50 |
pabelanger | fungi: yah, new virtual loopback might work too | 19:50 |
clarkb | pabelanger: I think the jobs need to workaround them if using broken software yes | 19:51 |
fungi | pabelanger: foremost because using the actual vm interface would make it too easy to accidentally start running an open recursive resolver on test nodes | 19:51 |
*** heyongli has quit IRC | 19:51 | |
clarkb | I don't think we can accomodate all broken software | 19:51 |
*** heyongli has joined #openstack-infra | 19:51 | |
pabelanger | fungi: +1 | 19:51 |
pabelanger | clarkb: ack | 19:51 |
clarkb | this is a long standing issue in docker, they added an opt in work around to docker itself you can use | 19:52 |
clarkb | you basically have to set up your networking the way lxc does it aiui | 19:52 |
clarkb | and docker runs a dns proxy effectively between the namespaces for you | 19:52 |
clarkb | but you have to configure docker to do that, I don't think infra can do that for every job that wants to use docker | 19:52 |
pabelanger | clarkb: do you remember where that code is for the rewrite to 8.8.8.8? | 19:53 |
*** ssbarnea[m] is now known as ssbarnea | 19:53 | |
clarkb | pabelanger: https://docs.docker.com/config/containers/container-networking/#ip-address-and-hostname look for "DNS services" | 19:53 |
clarkb | "If the container cannot reach any of the IP addresses you specify, Google’s public DNS server 8.8.8.8 is added, so that your container can resolve internet domains." | 19:54 |
pabelanger | thanks | 19:54 |
mwhahaha | but those types of requests aren't from within docker containers, it's docker itself | 19:55 |
mwhahaha | it's when we go to pull an image | 19:55 |
mwhahaha | so not sure if it's the same setting | 19:56 |
pabelanger | that was why I was thinking it had to do with low TTL on docker.com (60 seconds) unbound is only caching that for a short time, before hitting forwarding.conf servers again | 19:56 |
pabelanger | I thought dmsimard found a setting in unbound to override it | 19:57 |
pabelanger | but unsure if we actually landed any change for that | 19:57 |
dmsimard | we did | 19:57 |
clarkb | looks like it is using tcp for dns lookup I don't expect htat is a problem but is odd | 19:58 |
mwhahaha | http://logs.openstack.org/13/528213/42/gate/tripleo-ci-centos-7-containerized-undercloud-upgrades/58315c1/logs/undercloud/etc/unbound/conf.d/ttl.conf.txt.gz | 19:58 |
dmsimard | pabelanger: http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/commit/?id=f795128f21ba943a58ed7f257952c90b38a05f21 | 19:59 |
mwhahaha | of course the unbound logs are blank again | 19:59 |
dmsimard | pabelanger: http://codesearch.openstack.org/?q=unbound_cache&i=nope&files=&repos= | 19:59 |
fungi | clarkb: since that record is a largeish rr set, it may be in excess of the 512b limit for datagram responses | 19:59 |
fungi | i get 8 records back | 20:00 |
clarkb | ah | 20:00 |
clarkb | double check unbound is listening on tcp:53 maybe? pretty sure it is based on memory | 20:01 |
*** heyongli has quit IRC | 20:01 | |
*** heyongli has joined #openstack-infra | 20:01 | |
clarkb | tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 696/unbound | 20:02 |
clarkb | looks like it is | 20:02 |
*** vivsoni_ has quit IRC | 20:03 | |
*** vivsoni has joined #openstack-infra | 20:03 | |
clarkb | if that rr is dynamically populated using a healthcheck is it possible that none of the servers wree up for a short period an dthe record was empty? | 20:06 |
clarkb | we just happened to catch the upstream outage? | 20:06 |
fungi | also they're using amazon's dns farm. in my experience this sort of thing is almost always a glitch with one of the master nameservers briefly returning no-such-domain responses which end up getting cached for the duration of the negative cache ttl by a random set of intermediary recursive resolvers, resulting in inconsistent resolution for a while after the original event is resolved | 20:09 |
openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Add zuul.child_jobs in ansible inventory file https://review.openstack.org/578181 | 20:09 |
*** zul has quit IRC | 20:10 | |
*** heyongli has quit IRC | 20:11 | |
*** heyongli has joined #openstack-infra | 20:12 | |
fungi | if that's the case expect some percentage of lookup failures roughly proportional with the inverse of the number of authoritative servers (or some multiple of it) for anywhere from the negative cache ttl duration minus the record ttl, to the full negative cache ttl duration (in this case the two are basically the same given the disparity between the two values) | 20:14 |
fungi | when the record and negative cache ttls are closer, you tend to get a half-life sort of envelope | 20:15 |
weshay | k.. checking what we can do on our end.. if we think that's just a blip on the radar we'll hit from time to time | 20:21 |
weshay | thank you | 20:21 |
*** heyongli has quit IRC | 20:21 | |
*** heyongli has joined #openstack-infra | 20:22 | |
*** kgiusti has left #openstack-infra | 20:24 | |
clarkb | fungi: I'm still trying to sort out dynamic mtu value setting and ianw makes agreat point that my current setup isn't idempotent and subsequent overlays will have a lower mtu because the previous overlay had a lower mtu and we find the lowest mtu and subtract from that again... | 20:25 |
clarkb | fungi: I'm trying to determine if there is a good way to filter the interfaces on the host to only get the "real" ones | 20:25 |
clarkb | basically anything that models physical hardware, any idea if there is a good way to do that? | 20:26 |
clarkb | (I'm poking around in sysfs now and now finding a good answer to that) | 20:26 |
clarkb | glean checks the mac addr type (if it is permament or not) | 20:26 |
clarkb | maybe I will just do like glean to start | 20:28 |
*** kjackal has quit IRC | 20:31 | |
*** kjackal has joined #openstack-infra | 20:31 | |
*** heyongli has quit IRC | 20:32 | |
*** heyongli has joined #openstack-infra | 20:32 | |
*** tonytan4ever_brb has joined #openstack-infra | 20:37 | |
*** d0ugal has joined #openstack-infra | 20:38 | |
*** tonytan4ever has quit IRC | 20:39 | |
fungi | mmm | 20:39 |
fungi | well, the configdrive or metadata server will only list real interfaces, right? | 20:39 |
fungi | though that's not very friendly to people running not in a cloud | 20:39 |
fungi | *thinking cap on* | 20:40 |
fungi | do we care about interfaces besides those with a default route, in this scenario? | 20:40 |
fungi | just wondering whether the routing tables can be used as a clue | 20:40 |
clarkb | yes, rax is an example of where we do care about tha as we use the private network interface which isn't the default route interface | 20:41 |
clarkb | ianw suggested default route as a hint too but I don't think we can rely on that | 20:41 |
clarkb | however the glean test will work for the rax caes | 20:41 |
clarkb | I've got the glean test coded up now in zuul-jobs if we think that is a reasonable approximation | 20:41 |
fungi | i still wonder why we use the other interface in rax | 20:41 |
clarkb | fungi: it is to avoid NAT in clouds with floating IPs | 20:42 |
*** heyongli has quit IRC | 20:42 | |
clarkb | the private IP in rax is a separate interface (but typically the same interface on clouds with floating IPs) | 20:42 |
*** heyongli has joined #openstack-infra | 20:42 | |
corvus | to be clear, nothing about floating ips requires nat. we're avoiding nat in clouds with nat, right? :) | 20:43 |
clarkb | corvus: floating Ips are implemented using NAT in neutron. floating ips imply nat always aiui | 20:43 |
stevebaker | weshay: hey. yeah if skopeo inspects are failing lets add a retry decorator. I think its the only thing in the uploader code which doesn't have it now | 20:43 |
weshay | stevebaker, rock on | 20:44 |
corvus | clarkb: :( | 20:45 |
*** camunoz has quit IRC | 20:45 | |
openstackgerrit | Clark Boylan proposed openstack-infra/zuul-jobs master: Dynamically determine overlay network mtu https://review.openstack.org/578153 | 20:48 |
clarkb | that is what the glean filter looks like | 20:48 |
*** heyongli has quit IRC | 20:52 | |
corvus | clarkb: that almost looks like it wants to be a module :) | 20:52 |
*** heyongli has joined #openstack-infra | 20:53 | |
clarkb | corvus: possibly, I tend to find those harder to debug personally (I have tested this ssh script just by running it locally) | 20:53 |
corvus | clarkb: oh, you should take a look at what i did in devstack -- i made a unit test framework for the localrc module | 20:53 |
corvus | clarkb: to be clear, i'm not suggesting you should do that right now. this one seems solidly right on the border of complexity. | 20:54 |
corvus | clarkb: http://git.openstack.org/cgit/openstack-dev/devstack/tree/roles/write-devstack-local-conf/library/test.py for reference | 20:54 |
corvus | clarkb: change lgtm and works on my ubuntu machine... i assume you have some depends-on pointed at that? | 20:55 |
clarkb | ya, its a balance between data structures too complex for bash and ease of executing the code. | 20:56 |
clarkb | corvus: yes I have a devstack change I need to recheck also our ozj tests for multinode test changes to zuul-jobs | 20:56 |
clarkb | https://review.openstack.org/#/c/578157/1 is the change I have depends on for extra testing | 20:57 |
*** d0ugal has quit IRC | 20:57 | |
clarkb | if we like this approach I will update my devstack-gate change too | 20:57 |
clarkb | one thing that might make this more robust is take min(real interface mtus - 50 # the number we calculate here, other interface mtus) | 20:59 |
clarkb | I don't know that that is necessary though | 21:00 |
*** ihar has quit IRC | 21:01 | |
*** heyongli has quit IRC | 21:02 | |
*** heyongli has joined #openstack-infra | 21:03 | |
*** trown is now known as trown|outtypewww | 21:03 | |
clarkb | corvus: it has been ~2 hours since the zuul main.yaml projects list was updated but https://review.openstack.org/#/c/578563/ continues to fail. We do a sighup or similar to have zuul pick these changes up right? (its not part of dynamically reconfigured configs) | 21:05 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: Match tag items against containing branches https://review.openstack.org/578557 | 21:07 |
corvus | clarkb: yes i'll look | 21:07 |
clarkb | corvus: looks like puppet may have only just run | 21:07 |
clarkb | there was a ~2.5 hour gap in runs | 21:07 |
clarkb | I'm going to try rechecking now to see if that was the only issue | 21:07 |
corvus | this reminds me i should review the spec :) | 21:07 |
clarkb | bah bad timezone math but yes hasn't run recently | 21:09 |
clarkb | 2018-06-28 19:22:25,497 p=8359 u=root | skipping: [mirror01.cn1.linaro.openstack.org] | 21:09 |
clarkb | is the last thing from ansible running puppet | 21:09 |
corvus | there are no "Full reconfiguration" events in zuul's log, consistent with puppet not triggering it | 21:10 |
clarkb | ansible is not running, it must've just timed out | 21:10 |
clarkb | lets see if it starts again in 5 minutes (it should) | 21:11 |
*** heyongli has quit IRC | 21:13 | |
*** heyongli has joined #openstack-infra | 21:13 | |
fungi | i wonder what's caused that delay | 21:16 |
*** ldnunes has quit IRC | 21:16 | |
clarkb | it has started running again | 21:17 |
*** rosmaita has quit IRC | 21:17 | |
mordred | corvus: reading specs is lame - just blindly +2 to indicate you just agree with whatever I said | 21:17 |
clarkb | fungi: I'm guessing it is netowrk connectivity issues to some node(s) | 21:18 |
clarkb | fungi: we will probably have to catch it in the act | 21:18 |
fungi | looks like there was a similar gap from 17:06:38 to 19:00:06 | 21:20 |
clarkb | fungi: and same last line logged | 21:20 |
*** rwsu has joined #openstack-infra | 21:20 | |
corvus | mordred: oy. you used your outside voice to say what you really think again. | 21:20 |
mordred | corvus: BLAST | 21:20 |
fungi | 14:52:40 to 16:45:04 also | 21:21 |
*** heyongli has quit IRC | 21:23 | |
*** heyongli has joined #openstack-infra | 21:23 | |
clarkb | fungi: in about 20 minutes we should get to where the logging stops and we can look at what the leftover ansible processes are doing | 21:23 |
clarkb | unrelated (I thinK) zuul is really busy today. Just did 1373 jobs in the last hour as reported by graphite | 21:24 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: Report to gerrit over HTTP https://review.openstack.org/577027 | 21:25 |
corvus | that's down from our record of 2kph :( | 21:26 |
corvus | 2kjph | 21:26 |
* mordred misses the 2kjph days | 21:26 | |
clarkb | I think easy mode multinode jobs and spectre/meltdown have affected that throughput | 21:27 |
*** edmondsw has quit IRC | 21:31 | |
*** hasharAway has quit IRC | 21:32 | |
*** heyongli has quit IRC | 21:33 | |
*** heyongli has joined #openstack-infra | 21:34 | |
*** numans_ has quit IRC | 21:35 | |
*** numans has joined #openstack-infra | 21:36 | |
clarkb | fungi: ianw it looks like nb03 may be what we are hanging the puppet runs on | 21:40 |
fungi | interesting. did it die? | 21:41 |
fungi | i can ssh to it from the puppetmaster at least | 21:41 |
fungi | it last rebooted ~19.5 hours ago | 21:42 |
*** rcernin has joined #openstack-infra | 21:42 | |
clarkb | and root has an ssh connection from puppetmaster | 21:42 |
ianw | clarkb / fungi : i'm not suprised, the network is unfortunately very flaky and weird | 21:42 |
clarkb | I don't see any puppet-user logs in syslog though | 21:43 |
clarkb | Jun 28 21:23:41 nb03 ansible-setup: Invoked with filter=* gather_subset=['all'] fact_path=/etc/ansible/facts.d gather_timeout=10 is lastansible related thing in syslog | 21:43 |
*** heyongli has quit IRC | 21:43 | |
*** heyongli has joined #openstack-infra | 21:44 | |
fungi | pmtud blackhole? | 21:44 |
*** boden has quit IRC | 21:44 | |
ianw | it usually pings, isn't that icmp-ish based? | 21:45 |
clarkb | fungi: looks like they may have increased the mtu there | 21:45 |
clarkb | oh but its 8950 now :/ | 21:45 |
clarkb | so we'll have the ooposite problem | 21:45 |
fungi | yup | 21:45 |
clarkb | they need to set that neutron setting that logan- mentioned that makes it 1500 even if the host interfaces are 9000 | 21:46 |
fungi | `sudo scp nb03.openstack.org:/var/log/syslog.1 ./` hangs a quarter of the way through | 21:46 |
ianw | clarkb: yep, there is a ticket open for that | 21:46 |
fungi | so unsurprising that fact collection is similarly hanging | 21:46 |
clarkb | any objection to me manually updating that mtu on nb03 for now? | 21:46 |
fungi | wfm | 21:46 |
fungi | though that won't persist on reboot unless you also add it to the interfaces file, presumably | 21:46 |
ianw | https://servicedesk.linaro.org/servicedesk/customer/portal/11/DC-369 for reference | 21:47 |
clarkb | fungi: ya lets hope they fix ^ before next reboot. But at least this way we can confirm it is the issue | 21:47 |
fungi | agreed | 21:47 |
clarkb | I ran `sudo ip link set eth0 mtu 1500` | 21:48 |
fungi | surprised we don't experience log collection issues from job nodes there | 21:48 |
fungi | or maybe we do? | 21:48 |
clarkb | fungi: considering puppet was running yesterday iirc I'm guessing this is all very new change and we may not have noticed jobs being sad there | 21:48 |
fungi | sure, since it's only arm64 jobs presumably | 21:49 |
ianw | i'm not sure it's helped that much | 21:49 |
ianw | my test to pypi from yesterday http://paste.openstack.org/show/724499/ still works and fails about equally | 21:50 |
clarkb | ianw: it being mtu 1500? | 21:50 |
ianw | yes sorry | 21:50 |
clarkb | I only did nb03 not the mirror | 21:50 |
fungi | my syslog.1 scp now hangs at ~3/4 done | 21:50 |
ianw | clarkb: that's on nb03 ... you can see dib has just been trying over and over but can never get everything it needs for a build | 21:51 |
fungi | tried again and it hung around 1/4 done | 21:51 |
clarkb | oh I read pypi as testing our mirror, got it | 21:51 |
clarkb | I wonder if one of the links is in that cloud is not jumbo frame capable | 21:52 |
clarkb | and so even 1500 would fail | 21:52 |
ianw | fungi: yeah, and from a command line, basic interaction worked, but i noticed that "top" (which i guess sends pages of output) was a pretty good way to get it to stop | 21:52 |
clarkb | (because 1500 + overlay overhead needs jumbo frames) | 21:52 |
fungi | yup | 21:52 |
ianw | we should be able to test by increasing size of ping packets? | 21:53 |
clarkb | ianw: or I can reduce the MTU to say 1450 | 21:53 |
clarkb | (iirc that is what it was set to before) | 21:53 |
ianw | yeah, start there, can't make it worse :) | 21:54 |
clarkb | done | 21:54 |
fungi | From 103.20.251.147 icmp_seq=1 Frag needed and DF set (mtu = 1450) | 21:54 |
*** heyongli has quit IRC | 21:54 | |
*** heyongli has joined #openstack-infra | 21:54 | |
fungi | that was when trying to ping it with large packets | 21:55 |
clarkb | fungi: is your scp happier now? | 21:55 |
*** salv-orlando has joined #openstack-infra | 21:55 | |
clarkb | it should fragement properly if jumbo frames not working is the issue | 21:55 |
clarkb | *it should fragement properly now if jumbo frames not working is the issue | 21:56 |
fungi | yep, reran the same scp 5 times and worked every time | 21:56 |
ianw | ok, my "wget get-pip.py" is running reliably now i think | 21:56 |
clarkb | ya so at 1500 with overlay overhead we were still in jumbo frame territory and those must not be owrking on all links | 21:56 |
clarkb | but at 1450, we fit in at 1500 total and it fragments and is happy | 21:56 |
ianw | clarkb: do you have access to that linaro tracker? i can update their bug with this info | 21:57 |
clarkb | ianw: I don't know that I do, is it in the shared passwords file? | 21:57 |
clarkb | or do we just make our own accounts? | 21:57 |
clarkb | looks like I hvae to already hav an account, if you want to update it that would be great | 21:58 |
ianw | it's just been own accounts but not sure if you have to be added to the issue. will do, anyway gema will see this at some point i'm sure :) | 21:58 |
ianw | great find, i'm going to hope your mind just jumped to mtu because of all our recent stuff :) should of thought of that last night | 21:59 |
*** salv-orlando has quit IRC | 22:00 | |
clarkb | ya not sure how fungi immediately jumped to that | 22:01 |
clarkb | our existing ssh connection for ansible is unhappy | 22:02 |
clarkb | any reason I shouldn't just kill that process? | 22:02 |
clarkb | next pass through should be fine with the temporary 1450 value | 22:02 |
fungi | i spent years staring at various sorts of mtu-related problems | 22:03 |
fungi | i see them in my sleep | 22:03 |
ianw | so it started at 8950, 1500 didn't work and now we're at 1450 is the short story right? | 22:04 |
clarkb | ianw: yes | 22:04 |
*** heyongli has quit IRC | 22:04 | |
clarkb | I am going to kill 7039 on puppetmaster, this is the ansible process for nb03 that is out to lunch | 22:04 |
clarkb | actually before I do that I will update mtu on the mirror too as I expect it too will be unhappy | 22:04 |
*** heyongli has joined #openstack-infra | 22:04 | |
*** lxkong has joined #openstack-infra | 22:04 | |
smcginnis | Simple tagging ACL removal that has +1 from the project's PTL - https://review.openstack.org/#/c/578381/ | 22:04 |
fungi | when you give customers the ability to plug their network gear into yours, you get every crazy misconfiguration you can imagine plus a bunch more you can't | 22:05 |
openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Support skip_child_jobs via zuul_return https://review.openstack.org/578230 | 22:05 |
clarkb | mirror node is already set to 1450 | 22:06 |
clarkb | killing that process allowed ansible to finish | 22:07 |
ianw | clarkb: ahhh, because that hasn't been rebooted. | 22:08 |
ianw | https://nb03.openstack.org/ubuntu-xenial-arm64-0000007617.log is running now, and will be a good test | 22:08 |
*** rlandy has quit IRC | 22:12 | |
*** aeng has joined #openstack-infra | 22:12 | |
ianw | fungi: my network would be fine if it wasn't for all you damn clients! :) | 22:14 |
*** heyongli has quit IRC | 22:14 | |
*** nicolasbock has quit IRC | 22:14 | |
clarkb | ianw: that is what I tell my wife when her phone isn't able to upload to nextcloud | 22:15 |
*** heyongli has joined #openstack-infra | 22:15 | |
clarkb | and 9 times out of 10 it is being on the wrong wifi network | 22:15 |
*** hongbin has quit IRC | 22:16 | |
*** mpjetta has quit IRC | 22:19 | |
*** bnemec has quit IRC | 22:19 | |
*** heyongli has quit IRC | 22:24 | |
*** heyongli has joined #openstack-infra | 22:25 | |
*** kjackal has quit IRC | 22:27 | |
*** hamzy has quit IRC | 22:30 | |
clarkb | nodepool has almost caught up on the noderequest backlog | 22:30 |
*** hamzy has joined #openstack-infra | 22:30 | |
clarkb | hopefully that means zuul dashboard will update soon | 22:34 |
*** heyongli has quit IRC | 22:35 | |
*** hamzy has quit IRC | 22:35 | |
*** heyongli has joined #openstack-infra | 22:35 | |
*** jamesmcarthur has joined #openstack-infra | 22:35 | |
notmyname | is infra still using launchpad for bugs. ie if someone files a bug in LP for openstack-infra, does it get looked at? or is LP ignored and everything is storyboard now? | 22:36 |
*** hamzy has joined #openstack-infra | 22:37 | |
clarkb | notmyname: should all be storyboard now | 22:38 |
notmyname | ah | 22:38 |
*** linkmark has quit IRC | 22:38 | |
clarkb | the one exception is we hav ea launchpad bug tracker for the "gate" so that elastic-recheck can track against those bugs | 22:38 |
notmyname | https://bugs.launchpad.net/openstack-infra/+bug/1779203 | 22:38 |
openstack | Launchpad bug 1779203 in openstack-infra/networking-spp-ci "bindep does not work on Alpine linux" [Undecided,New] | 22:38 |
*** jamesmcarthur has quit IRC | 22:40 | |
fungi | yeah, infra was the first team to migrate from lp to sb, several years ago now | 22:40 |
notmyname | where should bugs be filed now? | 22:40 |
clarkb | notmyname: for bindep against the bindep project in storyboard | 22:40 |
notmyname | I mean, I could google for "openstack infra bug" but a link would be more helpful ;-) | 22:41 |
fungi | the bindep readme or contributing doc should hopefully say | 22:41 |
notmyname | fungi: unfortunately, no | 22:42 |
fungi | but basically yes, log in at storyboard.o.o, create a new story and put in openstack-infra/bindep as the project for the initial task | 22:42 |
notmyname | ah! in the contributing doc. not readme | 22:42 |
notmyname | project 811, naturally ;-) | 22:42 |
clarkb | notmyname: I'm not sure that is a bug as much as a feature request, bindep doesn't have support for alpines package manager | 22:43 |
fungi | https://git.openstack.org/cgit/openstack-infra/bindep/tree/CONTRIBUTING.rst#n18 | 22:43 |
fungi | yup | 22:43 |
clarkb | the changes I made made it possible to add that support, but you still need to teach bindep how to tlak to alpines package manager | 22:43 |
notmyname | clarkb: good news! storyboard doesn't have bugs anyway ;-) | 22:44 |
fungi | notmyname: the project names in project urls feature has actually merged, but i'm still struggling with a bug causing the webclient to do url escaping on the slashes in project names which the api doesn't know to unescape yet | 22:44 |
fungi | so close! | 22:44 |
notmyname | ok, I'll pass this on to ehud (who filed the bug in LP) | 22:44 |
clarkb | we could probably clean up the error message though (that is not really firendly way to say that the platform package manager isn't known | 22:45 |
*** heyongli has quit IRC | 22:45 | |
*** heyongli has joined #openstack-infra | 22:45 | |
fungi | thanks for bringing it to our attention, notmyname! | 22:47 |
notmyname | new to storyboard here... if I am viewing a project's stories and click the "dashboard" button on the left, then the browser back button no longer works. is this a known issue, or maybe I'm doing it wrong? | 22:52 |
*** tosky has quit IRC | 22:53 | |
*** rwsu has quit IRC | 22:54 | |
*** rwsu has joined #openstack-infra | 22:55 | |
*** heyongli has quit IRC | 22:55 | |
fungi | it's likely somewhere between a bug and a bad implementation of the dashboard link | 22:55 |
*** heyongli has joined #openstack-infra | 22:56 | |
*** salv-orlando has joined #openstack-infra | 22:56 | |
notmyname | is the "jump to..." box shortcuts for commands or a search box for projects? | 22:57 |
*** agopi has quit IRC | 22:58 | |
notmyname | (answer: search, mostly) | 23:00 |
*** salv-orlando has quit IRC | 23:01 | |
*** tonytan4ever_brb has quit IRC | 23:03 | |
*** tonytan4ever has joined #openstack-infra | 23:03 | |
*** caphrim007_ has joined #openstack-infra | 23:04 | |
*** caphrim007_ has quit IRC | 23:05 | |
*** threestrands has joined #openstack-infra | 23:05 | |
*** threestrands has quit IRC | 23:05 | |
*** threestrands has joined #openstack-infra | 23:05 | |
*** caphrim007_ has joined #openstack-infra | 23:05 | |
*** heyongli has quit IRC | 23:05 | |
*** heyongli has joined #openstack-infra | 23:06 | |
*** yolanda__ has joined #openstack-infra | 23:07 | |
*** caphrim007 has quit IRC | 23:07 | |
*** caphrim007_ has quit IRC | 23:09 | |
openstackgerrit | Clark Boylan proposed openstack-infra/bindep master: Clearer unsupported platform error message https://review.openstack.org/579029 | 23:09 |
clarkb | notmyname: fungi ^ that should make these errors more clear | 23:10 |
notmyname | cool | 23:10 |
*** yolanda_ has quit IRC | 23:11 | |
clarkb | oh forgot to git add the os-release file | 23:11 |
*** caphrim007 has joined #openstack-infra | 23:11 | |
openstackgerrit | Clark Boylan proposed openstack-infra/bindep master: Clearer unsupported platform error message https://review.openstack.org/579029 | 23:11 |
*** heyongli has quit IRC | 23:16 | |
*** caphrim007 has quit IRC | 23:16 | |
*** heyongli has joined #openstack-infra | 23:16 | |
*** caphrim007 has joined #openstack-infra | 23:17 | |
*** yolanda__ has quit IRC | 23:18 | |
*** caphrim007 has quit IRC | 23:21 | |
openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Support skip_child_jobs via zuul_return https://review.openstack.org/578230 | 23:26 |
*** heyongli has quit IRC | 23:26 | |
*** heyongli has joined #openstack-infra | 23:26 | |
*** dhill_ has quit IRC | 23:27 | |
openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: WIP: Add a role to return file comments https://review.openstack.org/579033 | 23:28 |
*** dpawlik has joined #openstack-infra | 23:29 | |
clarkb | corvus: as a heads up the zuul post job ran which may have incomplete dashboard | 23:29 |
clarkb | the followup is queued | 23:30 |
openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: Support line comments in Gerrit https://review.openstack.org/577035 | 23:31 |
ianw | can i stop nodepool-builder picking up a new build, but let it finish the current one? | 23:33 |
*** dpawlik has quit IRC | 23:33 | |
corvus | clarkb: post job have dashboard? i'm confused | 23:34 |
ianw | i want to attach a new volume to nb03.o.o as space is getting tight now we build bionic too | 23:34 |
corvus | clarkb: our post jobs don't deploy the dashboard | 23:34 |
clarkb | corvus: they create the tarball that puppet deploys | 23:34 |
corvus | ooh | 23:34 |
pabelanger | ianw: no, i believe DIB will stop or try too | 23:35 |
corvus | i should resume work on the promote pipeline | 23:36 |
*** heyongli has quit IRC | 23:36 | |
*** heyongli has joined #openstack-infra | 23:37 | |
clarkb | corvus: that would help here ++ | 23:37 |
clarkb | looks like part of our backlog is the integrated gate has been flapping, nova just had a lower constraints failure | 23:37 |
*** threestrands has quit IRC | 23:38 | |
openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: WIP: Add a role to return file comments https://review.openstack.org/579033 | 23:38 |
*** tpsilva has quit IRC | 23:40 | |
*** agopi has joined #openstack-infra | 23:46 | |
*** heyongli has quit IRC | 23:46 | |
*** heyongli has joined #openstack-infra | 23:47 | |
*** masber has quit IRC | 23:48 | |
*** sthussey has quit IRC | 23:48 | |
*** stakeda has joined #openstack-infra | 23:54 | |
*** mpjetta has joined #openstack-infra | 23:55 | |
*** gagehugo has quit IRC | 23:56 | |
*** heyongli has quit IRC | 23:57 | |
*** heyongli has joined #openstack-infra | 23:57 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!