ianw | mtreinish: ahh, there is a file there | 00:00 |
---|---|---|
fungi | ianw: could it have already been dead at that point? | 00:00 |
fungi | timestamp on the crash file? | 00:00 |
ianw | with a base64 encoded coredump | 00:00 |
fungi | yeah, there's a tool to extract the coredump from that | 00:00 |
ianw | -rw-r----- 1 root root 5.0M May 27 06:25 _usr_sbin_mosquitto.0.crash | 00:00 |
*** VW has quit IRC | 00:00 | |
fungi | right, so you were looking at too recent of a log | 00:01 |
fungi | but the timing there does look like log rotation again | 00:01 |
fungi | just a couple days prior | 00:01 |
ianw | oh, it lines up. the restart thought it was pid 8816 (from old pid file i guess) and the coredump thingy is saying it was for pid 8816 | 00:02 |
fungi | oh, sorry, i was looking at your start-stop-daemon output and mistaking it for something from syslog during log rotation | 00:03 |
fungi | yeah, seems pretty likely to be the same problem | 00:03 |
fungi | and i agree with clarkb, we've actually previously discussed changing the logrotate config to use a different rotation mechanism | 00:03 |
fungi | as a workaround | 00:03 |
ianw | # apport-retrace -g ./_usr_sbin_mosquitto.0.crash | 00:04 |
ianw | ERROR: report file does not contain one of the required fields: CoreDump DistroRelease Package ExecutablePath | 00:04 |
ianw | boo | 00:04 |
openstackgerrit | Michael Johnson proposed openstack-infra/zuul-jobs master: Collect the coverage report for npm test jobs https://review.openstack.org/570260 | 00:04 |
fungi | but wanted to collect some good debugging info first, since obviously having a daemon which crashes on sighup could be similarly problematic for, e.g., configuration reloads | 00:04 |
*** geekinutah has quit IRC | 00:04 | |
*** edwarnicke has quit IRC | 00:04 | |
*** jamespage has quit IRC | 00:04 | |
*** zul has quit IRC | 00:04 | |
fungi | ianw: you want apport-unpack | 00:04 |
*** auggy_ has joined #openstack-infra | 00:05 | |
*** electrical__ has joined #openstack-infra | 00:05 | |
*** mgkwill_ has joined #openstack-infra | 00:05 | |
*** ameade_ has joined #openstack-infra | 00:05 | |
*** lamt__ has joined #openstack-infra | 00:05 | |
fungi | e.g., `apport-unpack _usr_sbin_mosquitto.0.crash foo` | 00:05 |
*** patrickeast has quit IRC | 00:05 | |
*** auggy has quit IRC | 00:05 | |
*** mgkwill has quit IRC | 00:05 | |
*** auggy_ is now known as auggy | 00:05 | |
*** mgkwill_ is now known as mgkwill | 00:05 | |
mtreinish | fungi: to be fair a full restart is probably safe enough for mosquitto because mqtt handles the blips pretty well | 00:05 |
*** ildikov_ has joined #openstack-infra | 00:05 | |
*** patrickeast has joined #openstack-infra | 00:05 | |
fungi | ianw: followed by `gdb /usr/sbin/mosquitto foo/CoreDump` | 00:05 |
*** csatari_ has joined #openstack-infra | 00:05 | |
mtreinish | well with the exception of the code I wrote which doesn't have any real error ahndling :p | 00:05 |
*** csatari has quit IRC | 00:05 | |
*** ildikov has quit IRC | 00:05 | |
*** ameade has quit IRC | 00:05 | |
*** csatari_ is now known as csatari | 00:05 | |
*** electrical_ has quit IRC | 00:05 | |
*** odyssey4me has quit IRC | 00:05 | |
*** ameade_ is now known as ameade | 00:05 | |
*** ildikov_ is now known as ildikov | 00:05 | |
*** electrical__ is now known as electrical_ | 00:05 | |
*** nicolasbock has quit IRC | 00:06 | |
*** MasterofJOKers has quit IRC | 00:06 | |
*** mordred has quit IRC | 00:06 | |
ianw | i hacked in a Package: field anyway | 00:06 |
ianw | http://paste.openstack.org/show/722314/ | 00:06 |
*** pahuang has quit IRC | 00:06 | |
fungi | ianw: and before you ask, i'm just pulling that from my command history of what i managed to figure out last time after lots of reading | 00:06 |
*** HenryG has quit IRC | 00:06 | |
*** zigo has quit IRC | 00:06 | |
*** dhellmann has quit IRC | 00:06 | |
*** Roamer` has quit IRC | 00:06 | |
*** afazekas has quit IRC | 00:06 | |
*** lamt has quit IRC | 00:06 | |
*** melwitt has quit IRC | 00:06 | |
*** lamt__ is now known as lamt | 00:06 | |
*** zigo_ has joined #openstack-infra | 00:06 | |
mtreinish | ianw: yeah that looks the same as what we hit last time | 00:07 |
*** HenryG has joined #openstack-infra | 00:07 | |
*** r-daneel has quit IRC | 00:07 | |
fungi | agreed | 00:07 |
ianw | fungi: so what do you do when the tool to extract the crash dump crashes? http://paste.openstack.org/show/722315/ | 00:07 |
*** Roamer` has joined #openstack-infra | 00:08 | |
fungi | hah! | 00:08 |
mtreinish | lol | 00:08 |
fungi | yeah, the old dump is at /home/fungi/_usr_sbin_mosquitto.0.crash if you want to compare metadata | 00:09 |
ianw | but ok, if it's similar bt i won't keep digging | 00:09 |
*** MasterofJOKers has joined #openstack-infra | 00:10 | |
*** odyssey4me has joined #openstack-infra | 00:11 | |
*** melwitt has joined #openstack-infra | 00:11 | |
fungi | and apport-unpack still seems to create a CoreDump file from the new crash file successfully even if it raises an exception | 00:11 |
*** dhellmann has joined #openstack-infra | 00:11 | |
ianw | ok, yeah, bt looks the same in yours too | 00:11 |
*** afazekas has joined #openstack-infra | 00:11 | |
*** ChanServ has quit IRC | 00:12 | |
*** camunoz has quit IRC | 00:12 | |
clarkb | reading openstack_inventory.py I can probably just test it out of band of ansible | 00:13 |
*** sfeole` is now known as sfeole | 00:13 | |
clarkb | openstacksdk is installed on puppet master | 00:13 |
clarkb | heh ansibles dynamic inventory docs are all wrong becuse of this change too | 00:14 |
fungi | documentation for a free software project out of sync with reality? hey, at least it _has_ documentation! ;) | 00:15 |
*** mordred has joined #openstack-infra | 00:15 | |
clarkb | oh sure I'm not complaining as much as revelling in how far this thread goes as I pull it :) | 00:15 |
*** geekinutah has joined #openstack-infra | 00:15 | |
fungi | you can just blame mordred, he won't mind | 00:16 |
*** zul has joined #openstack-infra | 00:16 | |
*** jamespage has joined #openstack-infra | 00:16 | |
*** edwarnicke has joined #openstack-infra | 00:16 | |
clarkb | looks like I can run openstack_inventory --list --cloud somecloudname as a regular user and it won't try to update global caches | 00:18 |
clarkb | however that potential global state is my biggest worry right now | 00:18 |
*** yolanda_ has quit IRC | 00:19 | |
*** yolanda_ has joined #openstack-infra | 00:20 | |
clarkb | probably best to test that in the morning so I don't make ansible puppet less happy | 00:20 |
*** rwsu has quit IRC | 00:20 | |
clarkb | I'd test it locally except I want ot make sure that puppetmaster will work with it :/ | 00:21 |
*** vabada has quit IRC | 00:21 | |
fungi | completely understandable | 00:21 |
*** jamesmcarthur has joined #openstack-infra | 00:22 | |
*** pahuang has joined #openstack-infra | 00:23 | |
*** jamesmcarthur has quit IRC | 00:27 | |
*** caphrim007_ has quit IRC | 00:30 | |
*** rlandy|rover has quit IRC | 00:33 | |
*** ChanServ has joined #openstack-infra | 00:33 | |
*** barjavel.freenode.net sets mode: +o ChanServ | 00:33 | |
*** smcginnis[m] has quit IRC | 00:37 | |
clarkb | I'll pick it up in the morning I think the safe short term option is the pin on the repo, then we can test the newer version of the script and update puppetry to use that if/when it works | 00:37 |
*** lbragstad[m] has quit IRC | 00:37 | |
*** kbaikov[m] has quit IRC | 00:37 | |
*** masayukig1 has quit IRC | 00:37 | |
*** aspiers[m] has quit IRC | 00:37 | |
*** warlord77[m] has quit IRC | 00:38 | |
*** lamt is now known as Guest41358 | 00:39 | |
*** melwitt is now known as Guest48153 | 00:39 | |
*** vabada has joined #openstack-infra | 00:39 | |
*** nicolasbock has joined #openstack-infra | 00:40 | |
*** dpawlik has joined #openstack-infra | 00:40 | |
fungi | wfm | 00:41 |
*** VW has joined #openstack-infra | 00:41 | |
*** VW has quit IRC | 00:44 | |
*** dhinesh_ has joined #openstack-infra | 00:45 | |
*** dpawlik has quit IRC | 00:45 | |
*** rwsu has joined #openstack-infra | 00:45 | |
*** jesslampe has joined #openstack-infra | 00:50 | |
*** annp has quit IRC | 00:53 | |
*** annp has joined #openstack-infra | 00:54 | |
*** jesslampe has quit IRC | 00:54 | |
*** masayukig1 has joined #openstack-infra | 00:59 | |
*** harlowja has quit IRC | 01:00 | |
*** iyamahat has quit IRC | 01:06 | |
*** dingyichen has joined #openstack-infra | 01:12 | |
*** gongysh has joined #openstack-infra | 01:17 | |
*** gyankum has joined #openstack-infra | 01:29 | |
*** jamespage has quit IRC | 01:30 | |
*** mordred has quit IRC | 01:31 | |
*** gyee has quit IRC | 01:31 | |
*** sree has joined #openstack-infra | 01:34 | |
*** kien-ha has joined #openstack-infra | 01:37 | |
radez | ianw: ping, hey Ian, I saw there was a review merged to fix https://review.openstack.org/#/c/569121/ | 01:38 |
*** hongbin has joined #openstack-infra | 01:38 | |
*** sree has quit IRC | 01:38 | |
radez | but I appear to still be hitting it, is there a chance it hasn't propegated in some way or should I not be hitting it? | 01:38 |
*** warlord77[m] has joined #openstack-infra | 01:39 | |
*** kbaikov[m] has joined #openstack-infra | 01:39 | |
*** aspiers[m] has joined #openstack-infra | 01:39 | |
*** lbragstad[m] has joined #openstack-infra | 01:39 | |
*** smcginnis[m] has joined #openstack-infra | 01:39 | |
ianw | radez: was it rebased? | 01:39 |
radez | different project, I'm in openstack-requirements | 01:40 |
radez | should I have to rebase it there too> | 01:40 |
ianw | oh, hang on, it's hitting hte cmd2 stuff right? | 01:40 |
radez | ianw: yup | 01:41 |
*** gongysh has quit IRC | 01:41 | |
fungi | the hopeful initial fix https://review.openstack.org/570811 merged at 23:39 utc | 01:41 |
fungi | probably better to discuss in #openstack-requirements | 01:41 |
ianw | yeah, i think the earlier recheck probably just missed it, the last one done just now hopefully passes | 01:42 |
fungi | still need the equivalent fix merged to cliff and then a new cliff release made before it'll be fixed in unconstrained jobs | 01:42 |
fungi | the failing jobs in the reqs repo don't run with constraints applied (intentionally) | 01:42 |
fungi | though it'll just be the py27 unit tests and tox-validate failing on the recheck, presumably | 01:43 |
*** jogo has quit IRC | 01:43 | |
*** roman_g has quit IRC | 01:44 | |
*** mordred has joined #openstack-infra | 01:44 | |
*** jamespage has joined #openstack-infra | 01:44 | |
*** felipemonteiro has joined #openstack-infra | 01:44 | |
radez | oh so I should have to rebase in the reqs repo? | 01:45 |
fungi | no, but the couple of jobs i mentioned will continue to fail for changes to reqs until a new cliff release happens | 01:45 |
prometheanfire | well, cliff passed | 01:46 |
fungi | might want to drum up support in #openstack-oslo if nobody has already | 01:47 |
*** armaan has quit IRC | 01:48 | |
*** armaan has joined #openstack-infra | 01:49 | |
*** roman_g has joined #openstack-infra | 01:58 | |
*** aeng has quit IRC | 02:06 | |
*** dhinesh_ has quit IRC | 02:06 | |
*** Guest41358 is now known as lamt | 02:06 | |
*** aeng has joined #openstack-infra | 02:07 | |
*** pahuang has quit IRC | 02:11 | |
*** rajinir has joined #openstack-infra | 02:17 | |
*** yamahata has quit IRC | 02:20 | |
*** caphrim007 has joined #openstack-infra | 02:21 | |
*** iyamahat has joined #openstack-infra | 02:21 | |
*** iyamahat has quit IRC | 02:22 | |
*** pahuang has joined #openstack-infra | 02:24 | |
*** caphrim007 has quit IRC | 02:26 | |
*** iyamahat has joined #openstack-infra | 02:26 | |
*** iyamahat has quit IRC | 02:27 | |
*** iyamahat has joined #openstack-infra | 02:28 | |
*** iyamahat has quit IRC | 02:35 | |
*** caphrim007 has joined #openstack-infra | 02:36 | |
*** dklyle has joined #openstack-infra | 02:40 | |
*** jogo has joined #openstack-infra | 02:41 | |
*** psachin has joined #openstack-infra | 02:41 | |
*** zhangfei has joined #openstack-infra | 02:43 | |
*** caphrim007 has quit IRC | 02:44 | |
*** caphrim007 has joined #openstack-infra | 02:45 | |
*** markvoelker_ has quit IRC | 02:45 | |
*** VW has joined #openstack-infra | 02:46 | |
*** ykarel|away has joined #openstack-infra | 02:46 | |
*** dbecker has quit IRC | 02:47 | |
*** namnh has joined #openstack-infra | 02:48 | |
*** dchen has joined #openstack-infra | 02:49 | |
*** rlandy has joined #openstack-infra | 02:51 | |
*** xinliang has quit IRC | 02:52 | |
*** dklyle has quit IRC | 02:54 | |
*** rosmaita has quit IRC | 02:57 | |
*** xinliang has joined #openstack-infra | 02:58 | |
*** dbecker has joined #openstack-infra | 03:01 | |
*** masayuki- is now known as masayukig_ | 03:08 | |
*** masayukig is now known as masayukig__ | 03:11 | |
*** masayukig_ is now known as masayukig | 03:12 | |
*** janki has joined #openstack-infra | 03:12 | |
*** masayukig__ is now known as masayukig_ | 03:13 | |
*** auristor has quit IRC | 03:21 | |
*** auristor has joined #openstack-infra | 03:27 | |
*** harlowja has joined #openstack-infra | 03:29 | |
*** felipemonteiro has quit IRC | 03:32 | |
*** pahuang has quit IRC | 03:34 | |
*** markvoelker has joined #openstack-infra | 03:35 | |
*** jesslampe has joined #openstack-infra | 03:35 | |
*** markvoelker has quit IRC | 03:39 | |
*** lpetrut has joined #openstack-infra | 03:46 | |
*** pahuang has joined #openstack-infra | 03:47 | |
*** felipemonteiro has joined #openstack-infra | 03:49 | |
*** hongbin has quit IRC | 03:53 | |
*** nicolasbock has quit IRC | 03:55 | |
*** udesale has joined #openstack-infra | 03:58 | |
*** ykarel|away is now known as ykarel | 03:59 | |
*** calebb has quit IRC | 04:05 | |
*** rlandy has quit IRC | 04:05 | |
*** calebb has joined #openstack-infra | 04:05 | |
*** ykarel is now known as ykarel|afk | 04:07 | |
*** links has joined #openstack-infra | 04:11 | |
*** annp has quit IRC | 04:13 | |
*** harlowja has quit IRC | 04:14 | |
*** lpetrut has quit IRC | 04:19 | |
*** markvoelker has joined #openstack-infra | 04:29 | |
*** sree has joined #openstack-infra | 04:31 | |
*** dpawlik has joined #openstack-infra | 04:42 | |
*** pgadiya has joined #openstack-infra | 04:42 | |
*** pgadiya has quit IRC | 04:44 | |
*** psachin has quit IRC | 04:46 | |
*** dpawlik has quit IRC | 04:47 | |
*** katyafervent_awa has quit IRC | 04:51 | |
*** rakhmerov has quit IRC | 04:51 | |
*** igormarnat has quit IRC | 04:51 | |
*** dteselkin has quit IRC | 04:51 | |
*** asilenkov has quit IRC | 04:51 | |
*** VW has quit IRC | 04:52 | |
*** xarses has joined #openstack-infra | 04:55 | |
*** xarses has quit IRC | 04:56 | |
*** xarses has joined #openstack-infra | 04:56 | |
*** xarses_ has quit IRC | 04:58 | |
*** e0ne has joined #openstack-infra | 05:00 | |
*** pahuang has quit IRC | 05:00 | |
*** markvoelker has quit IRC | 05:03 | |
*** pahuang has joined #openstack-infra | 05:04 | |
*** xarses has quit IRC | 05:04 | |
*** felipemonteiro has quit IRC | 05:05 | |
*** dingyichen has quit IRC | 05:05 | |
*** xarses has joined #openstack-infra | 05:05 | |
*** xarses has quit IRC | 05:05 | |
*** dingyichen has joined #openstack-infra | 05:05 | |
*** xarses has joined #openstack-infra | 05:05 | |
openstackgerrit | sebastian marcet proposed openstack-infra/openstackid master: Fixed initial DB migrations/seeds https://review.openstack.org/571092 | 05:08 |
*** bhujay has joined #openstack-infra | 05:10 | |
*** eandersson has joined #openstack-infra | 05:10 | |
*** snapiri- has quit IRC | 05:11 | |
*** snapiri has joined #openstack-infra | 05:11 | |
*** dhajare_ has joined #openstack-infra | 05:11 | |
*** rakhmerov has joined #openstack-infra | 05:16 | |
openstackgerrit | Merged openstack-infra/openstackid master: Fixed initial DB migrations/seeds https://review.openstack.org/571092 | 05:18 |
*** ykarel|afk is now known as ykarel | 05:18 | |
*** lifeless_ has quit IRC | 05:19 | |
*** lifeless has joined #openstack-infra | 05:19 | |
*** katyafervent_awa has joined #openstack-infra | 05:20 | |
*** dteselkin has joined #openstack-infra | 05:21 | |
*** asilenkov has joined #openstack-infra | 05:24 | |
*** igormarnat has joined #openstack-infra | 05:25 | |
*** kien-ha has quit IRC | 05:27 | |
*** felipemonteiro has joined #openstack-infra | 05:27 | |
*** iranzo has joined #openstack-infra | 05:41 | |
*** bhujay has quit IRC | 05:41 | |
*** jesslampe has quit IRC | 05:42 | |
*** dhajare_ has quit IRC | 05:42 | |
*** eandersson has quit IRC | 05:48 | |
*** eandersson has joined #openstack-infra | 05:49 | |
*** felipemonteiro has quit IRC | 05:50 | |
*** felipemonteiro has joined #openstack-infra | 05:53 | |
*** bhujay has joined #openstack-infra | 05:54 | |
*** dhajare_ has joined #openstack-infra | 05:55 | |
*** igormarnat has quit IRC | 05:59 | |
*** asilenkov has quit IRC | 05:59 | |
*** katyafervent_awa has quit IRC | 05:59 | |
*** hashar has joined #openstack-infra | 05:59 | |
*** rakhmerov has quit IRC | 05:59 | |
*** dteselkin has quit IRC | 06:00 | |
*** cshastri has joined #openstack-infra | 06:02 | |
openstackgerrit | Merged openstack-infra/yaml2ical master: Fix output used by http://eavesdrop.openstack.org https://review.openstack.org/568596 | 06:03 |
*** dhajare_ has quit IRC | 06:06 | |
*** e0ne has quit IRC | 06:07 | |
*** armaan has quit IRC | 06:09 | |
*** armaan has joined #openstack-infra | 06:09 | |
*** jesslampe has joined #openstack-infra | 06:13 | |
*** jesslampe has quit IRC | 06:15 | |
*** katyafervent_awa has joined #openstack-infra | 06:18 | |
*** rakhmerov has joined #openstack-infra | 06:19 | |
*** dteselkin has joined #openstack-infra | 06:20 | |
*** markvoelker has joined #openstack-infra | 06:20 | |
*** alexchadin has joined #openstack-infra | 06:20 | |
*** asilenkov has joined #openstack-infra | 06:21 | |
*** e0ne has joined #openstack-infra | 06:22 | |
*** igormarnat has joined #openstack-infra | 06:22 | |
*** lpetrut has joined #openstack-infra | 06:22 | |
*** lifeless has quit IRC | 06:23 | |
*** e0ne has quit IRC | 06:25 | |
*** lifeless has joined #openstack-infra | 06:25 | |
*** jtomasek has joined #openstack-infra | 06:25 | |
*** rajinir has quit IRC | 06:27 | |
*** xinliang3 has quit IRC | 06:27 | |
*** xinliang3 has joined #openstack-infra | 06:28 | |
*** jtomasek has quit IRC | 06:28 | |
*** pcaruana has joined #openstack-infra | 06:33 | |
*** dpawlik has joined #openstack-infra | 06:41 | |
*** sree has quit IRC | 06:44 | |
*** dpawlik has quit IRC | 06:45 | |
*** annp has joined #openstack-infra | 06:52 | |
*** sree has joined #openstack-infra | 06:52 | |
*** markvoelker has quit IRC | 06:54 | |
*** apple01 has joined #openstack-infra | 06:54 | |
*** cshastri has quit IRC | 06:57 | |
*** ssbarnea_ has joined #openstack-infra | 06:57 | |
*** tosky has joined #openstack-infra | 07:06 | |
*** gongysh has joined #openstack-infra | 07:06 | |
*** ccamacho has joined #openstack-infra | 07:11 | |
*** jaosorior has joined #openstack-infra | 07:15 | |
*** apple01 has quit IRC | 07:17 | |
*** sshnaidm is now known as sshnaidm_pto | 07:20 | |
*** adriant has quit IRC | 07:20 | |
*** tesseract has joined #openstack-infra | 07:20 | |
*** adriant has joined #openstack-infra | 07:20 | |
*** apetrich has quit IRC | 07:21 | |
*** cshastri has joined #openstack-infra | 07:22 | |
*** apetrich has joined #openstack-infra | 07:23 | |
*** florianf has joined #openstack-infra | 07:23 | |
*** jtomasek has joined #openstack-infra | 07:25 | |
*** rcernin has quit IRC | 07:27 | |
*** amoralej|off is now known as amoralej | 07:27 | |
*** alexchadin has quit IRC | 07:28 | |
*** ociuhandu has joined #openstack-infra | 07:29 | |
*** jtomasek has quit IRC | 07:31 | |
*** alexchadin has joined #openstack-infra | 07:34 | |
*** dpawlik has joined #openstack-infra | 07:34 | |
*** ykarel is now known as ykarel|lunch | 07:36 | |
*** aojea has joined #openstack-infra | 07:37 | |
*** bhujay has quit IRC | 07:37 | |
*** dpawlik has quit IRC | 07:39 | |
*** gongysh has quit IRC | 07:41 | |
*** dpawlik has joined #openstack-infra | 07:41 | |
*** markvoelker has joined #openstack-infra | 07:45 | |
*** dpawlik has quit IRC | 07:45 | |
*** dpawlik_ has joined #openstack-infra | 07:45 | |
*** jpena|off is now known as jpena | 07:45 | |
*** ociuhandu has quit IRC | 07:45 | |
*** apple01 has joined #openstack-infra | 07:46 | |
*** ykarel|lunch has quit IRC | 07:47 | |
*** armaan has quit IRC | 07:48 | |
*** armaan has joined #openstack-infra | 07:49 | |
*** markvoelker has quit IRC | 07:49 | |
*** apple01 has quit IRC | 07:51 | |
*** zhangfei has quit IRC | 07:53 | |
*** jpich has joined #openstack-infra | 07:53 | |
*** gongysh has joined #openstack-infra | 07:53 | |
*** bhujay has joined #openstack-infra | 07:59 | |
*** armaan has quit IRC | 08:04 | |
*** armaan has joined #openstack-infra | 08:04 | |
*** zhangfei has joined #openstack-infra | 08:05 | |
*** hrubi has joined #openstack-infra | 08:07 | |
*** hrubi_ has quit IRC | 08:07 | |
*** bhujay has quit IRC | 08:12 | |
*** felipemonteiro has quit IRC | 08:15 | |
*** rossella_s has quit IRC | 08:15 | |
*** rossella_s has joined #openstack-infra | 08:16 | |
*** rossella_s has quit IRC | 08:22 | |
*** xarses_ has joined #openstack-infra | 08:22 | |
*** vivsoni has quit IRC | 08:25 | |
*** xarses has quit IRC | 08:25 | |
*** rossella_s has joined #openstack-infra | 08:25 | |
hwoarang | hello infra. Anyone willing to look at https://review.openstack.org/#/c/570569/ which hopefully enables opensuse leap 15 images on nodepool? thanks! | 08:28 |
*** zhangfei has quit IRC | 08:29 | |
*** markvoelker has joined #openstack-infra | 08:38 | |
*** rossella_s has quit IRC | 08:39 | |
zigo_ | ianw: The fix of yesterday really fixed the repo, that worked. However, I still have an issue with puppet-openstack that believes it can't enable redis-server service, when it works on my local laptop. Could you get a VM stopped again, so I can investigate? | 08:40 |
*** gfidente has joined #openstack-infra | 08:41 | |
*** gfidente has joined #openstack-infra | 08:41 | |
*** zhangfei has joined #openstack-infra | 08:41 | |
*** rossella_s has joined #openstack-infra | 08:42 | |
*** lifeless_ has joined #openstack-infra | 08:44 | |
*** lifeless has quit IRC | 08:44 | |
*** xarses has joined #openstack-infra | 08:45 | |
*** rossella_s has quit IRC | 08:47 | |
*** xarses_ has quit IRC | 08:48 | |
*** ykarel|lunch has joined #openstack-infra | 08:48 | |
*** vivsoni has joined #openstack-infra | 08:50 | |
*** alexchadin has quit IRC | 08:51 | |
*** rossella_s has joined #openstack-infra | 08:52 | |
*** dingyichen has quit IRC | 08:54 | |
*** dchen has quit IRC | 08:55 | |
*** apple01 has joined #openstack-infra | 08:56 | |
*** bhujay has joined #openstack-infra | 08:56 | |
*** ykarel|lunch is now known as ykarel | 08:57 | |
*** alexchadin has joined #openstack-infra | 08:58 | |
*** apple01 has quit IRC | 09:01 | |
*** salv-orlando has joined #openstack-infra | 09:02 | |
*** gongysh has quit IRC | 09:04 | |
*** armaan has quit IRC | 09:05 | |
*** armaan has joined #openstack-infra | 09:05 | |
*** ykarel is now known as ykarel|away | 09:06 | |
*** dpawlik_ has quit IRC | 09:09 | |
*** dpawlik has joined #openstack-infra | 09:09 | |
*** armaan has quit IRC | 09:09 | |
*** alexchadin has quit IRC | 09:11 | |
*** ykarel|away has quit IRC | 09:12 | |
*** markvoelker has quit IRC | 09:12 | |
*** armaan has joined #openstack-infra | 09:13 | |
*** pcichy has joined #openstack-infra | 09:13 | |
*** dpawlik has quit IRC | 09:13 | |
*** dpawlik has joined #openstack-infra | 09:18 | |
*** gcb has joined #openstack-infra | 09:19 | |
*** yamamoto has quit IRC | 09:21 | |
ianw | zigo_: ahh, let me see | 09:22 |
*** hashar has quit IRC | 09:23 | |
*** hashar has joined #openstack-infra | 09:23 | |
zigo_ | ianw: I probably have found the issue. | 09:23 |
zigo_ | ianw: What I'm experiencing is maybe due to https://bugs.debian.org/850534 | 09:24 |
openstack | Debian bug 850534 in redis-server "redis-server.service warns Unknown lvalue 'RunTimeDirectory' in section 'Service'" [Normal,Fixed] | 09:24 |
*** jtomasek has joined #openstack-infra | 09:24 | |
zigo_ | ie: RunTimeDirectory -> RuntimeDirectory | 09:24 |
ianw | zigo_: do you need arm64 or just a debian node? | 09:25 |
*** jtomasek has quit IRC | 09:25 | |
zigo_ | ianw: Hang on until I do one last test with a modified redis-server package first. | 09:25 |
zigo_ | No need to bother just yet, I'll ping you (or someone else) if I can't fix it with that modified package. | 09:25 |
ianw | alright, lmn | 09:26 |
zigo_ | Thanks. :) | 09:26 |
*** dhajare_ has joined #openstack-infra | 09:26 | |
*** zoli is now known as zoli|lunch | 09:31 | |
*** udesale_ has joined #openstack-infra | 09:31 | |
*** udesale_ has quit IRC | 09:32 | |
*** udesale_ has joined #openstack-infra | 09:32 | |
*** udesale_ has quit IRC | 09:32 | |
*** udesale_ has joined #openstack-infra | 09:33 | |
*** udesale has quit IRC | 09:33 | |
*** dtantsur|afk is now known as dtantsur | 09:38 | |
*** hashar is now known as hasharAway | 09:50 | |
*** yamamoto has joined #openstack-infra | 09:50 | |
stephenfin | corvus: I think this is ready to go now, whenever you're about https://review.openstack.org/#/c/559348/6 | 09:50 |
zigo_ | ianw: My patch to the redis-server.service didn't help, so I do need to debug on the VM. | 09:55 |
zigo_ | ianw: The patch is this one: https://review.openstack.org/#/c/564328 and the job is puppet-openstack-integration-4-scenario001-tempest-debian-stable (for the puppet-nova repo) | 09:56 |
*** zigo_ is now known as zigo | 09:56 | |
*** sree has quit IRC | 09:57 | |
*** rossella_s has quit IRC | 10:00 | |
*** rossella_s has joined #openstack-infra | 10:00 | |
*** annp has quit IRC | 10:01 | |
*** udesale__ has joined #openstack-infra | 10:02 | |
*** udesale__ has quit IRC | 10:02 | |
ianw | | openstack | git.openstack.org/openstack/puppet-nova | puppet-openstack-integration-4-scenario001-tempest-debian-stable | .* | 1 | ianw: 564328 | | 10:03 |
ianw | ok, we should hold one | 10:03 |
*** udesale has joined #openstack-infra | 10:03 | |
zigo | ianw: Did you do a recheck? :) | 10:05 |
*** udesale_ has quit IRC | 10:05 | |
zigo | Ah yeah, thanks. | 10:05 |
zigo | rebased ... | 10:05 |
ianw | to get it to cancel inflight | 10:05 |
*** rossella_s has quit IRC | 10:06 | |
*** apple01 has joined #openstack-infra | 10:06 | |
*** namnh has quit IRC | 10:08 | |
*** rossella_s has joined #openstack-infra | 10:09 | |
*** jaosorior has quit IRC | 10:14 | |
*** apple01 has quit IRC | 10:14 | |
*** psachin has joined #openstack-infra | 10:20 | |
*** sambetts|afk is now known as sambetts | 10:23 | |
*** salv-orlando has quit IRC | 10:26 | |
dtantsur | hi folks! is it possible to make https://docs.openstack.org/project-install-guide/baremetal/draft/ a redirect to https://docs.openstack.org/ironic/latest/install/ please? | 10:30 |
dtantsur | the former is no longer updated, but people still find it somehow.. | 10:30 |
*** jaosorior has joined #openstack-infra | 10:31 | |
*** lifeless_ has quit IRC | 10:32 | |
*** markvoelker has joined #openstack-infra | 10:35 | |
*** rossella_s has quit IRC | 10:36 | |
*** rossella_s has joined #openstack-infra | 10:38 | |
*** gyankum has quit IRC | 10:41 | |
*** ssbarnea_ has quit IRC | 10:42 | |
zigo | ianw: The job should be stopped and VM ready. My ssh key: http://thomas.goirand.fr/ssh-pubkey.txt | 10:48 |
*** nicolasbock has joined #openstack-infra | 10:50 | |
ianw | zigo: root@104.239.240.37 | 10:52 |
zigo | ianw: Cheers. I'll do my investigations after lunch. | 10:53 |
*** zoli|lunch is now known as zoli | 11:00 | |
*** adreznec has quit IRC | 11:00 | |
*** lifeless has joined #openstack-infra | 11:01 | |
*** adreznec has joined #openstack-infra | 11:01 | |
*** apple01 has joined #openstack-infra | 11:04 | |
*** markvoelker has quit IRC | 11:06 | |
*** apple01 has quit IRC | 11:08 | |
*** armaan has quit IRC | 11:16 | |
*** armaan has joined #openstack-infra | 11:17 | |
*** ociuhandu has joined #openstack-infra | 11:19 | |
*** ssbarnea_ has joined #openstack-infra | 11:19 | |
*** apple01 has joined #openstack-infra | 11:20 | |
*** dave-mccowan has joined #openstack-infra | 11:21 | |
*** zhangfei has quit IRC | 11:23 | |
*** derekh has joined #openstack-infra | 11:25 | |
*** dave-mccowan has quit IRC | 11:26 | |
*** dpawlik has quit IRC | 11:27 | |
*** dave-mcc_ has joined #openstack-infra | 11:27 | |
*** ykarel has joined #openstack-infra | 11:28 | |
*** udesale_ has joined #openstack-infra | 11:28 | |
*** yamamoto has quit IRC | 11:29 | |
*** udesale has quit IRC | 11:31 | |
*** ldnunes has joined #openstack-infra | 11:32 | |
*** udesale_ has quit IRC | 11:33 | |
*** armaan has quit IRC | 11:33 | |
*** armaan_ has joined #openstack-infra | 11:33 | |
*** jpena is now known as jpena|lunch | 11:33 | |
*** tpsilva has joined #openstack-infra | 11:37 | |
*** lifeless_ has joined #openstack-infra | 11:37 | |
*** ykarel has quit IRC | 11:37 | |
*** lifeless has quit IRC | 11:38 | |
*** ociuhandu has quit IRC | 11:38 | |
*** amoralej is now known as amoralej|lunch | 11:38 | |
hwoarang | ianw: is there something else that needs to be done for https://review.openstack.org/#/c/566924/ ? perhaps manual testing before W+1? | 11:40 |
*** dpawlik has joined #openstack-infra | 11:41 | |
*** apple01 has quit IRC | 11:42 | |
ianw | hwoarang: hmm, i don't think so, maybe jhesketh was thinking to run it manually? | 11:43 |
*** alexchadin has joined #openstack-infra | 11:44 | |
ianw | is it going to double disk usage? | 11:44 |
hwoarang | yep... do you want to omit 42.3 since it should go away soon? | 11:44 |
jhesketh | I was jet lagged and not going to be around much longer when I +2'd that so I couldn't babysit it (not that it necessarily needs it) | 11:45 |
jhesketh | I'm not around much longer tonight either, so probably not the best person to +w | 11:45 |
hwoarang | or we can get it as is and will clean it up when we remove 42.3 images | 11:46 |
ianw | ianw@mirror-update01:/afs/openstack.org/mirror/opensuse$ fs quota | 11:46 |
ianw | 86% of quota used. | 11:46 |
ianw | so it doesn't have room for essentially doubling itself | 11:46 |
ianw | hwoarang: i also have to head off (you got the two .au people to review :) but i can sync it in the morning, and we can remove 42.3 asap? | 11:48 |
hwoarang | ianw: current code is broken becuase it syncs all distros instead of the ones we want | 11:48 |
hwoarang | so disk usage will actually go down | 11:49 |
*** apple01 has joined #openstack-infra | 11:49 | |
*** armaan_ has quit IRC | 11:49 | |
*** rosmaita has joined #openstack-infra | 11:49 | |
hwoarang | no worries. feel free to merge it tomorrow and i will be around to keep an eye on it during EU working hours | 11:49 |
ianw | hwoarang: alright, yeah, so babysitting still seems like a good idea :) i will take it as a todo item to do it in my morning | 11:50 |
hwoarang | great thanks! | 11:50 |
*** neiloy has joined #openstack-infra | 11:53 | |
*** gyankum has joined #openstack-infra | 11:54 | |
*** rlandy has joined #openstack-infra | 11:56 | |
*** mikal has joined #openstack-infra | 11:57 | |
*** mikal_ has quit IRC | 11:59 | |
*** rlandy is now known as rlandy|rover | 12:03 | |
*** yamamoto has joined #openstack-infra | 12:07 | |
*** kgiusti has joined #openstack-infra | 12:08 | |
*** panda|off is now known as panda | 12:10 | |
*** Tahvok has quit IRC | 12:15 | |
*** sree_ has joined #openstack-infra | 12:19 | |
*** sree_ is now known as Guest75728 | 12:20 | |
*** Guest75728 has quit IRC | 12:24 | |
*** jpena|lunch is now known as jpena | 12:28 | |
*** udesale has joined #openstack-infra | 12:29 | |
*** Odd_Bloke has joined #openstack-infra | 12:31 | |
*** trown|outtypewww is now known as trown | 12:31 | |
*** armaan has joined #openstack-infra | 12:32 | |
*** alexchadin has quit IRC | 12:33 | |
*** uberjay has quit IRC | 12:34 | |
*** uberjay has joined #openstack-infra | 12:35 | |
*** e0ne has joined #openstack-infra | 12:36 | |
openstackgerrit | Brian Rosmaita proposed openstack-infra/zuul-jobs master: Handle -/_ ambiguity in package names https://review.openstack.org/571005 | 12:36 |
*** Odd_Bloke has left #openstack-infra | 12:38 | |
*** ykarel has joined #openstack-infra | 12:38 | |
*** lifeless_ has quit IRC | 12:41 | |
*** amoralej|lunch is now known as amoralej | 12:41 | |
*** salv-orlando has joined #openstack-infra | 12:43 | |
*** salv-orlando has quit IRC | 12:43 | |
*** salv-orlando has joined #openstack-infra | 12:44 | |
*** salv-orlando has quit IRC | 12:44 | |
*** armaan has quit IRC | 12:44 | |
*** salv-orlando has joined #openstack-infra | 12:44 | |
*** salv-orlando has quit IRC | 12:44 | |
*** salv-orlando has joined #openstack-infra | 12:44 | |
*** apple01 has quit IRC | 12:44 | |
*** salv-orlando has quit IRC | 12:45 | |
*** salv-orlando has joined #openstack-infra | 12:45 | |
*** salv-orlando has quit IRC | 12:45 | |
*** salv-orlando has joined #openstack-infra | 12:46 | |
*** eharney has quit IRC | 12:46 | |
*** salv-orlando has quit IRC | 12:46 | |
*** udesale has quit IRC | 12:47 | |
*** udesale has joined #openstack-infra | 12:47 | |
*** edmondsw has joined #openstack-infra | 12:50 | |
fungi | dtantsur: i think you want to propose it to https://git.openstack.org/cgit/openstack/openstack-manuals/tree/www/.htaccess but better to ask the docs team in #openstack-doc | 12:52 |
*** jamesmcarthur has joined #openstack-infra | 12:53 | |
*** markvoelker has joined #openstack-infra | 12:54 | |
dtantsur | k thanks fungi | 12:54 |
*** vickyc has joined #openstack-infra | 12:57 | |
*** mriedem has joined #openstack-infra | 12:59 | |
openstackgerrit | Alex Schultz proposed openstack-infra/project-config master: Add python jobs to tripleo-image-elements https://review.openstack.org/571198 | 12:59 |
*** yamamoto has quit IRC | 13:04 | |
*** jcoufal has joined #openstack-infra | 13:05 | |
*** caphrim007 has quit IRC | 13:06 | |
*** benj_ has quit IRC | 13:08 | |
*** armaan has joined #openstack-infra | 13:08 | |
*** benj_ has joined #openstack-infra | 13:09 | |
*** ykarel_ has joined #openstack-infra | 13:13 | |
*** ykarel has quit IRC | 13:16 | |
*** rossella_s has quit IRC | 13:16 | |
*** wolverineav has joined #openstack-infra | 13:18 | |
*** rossella_s has joined #openstack-infra | 13:20 | |
*** ykarel_ is now known as ykarel | 13:21 | |
*** cshastri has quit IRC | 13:23 | |
mordred | clarkb: sorry about the issue with the inventory script - the main issue is that 'import openstack' breaks in a script called 'openstack' | 13:24 |
mordred | this is a good reminder that we should switch from the inventory script to the inventory plugin though - I'll put that into the ansible spec | 13:24 |
*** markvoelker has quit IRC | 13:24 | |
*** rossella_s has quit IRC | 13:25 | |
*** esberglu has joined #openstack-infra | 13:26 | |
*** rossella_s has joined #openstack-infra | 13:27 | |
*** links has quit IRC | 13:29 | |
*** links has joined #openstack-infra | 13:30 | |
*** eharney has joined #openstack-infra | 13:32 | |
*** yamamoto has joined #openstack-infra | 13:35 | |
*** cshastri has joined #openstack-infra | 13:35 | |
*** yamamoto has quit IRC | 13:39 | |
*** nicolasbock has quit IRC | 13:40 | |
mordred | fungi, stephenfin: https://review.openstack.org/#/c/559348/ should be ready to land now - are we comfortable doing that? | 13:40 |
stephenfin | mordred: Yeah, I think it's good. Have tested with various DNM patches and submitted the patch to the zuul ML more than 2 weeks ago | 13:41 |
mordred | kk. I'm going to pull the trigger. let's keep our eyes out for unexpected carnage | 13:42 |
*** rossella_s has quit IRC | 13:43 | |
*** rossella_s has joined #openstack-infra | 13:44 | |
*** jesslampe has joined #openstack-infra | 13:44 | |
*** links has quit IRC | 13:44 | |
*** cshastri has quit IRC | 13:45 | |
fungi | seems like it's time, yes | 13:47 |
*** rossella_s has quit IRC | 13:49 | |
*** rossella_s has joined #openstack-infra | 13:51 | |
*** nicolasbock has joined #openstack-infra | 13:53 | |
corvus | An unexpected error occurred: "https://registry.yarnpkg.com/@babel/traverse/-/traverse-7.0.0-beta.42.tgz: Request failed \"503 Service Unavailable\"". | 13:54 |
corvus | it's not a good week for js package hosting | 13:54 |
*** VW has joined #openstack-infra | 13:56 | |
*** sree has joined #openstack-infra | 13:59 | |
*** lpetrut has quit IRC | 14:00 | |
*** Tahvok has joined #openstack-infra | 14:00 | |
*** bhujay has quit IRC | 14:08 | |
*** EmilienM_PTO is now known as EmilienM | 14:08 | |
*** Guest62139 has quit IRC | 14:13 | |
*** janki has quit IRC | 14:15 | |
*** armaan has quit IRC | 14:18 | |
*** signed8bit has joined #openstack-infra | 14:18 | |
*** r-daneel has joined #openstack-infra | 14:18 | |
*** armaan has joined #openstack-infra | 14:19 | |
*** flaper87 has joined #openstack-infra | 14:20 | |
*** rajinir has joined #openstack-infra | 14:25 | |
*** felipemonteiro has joined #openstack-infra | 14:25 | |
*** kien-ha has joined #openstack-infra | 14:29 | |
corvus | that's 2 failures. http://status.yarnpkg.com/ is green. | 14:30 |
corvus | aiui, the npm status page was also "green" during the recent bad times | 14:30 |
corvus | we should totally have an infra status page that's always green | 14:31 |
*** rpioso|afk is now known as rpioso | 14:33 | |
*** hongbin has joined #openstack-infra | 14:33 | |
*** kjackal has joined #openstack-infra | 14:33 | |
*** dklyle has joined #openstack-infra | 14:33 | |
mordred | corvus: we can make one of those :) | 14:34 |
*** pblaho has quit IRC | 14:35 | |
mordred | corvus: incidentally though - the zuul jobs are not currently using the infra reverse-proxy mirror | 14:36 |
mordred | it's on my list of things to work on today | 14:36 |
*** r-daneel_ has joined #openstack-infra | 14:36 | |
*** r-daneel has quit IRC | 14:37 | |
*** r-daneel_ is now known as r-daneel | 14:37 | |
mordred | (for npm things) | 14:38 |
*** felipemonteiro has quit IRC | 14:42 | |
corvus | mordred: and i guess yarn things too? (i'm assuming yarn needs its own configuration separate from anything you'd do for npm?) | 14:42 |
*** jamesmcarthur has quit IRC | 14:42 | |
*** benj_ has quit IRC | 14:44 | |
*** sree has quit IRC | 14:45 | |
*** sree has joined #openstack-infra | 14:45 | |
*** eernst has joined #openstack-infra | 14:46 | |
*** benj_ has joined #openstack-infra | 14:46 | |
clarkb | mordred: thats fine however the old script could've remained in place and continued to use shade | 14:48 |
clarkb | mordred: allowing users to migrate from one to the other more gracefully | 14:48 |
clarkb | but I guess that still has import problems | 14:48 |
*** armaan has quit IRC | 14:49 | |
clarkb | mordred: as for switching to the _inventory.py script should it just work and I should test it and switch pupept to that or should I move ahead with the pin instead? | 14:49 |
*** sree has quit IRC | 14:50 | |
fungi | clarkb: where did the survey.o.o certs/key end up? didn't see them at puppetmaster:~root/certs/ | 14:51 |
clarkb | mordred: basically my bigger concern at this point is less the rename of the script and more that the script internals changed too. A legacy_openstack.py inventory script would be helpful | 14:52 |
clarkb | fungi: oh they are still in my inbox because summit. I will copy them now | 14:52 |
fungi | no worries, just trying to get everything into hiera in preparation for a production build | 14:52 |
*** ykarel is now known as ykarel|away | 14:53 | |
fungi | if you want to just stuff them into /etc/puppet/hieradata/production/group/survey.yaml that works too | 14:53 |
mordred | clarkb: yes - it should just work - no internals should have changed at all | 14:54 |
mordred | clarkb: the shade code in openstacksdk is a direct copy of the code in shade | 14:55 |
*** dave-mcc_ is now known as dave-mccowan | 14:55 | |
clarkb | mordred: ok that commit message implied things were rewritten but I didn't have time to look at the diff yet | 14:55 |
clarkb | fungi: I put them in /root/certs | 14:56 |
mordred | clarkb: just to import from openstack instead of shade ... we could add an openstack.py script back that imports from shade - but we'd already been getting bug reports from people about the fact that it didn't work with newer shade because of the use of openstacksdk inside of os-client-config | 14:57 |
fungi | thanks, clarkb! | 14:57 |
clarkb | mordred: ok I'm gonna run a --list --cloud somecloud test and if that works I'll convert the puppetry over to use _inventory | 14:58 |
*** jgrassler has quit IRC | 14:58 | |
*** jamesmcarthur has joined #openstack-infra | 14:58 | |
clarkb | does not work :/ | 14:59 |
clarkb | keystoneauth1.exceptions.auth_plugins.MissingRequiredOptions: Auth plugin requires parameters which were not given: auth_url | 14:59 |
clarkb | I did --list --cloud openstackci-limestone as a regular user. auth_url is set for openstackci-limestone in /etc/openstack/clouds.yaml | 15:00 |
*** armaan has joined #openstack-infra | 15:01 | |
mordred | clarkb: ok. lemme look real quick - is this on puppetmaster? | 15:01 |
clarkb | mordred: yes | 15:01 |
clarkb | clarkb@puppetmaster:~$ /opt/ansible/contrib/inventory/openstack_inventory.py --list --cloud openstackci-limestone | 15:01 |
mordred | is that user in the admin group? | 15:01 |
clarkb | yes | 15:01 |
*** ykarel|away is now known as ykarel | 15:02 | |
openstackgerrit | Merged openstack-infra/zuul-jobs master: Default warning-is-error to True for non-legacy Sphinx projects https://review.openstack.org/559348 | 15:02 |
clarkb | I can read /etc/openstack/clouds.yaml as this user | 15:02 |
mordred | yes. I can too - and I can confirm the same error | 15:02 |
*** gyee has joined #openstack-infra | 15:05 | |
*** zhangfei has joined #openstack-infra | 15:06 | |
*** apple01 has joined #openstack-infra | 15:07 | |
mordred | clarkb: python -m openstack.config.loader is printing out config correctly including openstackci-limestone - I'm digging further | 15:08 |
*** markvoelker has joined #openstack-infra | 15:08 | |
*** lpetrut has joined #openstack-infra | 15:09 | |
*** bhujay has joined #openstack-infra | 15:12 | |
*** bhujay has quit IRC | 15:16 | |
*** r-daneel has quit IRC | 15:16 | |
*** bhujay has joined #openstack-infra | 15:17 | |
fungi | heading out to grab some food, back in a bit | 15:19 |
*** dpawlik has quit IRC | 15:19 | |
*** rfolco_ has joined #openstack-infra | 15:21 | |
mordred | clarkb: ok. it's a bug. https://github.com/ansible/ansible/pull/40892 https://github.com/ansible/ansible/pull/40894 | 15:21 |
clarkb | mordred: weird, is it maybe not looking in the auth: dict? | 15:21 |
clarkb | oh cool you've figured it out | 15:21 |
mordred | Shrews: ^^ got a sec to fasttrack land a couple of tiny patches? | 15:21 |
Shrews | ya | 15:22 |
Shrews | mordred: let me know when shippable gives the ok | 15:23 |
*** dhinesh has joined #openstack-infra | 15:23 | |
clarkb | ah so it wasn't looking in the cloud that was specified. Was the behavior in that case to expect a single cloud in the clouds.yaml file? | 15:23 |
clarkb | I was still in keystoneauth wondering why I couldn't find where auth_url is defined as a required opt and that is because it is auth-url when defined as an opt | 15:23 |
*** rfolco has quit IRC | 15:23 | |
mordred | :) | 15:24 |
mordred | clarkb: yah - which is not the intent... | 15:24 |
clarkb | I wonder if I can teach git grep to treat _ and - the same when I use -i | 15:24 |
*** r-daneel has joined #openstack-infra | 15:25 | |
mordred | clarkb: that would be the best feature to git grep ever | 15:25 |
*** Guest48153 is now known as melwitt | 15:25 | |
*** zhangfei has quit IRC | 15:25 | |
clarkb | mordred: Shrews ansible might want to drop fedora 24 and 25 testing at this point, those releases are no longer supported aiui | 15:27 |
clarkb | oh those are the only fedora tests, probably want to add 26-28 first | 15:28 |
*** jesslampe has quit IRC | 15:28 | |
*** jamesmcarthur has quit IRC | 15:31 | |
*** janki has joined #openstack-infra | 15:33 | |
*** pcaruana has quit IRC | 15:33 | |
hrw | clarkb: abandon f26 too - it is close to EOL | 15:33 |
mordred | clarkb, Shrews: https://review.openstack.org/#/q/topic:ansible-openstacksdk+status:open are testing updated to make sure we're cross-testing ansible and openstacksdk | 15:36 |
mordred | (and also to test stable-2.5 of ansible against shade) | 15:36 |
*** udesale has quit IRC | 15:36 | |
Shrews | mordred: do we have any openstack-inventory tests? | 15:37 |
Shrews | obviously nothing that could have caught the bug above | 15:37 |
mordred | Shrews: I believe we have no openstack-inventory tests | 15:39 |
mordred | Shrews: or, rather, we have no tests of that script - we do have tests of the underlying inventory code itself | 15:40 |
mordred | but the ansible inventory script wrapper stuff (which is what broke here) is missing a test | 15:40 |
*** dhinesh has quit IRC | 15:41 | |
*** markvoelker has quit IRC | 15:41 | |
*** udesale has joined #openstack-infra | 15:43 | |
*** caphrim007 has joined #openstack-infra | 15:44 | |
*** jgrassler has joined #openstack-infra | 15:44 | |
*** jodh has joined #openstack-infra | 15:44 | |
*** tung_doan has joined #openstack-infra | 15:45 | |
*** jodh has left #openstack-infra | 15:46 | |
tung_doan | Hi Infra team, could you guys give one more +2 to our patch: https://review.openstack.org/#/c/570363/ | 15:46 |
tung_doan | Thanks | 15:46 |
clarkb | tung_doan: is there any reason to not add that config in your project itself? | 15:47 |
clarkb | I think we are going to have to start slowly pushing back on these chagnes that should be made in repo | 15:47 |
clarkb | but before I say you should do it in repo I want to make sure I'm not missing something important | 15:48 |
clarkb | oh monty approved | 15:49 |
openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Run openstacksdk ansible tests on stable-2.6 of ansible https://review.openstack.org/571238 | 15:49 |
clarkb | tung_doan: in general I think the preference would be to have that config in your project repos. We are in a transition state currently where config is still centrally managed in many cases but we should be shifting away from that over time | 15:50 |
mordred | clarkb: oh - I can remove the +W if you want | 15:50 |
clarkb | mordred: I think its fine if people are reviewing them | 15:50 |
*** ykarel is now known as ykarel|away | 15:50 | |
*** jgrassler has quit IRC | 15:50 | |
clarkb | its mostly an anti pattern for us to have to review those especially as part of being pinged imo though | 15:50 |
clarkb | if it is important and can be done in repo just get it done in repo :) | 15:50 |
Shrews | ++ | 15:50 |
mordred | kk. and I agree - I'd like to be as little in the business of central job management as possible | 15:50 |
*** felipemonteiro has joined #openstack-infra | 15:50 | |
tung_doan | clarkb: we are planning to add functional tests in repo. But for py35 and py27, you mean I can add them to my own repo? | 15:51 |
mordred | Shrews: one more ansible test patch: https://review.openstack.org/57123 | 15:51 |
clarkb | tung_doan: yes | 15:51 |
*** apple01 has quit IRC | 15:51 | |
Shrews | mordred: ??? | 15:51 |
Shrews | bad linky | 15:52 |
mordred | Shrews: I suck at copy-pasta https://review.openstack.org/571238 | 15:52 |
clarkb | mordred: wrong change link | 15:52 |
mordred | yah | 15:52 |
Shrews | mordred: zuul'd -1, so i skipped it | 15:52 |
Shrews | zuul -1'd, rather | 15:52 |
mordred | oh - that one needed a depends-on - I'll circle back around on it | 15:52 |
openstackgerrit | Merged openstack-infra/project-config master: Add pep8, py27, and py35 test to apmec project https://review.openstack.org/570363 | 15:53 |
tung_doan | clarkb: I did not know that before. I investigate that many projects just put func. tests in repo and move unit tests to infra :) | 15:53 |
clarkb | tung_doan: the only repo specific requirement (and I'm sure I'll get this wrong due to other corner cases) are roles playbooks used by jobs must be in the same repo as the job definition and secrets can't be used by jobs defined in other repos | 15:55 |
*** lpetrut has quit IRC | 15:55 | |
clarkb | tung_doan: you can consume jobs that are defined in other repos just fine | 15:55 |
Shrews | mordred: -1 from zuul on https://review.openstack.org/570682 also too as well | 15:56 |
*** mriedem has quit IRC | 15:56 | |
*** mriedem has joined #openstack-infra | 15:56 | |
* Shrews runs out of adjectives | 15:56 | |
*** jamesmcarthur has joined #openstack-infra | 15:56 | |
Shrews | or adverbs | 15:57 |
Shrews | or whatever | 15:57 |
tung_doan | clarkb: yep. got it | 15:57 |
*** dhinesh has joined #openstack-infra | 15:58 | |
*** hasharAway is now known as hashar | 16:01 | |
*** felipemonteiro has quit IRC | 16:01 | |
*** felipemonteiro has joined #openstack-infra | 16:01 | |
*** jamesmcarthur has quit IRC | 16:01 | |
*** ykarel|away has quit IRC | 16:02 | |
*** kiennt26 has joined #openstack-infra | 16:02 | |
openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Run openstacksdk ansible tests on stable-2.6 of ansible https://review.openstack.org/571238 | 16:02 |
clarkb | mordred: Shrews should I expect the ansible bug to be fixed quickly enough to switch over to that inventory script or should we consider https://review.openstack.org/#/c/571045/ to get puppet running again? | 16:03 |
mordred | Shrews: ok. that ^^ and https://review.openstack.org/571240 should be happier now | 16:03 |
clarkb | the background here is I'm trying to update the default apache vhost on our git mirrors so that clietns without SNI support hit git.openstack.org | 16:03 |
mordred | clarkb: either way - we'll get the ansible patch landed as soon as shippale returns green | 16:04 |
clarkb | they are the existing clients of the platform and zuul and airship and others can make sure their clients can SNI | 16:04 |
*** zoli is now known as zoli|gone | 16:04 | |
*** zoli|gone is now known as zoli | 16:04 | |
clarkb | mordred: it hasn't started any tests yet :/ | 16:04 |
clarkb | I'm not sure what shippables throughoput is typically like | 16:04 |
mordred | it might be a sec | 16:05 |
clarkb | doesn't look like they give me that global state info as an unauthenticated user | 16:05 |
mordred | clarkb: so maybe let's just land fungi's patch for now, and we can revert it as soon as things are landed | 16:05 |
mordred | oh - or your patch that is | 16:05 |
clarkb | I'm happy to do that and separately get the inventory stuff updated | 16:05 |
mordred | yah | 16:05 |
mordred | because honestly I want to get us updated to using inventory instead of inventory script anyway | 16:06 |
*** camunoz has joined #openstack-infra | 16:06 | |
clarkb | ansible has non contrib openstack inventory support? or you mean explicit file based inventory? | 16:06 |
*** dhinesh_ has joined #openstack-infra | 16:07 | |
*** tung_doan has quit IRC | 16:07 | |
mordred | no, I mean there is now non-contrib openstack inventory support - and the new system also supports stacking inventory sources | 16:07 |
*** electrofelix has quit IRC | 16:08 | |
mordred | https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/inventory/openstack.py | 16:08 |
mordred | clarkb: (but I'm going to include notes aout that in the next version of the config management spec, since it should ultimately allow us to have a less janky inventory and host/group var experience) | 16:10 |
*** dhinesh__ has joined #openstack-infra | 16:10 | |
*** dhinesh has quit IRC | 16:10 | |
*** dhinesh__ has quit IRC | 16:11 | |
*** dhinesh_ has quit IRC | 16:12 | |
fungi | back, catching up on which "fungi's patch" this is | 16:18 |
stephenfin | fungi: quick question - do infra consume zuul master or zuul packages? | 16:19 |
*** panda is now known as panda|off | 16:19 | |
mordred | fungi: it's the one that's not fungi's patch but that mordred misidentified as such | 16:19 |
fungi | clarkb: git grep will treat . the same as - and _ if that helps! ;) | 16:19 |
clarkb | fungi: was my patch to pin ansible. We are going to do that short term to get puppet working again then switch to the new inventory script when mordred's bug fixes get in upstream | 16:19 |
mordred | stephenfin: master-ish | 16:19 |
mordred | stephenfin: we run from master, but don't restart the scheduler necessarily on every patch | 16:19 |
stephenfin | mordred: So we'll see https://review.openstack.org/#/c/559348/ pop up before...3.0.3 (or whatever the next release) is cut? | 16:20 |
*** kjackal has quit IRC | 16:20 | |
mordred | stephenfin: although we do make sure to restart it before cutting release tags | 16:20 |
mordred | stephenfin: ah - zuul-jobs is from master | 16:20 |
mordred | stephenfin: zuul will automatically update its config the instant that lnads | 16:20 |
mordred | so - it should be live now | 16:20 |
stephenfin | mordred: Ah, different things of course. Cool cool | 16:20 |
*** dhajare_ has quit IRC | 16:20 | |
*** kiennt26 has quit IRC | 16:21 | |
stephenfin | dhellmann suggested I be a good citizen and tell people that was happening | 16:21 |
mordred | ++ | 16:21 |
dhellmann | because the answer to people complaining about the volume of email on openstack-dev is to fix the signal/noise ratio, and that means sending more email! | 16:22 |
fungi | clarkb: mordred: oh, yes, the pin ansible patch. i think i +2'd already but will double-check | 16:22 |
clarkb | fungi: you did | 16:22 |
fungi | perfect, one less thing for me to check | 16:22 |
mordred | dhellmann: https://xkcd.com/927/ | 16:22 |
dhellmann | mordred : s/standard/email thread/ | 16:23 |
clarkb | that moment you discover you have +2 on openstacksdk. /me promises to only use it for adding these tests :P | 16:23 |
*** pcaruana has joined #openstack-infra | 16:23 | |
mordred | clarkb: you are more than welcome to use it any time you want - you're not there by accident ;) | 16:24 |
*** lpetrut has joined #openstack-infra | 16:24 | |
*** trown is now known as trown|lunch | 16:26 | |
*** udesale has quit IRC | 16:26 | |
openstackgerrit | Merged openstack-infra/system-config master: Pin /opt/ansible as source of openstack inventory https://review.openstack.org/571045 | 16:28 |
* dtantsur knows one more person to ping for openstacksdk reviews, w00t :D | 16:28 | |
clarkb | mordred: I don't undertand what they one chagne keeps reporting it depends on a change that failed to merge | 16:31 |
clarkb | its git parent ran tests fine and its depends on is running tests too | 16:32 |
fungi | dtantsur: the infra-core group are transitively core on openstacksdk by way of having been core on shade from back when it was still an infra project | 16:32 |
fungi | i doubt too many of us are exercising that, however | 16:32 |
clarkb | mordred: (/Stage[main]/Openstack_project::Puppetmaster/Vcsrepo[/opt/ansible]/ensure) Please, commit your changes or stash them before you can switch branches. | 16:33 |
clarkb | mordred: I'm assuming the loca changes on the puppet master are you? | 16:33 |
clarkb | mordred: can we stash the delta or delete the changes? | 16:33 |
mordred | clarkb: fine to delete | 16:34 |
clarkb | ok doing that now | 16:34 |
*** kjackal has joined #openstack-infra | 16:34 | |
*** jgrassler has joined #openstack-infra | 16:35 | |
clarkb | ok next puppet run should get us back to puppeting the git servers, we can check git08 is happy then deploy the sni fix to the rest of the cluster | 16:35 |
mordred | woot | 16:36 |
clarkb | then remove the ansible pin once openstack_inventory fixes merge and are tested | 16:36 |
clarkb | I'll just fetch that file from ansible devel when it is updated and retest | 16:36 |
*** caphrim007 has quit IRC | 16:40 | |
*** jpena is now known as jpena|off | 16:41 | |
mordred | ++ | 16:42 |
*** marshallmargenau has joined #openstack-infra | 16:44 | |
*** signed8bit is now known as signed8bit_Zzz | 16:44 | |
*** salv-orlando has joined #openstack-infra | 16:47 | |
*** salv-orlando has quit IRC | 16:47 | |
*** salv-orlando has joined #openstack-infra | 16:48 | |
*** salv-orlando has quit IRC | 16:48 | |
*** salv-orlando has joined #openstack-infra | 16:48 | |
*** rlandy|rover is now known as rlandy|rover|brb | 16:49 | |
*** salv-orlando has quit IRC | 16:49 | |
*** salv-orlando has joined #openstack-infra | 16:49 | |
*** salv-orlando has quit IRC | 16:50 | |
*** salv-orlando has joined #openstack-infra | 16:50 | |
*** armaan has quit IRC | 16:50 | |
*** salv-orlando has quit IRC | 16:50 | |
*** salv-orlando has joined #openstack-infra | 16:51 | |
*** salv-orlando has quit IRC | 16:51 | |
*** jpena|off is now known as jpena | 16:52 | |
openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Allow templates to be optional for zuul-migrate https://review.openstack.org/570979 | 16:54 |
openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Allow for projects only names in zuul-migrate https://review.openstack.org/570980 | 16:54 |
clarkb | mordred: any idea why ansible doesn't use the master branch? | 16:55 |
clarkb | reviewing these new jobs it occurred to me that tool conventions are only going to be more important as we do more of this cross community stuff | 16:56 |
clarkb | and devel feels like a very unconventional git pattern | 16:56 |
*** sambetts is now known as sambetts|afk | 16:59 | |
clarkb | mordred: also http://logs.openstack.org/40/571240/1/check/openstacksdk-ansible-devel-functional-devstack/1a600cb/job-output.txt.gz#_2018-05-30_16_29_57_495957 do you want to merge the new jobs as is or fix that first? | 16:59 |
*** rpioso is now known as rpioso|eat | 16:59 | |
*** dtantsur is now known as dtantsur|afk | 17:00 | |
*** hashar is now known as hasharAway | 17:00 | |
mordred | clarkb: hysterical raisins, to the best of my knowledge | 17:01 |
mordred | clarkb: I can tell you it always messes with my muscle memory | 17:01 |
*** derekh has quit IRC | 17:01 | |
*** e0ne has quit IRC | 17:02 | |
*** e0ne has joined #openstack-infra | 17:02 | |
fungi | clarkb: i wonder if they were using svn before | 17:02 |
clarkb | I'm going to date myself. I jumped straight from rcs to bzr | 17:03 |
clarkb | did svn use devel? | 17:03 |
mordred | no - it was a conscious choice on the part of early devs | 17:03 |
fungi | basically it learned that pattern from rcs/cvs, sure. the predominant paradigm was to do development work on branches and commit stable code to the trunk | 17:03 |
mordred | someone told me why thatperson thought it was a good idea a while back but I've long since forgotten | 17:04 |
fungi | where git's "master" branch is the rough equivalent of the svn "trunk" | 17:04 |
clarkb | it used file level checkouts though right? | 17:04 |
clarkb | which are their own set of complications | 17:04 |
* clarkb glares at borland starteam | 17:04 | |
clarkb | ok git08 got puppetted | 17:04 |
clarkb | and puppet appears to have moved the 50- prefix file to the 25- prefix | 17:05 |
clarkb | testing sni now | 17:05 |
fungi | yeah, i'm doing my best not to bump th eneurons containing pre-git vcs memories | 17:05 |
clarkb | `openssl s_client -connect git08.openstack.org:4443` gives me git.o.o not git.airshipit.org now. | 17:05 |
fungi | yeah, that's looking right to me too | 17:05 |
clarkb | I'm going to add git08 back to haproxy then remove 01-07 from the ansible emergency file | 17:05 |
fungi | sounds great | 17:06 |
fungi | clarkb: /etc/httpd/conf.d/ doesn't seem to have ended up with two configs for git.o.o, so looking good | 17:07 |
*** amoralej is now known as amoralej|off | 17:07 | |
fungi | that puppet module is smarter than i'd expected | 17:07 |
clarkb | yup | 17:07 |
clarkb | I've gone ahead and put 08 back into haproxy and removed 01-07 from the emergency file. Next puppet run should update all the backends then I can circle back around on updating the inventory file source | 17:08 |
clarkb | efried: in about half an hour I think you can retest if ssl certs are happier | 17:08 |
clarkb | efried: I'll confirm with you when I think it is "fixed" on our end and you can double check | 17:08 |
*** tesseract has quit IRC | 17:10 | |
clarkb | Is it bad that I'm happy apache makes this intuitive | 17:10 |
clarkb | maybe happy is wrong word, surprised is probably better | 17:11 |
*** yamahata has joined #openstack-infra | 17:11 | |
*** signed8bit_Zzz is now known as signed8bit | 17:12 | |
fungi | it is a bit shocking that it works this conveniently, sure | 17:14 |
*** felipemonteiro has quit IRC | 17:15 | |
clarkb | mordred: what is the get_one behavior if you pass nothing? is it lookin for a cloud with name key None? | 17:15 |
clarkb | Shrews: https://github.com/ansible/ansible/pull/40892 passed tests | 17:16 |
*** felipemonteiro has joined #openstack-infra | 17:16 | |
*** jpena is now known as jpena|off | 17:18 | |
*** felipemonteiro has quit IRC | 17:20 | |
*** gyankum has quit IRC | 17:23 | |
*** felipemonteiro has joined #openstack-infra | 17:25 | |
*** ssbarnea_ has quit IRC | 17:27 | |
Shrews | clarkb: yeah, the other didn't though (though it seems unrelated). i'd like to land them both at the same time | 17:27 |
Shrews | clarkb: gotta step out for a bit, but will check them again when i return | 17:28 |
clarkb | Shrews: thanks | 17:29 |
*** markvoelker has joined #openstack-infra | 17:30 | |
mordred | clarkb: yes. if you only have one cloud defined, it'll use that one, but otherwise it's usually not particularly useful thing to do | 17:30 |
openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Switch to using new ansible openstack inventory script https://review.openstack.org/571263 | 17:31 |
clarkb | I WIP'd that I could've actually dependson'd it now that I think about it | 17:31 |
clarkb | oh well | 17:31 |
*** jpich has quit IRC | 17:31 | |
clarkb | mordred: but you might want to go ahead and review that ^ lines up with what you expect to be the future state | 17:32 |
*** aojea has quit IRC | 17:33 | |
*** signed8bit is now known as signed8bit_Zzz | 17:35 | |
*** dhinesh has joined #openstack-infra | 17:38 | |
*** felipemonteiro has quit IRC | 17:38 | |
*** trown|lunch is now known as trown | 17:39 | |
clarkb | #status log git08 added back to git.o.o haproxy and all git backends updated to make git.openstack.org vhost their default vhost. This means that https clients that don't speak SNI will get the cert for git.o.o (and talk to git.o.o vhost) by default. | 17:40 |
clarkb | efried: ^ I think you can check if things are happier for you now | 17:40 |
clarkb | hrm am I not identified | 17:41 |
openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul master: Add missing pathlib dependency https://review.openstack.org/570539 | 17:41 |
openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul master: Add missing pathlib dependency https://review.openstack.org/570539 | 17:42 |
clarkb | #status log git08 added back to git.o.o haproxy and all git backends updated to make git.openstack.org vhost their default vhost. This means that https clients that don't speak SNI will get the cert for git.o.o (and talk to git.o.o vhost) by default. | 17:42 |
openstackstatus | clarkb: finished logging | 17:42 |
fungi | clarkb: yeah, i think freenode wound up inadvertently clearing nickserv sessions over the weekend when doing services upgrades | 17:45 |
*** signed8bit_Zzz is now known as signed8bit | 17:45 | |
fungi | i have autokick set on my nick so spotted it right away | 17:45 |
*** ssbarnea_ has joined #openstack-infra | 17:45 | |
*** caphrim007 has joined #openstack-infra | 17:45 | |
clarkb | next fixing storyboard | 17:46 |
clarkb | fungi: you haven't had any great ideas re ^ have you? | 17:46 |
*** rlandy|rover|brb is now known as rlandy|rover | 17:46 | |
openstackgerrit | Merged openstack/gertty master: Add message attribute to DisplayError https://review.openstack.org/569683 | 17:46 |
*** caphrim007 has quit IRC | 17:47 | |
*** caphrim007 has joined #openstack-infra | 17:47 | |
fungi | clarkb: other than retooling storyboard-webclient as some sort of packaged distribution, not really | 17:50 |
corvus | zxiiro: i sent an email to the odl helpdesk asking for streamEvents to be enabled for openstack-zuul, that hasn't happened yet, and i don't think i got an acknowledgement. is that something you can check on? | 17:50 |
openstackgerrit | Jonathan Bryce proposed openstack-infra/system-config master: Creates new mailing list for discussion of container security https://review.openstack.org/571273 | 17:50 |
fungi | mordred: ^ perhaps this is somewhere in your wheelhouse? trying to figure out the sanest way to deploy storyboard-webclient as a continuous delivery from git master | 17:51 |
fungi | the hardest part is making sure the deployment location gets cleared of old cruft | 17:51 |
mordred | fungi: well - we should be publishing tarballs of the built sourcing | 17:51 |
clarkb | ya we have the published tarballs | 17:51 |
fungi | yeah, we're trying to deploy from those | 17:51 |
clarkb | I kinda want to exec a script to do this | 17:52 |
fungi | i suppose we could have a puppet exec which uses rsync --delete though that seems hacky | 17:52 |
mordred | that's what we do for zuul iirc | 17:52 |
clarkb | which can download the tarball to tmpdir, extract it there, rsync it with a delete to the dest and be done | 17:52 |
corvus | i am not going to suggest afs :) | 17:52 |
fungi | hah | 17:52 |
fungi | i bet we could deploy via nntp | 17:52 |
*** sree has joined #openstack-infra | 17:52 | |
mordred | $zuul_web_filename <-- that's the thing to look for in puppet-zuul | 17:52 |
clarkb | so rather than encode the download, delete, extract, copy in puppet we just have puppet call a script that does it | 17:53 |
*** lpetrut has quit IRC | 17:53 | |
clarkb | command => "rm -rf ${zuul_web_src_root}/* && tar -C ${zuul_web_src_root} -xzf ./${zuul_web_filename}" | 17:53 |
clarkb | that is the trick that zuul uses | 17:53 |
clarkb | that is simple enough I'll make a patch to storyboard that does that | 17:53 |
fungi | hax0rz ftw! | 17:53 |
fungi | let's call that precedent | 17:54 |
mordred | clarkb: amusingly enough, I think the zuul_web puppet code was originally copied from storyboard :) | 17:54 |
fungi | if so, that was long ago. i'm digging back through puppet-storyboard to see when that might have changed | 17:55 |
fungi | gone back as far as 2015 and not finding it | 17:56 |
*** sree has quit IRC | 17:57 | |
openstackgerrit | Clark Boylan proposed openstack-infra/puppet-storyboard master: Clean the webclient unpack dir safely. https://review.openstack.org/570004 | 17:57 |
*** psachin has quit IRC | 17:57 | |
clarkb | uses the old changeid but its a new patchset | 17:57 |
fungi | thx | 17:57 |
clarkb | now to double check if * gets expanded | 17:58 |
*** rpioso|eat is now known as rpioso | 17:58 | |
fungi | yeah, that looks sane assuming globbing works there | 17:58 |
clarkb | https://puppet.com/docs/puppet/5.3/types/exec.html#exec-provider-shell it will work if you set the provider to shell. Unsure if it figures that out on its own | 17:59 |
clarkb | I'll explicitly set it | 17:59 |
clarkb | then we can add it to the zuul resource if we need to | 17:59 |
mordred | fungi: exec { 'get-webclient': and exec { 'unpack-webclient': were copied - I changed the zuul to use rsync instead of the puppet file resource because webpack for zuul produces hashed filenames so without --delete the directory would just grow infinitely | 17:59 |
*** markvoelker has quit IRC | 18:00 | |
fungi | worst case we do like puppet-zuul and remove the dir, then pass -C s that it's created again | 18:00 |
mordred | yah - it's a temp dir anyway | 18:00 |
mordred | clarkb: patch lgtm - want me to wait on you verifying * ? | 18:00 |
clarkb | mordred: ya I'm just going to update it to explicitly set shell | 18:01 |
*** signed8bit is now known as signed8bit_Zzz | 18:01 | |
*** signed8bit_Zzz is now known as signed8bit | 18:01 | |
mordred | kk | 18:01 |
openstackgerrit | Clark Boylan proposed openstack-infra/puppet-storyboard master: Clean the webclient unpack dir safely. https://review.openstack.org/570004 | 18:02 |
clarkb | storyboard and sni were the two things I wanted to address that were largely ignored during the summit. Anything else I'm forgetting or continuing to ignore? | 18:04 |
openstackgerrit | Merged openstack-infra/zuul master: correct github review approval type is `approved` https://review.openstack.org/570378 | 18:06 |
fungi | not to my knowledge. i'm mostly focusing on limesurvey and mailman3 at this particular point in time | 18:08 |
*** felipemonteiro has joined #openstack-infra | 18:08 | |
*** pcaruana has quit IRC | 18:09 | |
*** ccamacho has quit IRC | 18:11 | |
*** felipemonteiro has quit IRC | 18:13 | |
*** lpetrut has joined #openstack-infra | 18:14 | |
*** gfidente is now known as gfidente|pto | 18:17 | |
*** gfidente|pto is now known as gfidente|afk | 18:17 | |
zxiiro | corvus: I could check. do you have a ticket? | 18:19 |
openstackgerrit | Merged openstack-infra/zuul master: Add missing pathlib dependency https://review.openstack.org/570539 | 18:19 |
*** mriedem1 has joined #openstack-infra | 18:21 | |
*** mriedem has quit IRC | 18:23 | |
zxiiro | corvus: I don't see any open tickets. which email did you send your request to? | 18:23 |
*** mriedem1 is now known as mriedem | 18:24 | |
*** lpetrut has quit IRC | 18:26 | |
corvus | zxiiro: i did not get a ticket response; i sent mail to helpdesk@opendaylight.org | 18:27 |
zxiiro | corvus: odd, i guess the mail didn't make it over. because I don't see any tickets open about this. Can you try sending again? | 18:28 |
zxiiro | I can watch the inbox and confirm it came in | 18:28 |
corvus | zxiiro: sent | 18:29 |
*** ssbarnea_ has quit IRC | 18:31 | |
zxiiro | corvus: odd. I don't see it. I'm asking internally if someone can check if we're rejecting the mail somehow. | 18:32 |
corvus | zxiiro: if it helps, here's the relevant log from my side (with the ack and queue id from the lf smpt server): 2018-05-30 18:23:11 1fO5kc-0004aB-MA => helpdesk@opendaylight.org R=dnslookup T=remote_smtp H=smtp1.linuxfoundation.org [140.211.169.13] X=TLS1.2:DHE_RSA_AES_128_CBC_SHA1:128 C="250 2.0.0 Ok: queued as 04A13732" | 18:35 |
zxiiro | thanks, I'll pass it along | 18:36 |
*** signed8bit is now known as signed8bit_Zzz | 18:38 | |
*** jcoufal_ has joined #openstack-infra | 18:40 | |
*** signed8bit_Zzz is now known as signed8bit | 18:41 | |
*** jcoufal has quit IRC | 18:44 | |
zxiiro | corvus: can you forward the email to jevans@linuxfoundation.org | 18:44 |
zxiiro | corvus: Apparently it's getting through from smtp level but something's happening inside the ticket system that we need to figure out. | 18:45 |
openstackgerrit | Merged openstack-infra/puppet-storyboard master: Clean the webclient unpack dir safely. https://review.openstack.org/570004 | 18:46 |
*** iranzo has quit IRC | 18:46 | |
corvus | zxiiro: i sent another copy with the same text to jevans (i figured that would be most useful since i don't otherwise have a copy of exactly what was delivered) | 18:48 |
*** diablo_rojo has joined #openstack-infra | 18:48 | |
*** dhinesh has quit IRC | 18:49 | |
*** e0ne has quit IRC | 18:50 | |
*** markvoelker has joined #openstack-infra | 18:50 | |
*** rwsu has quit IRC | 18:51 | |
zxiiro | corvus: thanks | 18:51 |
*** markvoelker has quit IRC | 18:55 | |
clarkb | fungi: did you want to remove storyboard from the emergency hosts file now that ^ is in? | 18:56 |
clarkb | fungi: I guess we'll see if it works on -dev first? | 18:56 |
*** yolanda has joined #openstack-infra | 18:57 | |
clarkb | I guess we also need a new storyboard client tarball to check it as well | 18:57 |
clarkb | can we get one of those simply by merging a commit or is it smarter than that? | 18:57 |
zxiiro | corvus: alright, looks like we got the email in RT. and I've also went ahead and resolved it. openstack-zuul should have stream-events now. | 18:57 |
clarkb | Gerrit 2.15 GC feature results in file descriptor leakages... | 18:58 |
*** yolanda_ has quit IRC | 18:59 | |
SotK | clarkb: merging a webclient commit will cause a new tarball | 19:00 |
clarkb | SotK: good to know thanks. We probably want to have that happen once we are ready to watch -dev then maybe against after we remove prod from the exclude list | 19:01 |
SotK | that should be easy enough, we have a few things waiting to merge I think | 19:02 |
*** felipemonteiro has joined #openstack-infra | 19:04 | |
*** janki has quit IRC | 19:08 | |
mordred | SotK: I just rechecked a couple of the patches in the project-name-in-url series so we could see previews | 19:18 |
fungi | clarkb: yeah, in theory we could merge a new sb-wc change to trigger deployment to sb-dev.o.o and see how it goes | 19:19 |
fungi | though also possible deleting the old tarball will work | 19:19 |
mordred | it should do ... the wget command will download it if tere isn't a local file that matches, and that shouldkick the remaining chain | 19:22 |
clarkb | ok I'll remove the existing tarball on -dev | 19:22 |
clarkb | moved it aside | 19:23 |
clarkb | next puppet run starting in 5 minutes should test everything out | 19:25 |
*** felipemonteiro has quit IRC | 19:26 | |
*** felipemonteiro has joined #openstack-infra | 19:27 | |
*** gfidente|afk has quit IRC | 19:31 | |
openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Add the ability for a user to skip macros https://review.openstack.org/571287 | 19:31 |
*** lifeless has joined #openstack-infra | 19:31 | |
*** nicolasbock has quit IRC | 19:33 | |
*** ssbarnea_ has joined #openstack-infra | 19:33 | |
*** wolverineav has quit IRC | 19:35 | |
*** wolverineav has joined #openstack-infra | 19:35 | |
*** wolverineav has quit IRC | 19:40 | |
*** signed8bit is now known as signed8bit_Zzz | 19:40 | |
*** signed8bit_Zzz is now known as signed8bit | 19:43 | |
*** markvoelker has joined #openstack-infra | 19:45 | |
*** slaweq_ has joined #openstack-infra | 19:47 | |
*** slaweq has quit IRC | 19:48 | |
*** nicolasbock has joined #openstack-infra | 19:48 | |
*** eharney has quit IRC | 19:49 | |
clarkb | storyboard-dev.openstack.org appears to have done the correct thing. I think we want it to have another puppet run to be sure that the next time around it won't break everything | 19:49 |
clarkb | but if that ends up being happy I think we can remove storyboard.openstack.org from the emergency file and manually remove its content tarball to force it to rerun there as well | 19:50 |
clarkb | fungi: ^ | 19:50 |
clarkb | I'm going to find lunch now | 19:51 |
fungi | great! | 19:51 |
*** kjackal has quit IRC | 19:53 | |
*** markvoelker has quit IRC | 19:54 | |
*** kien-ha has quit IRC | 19:59 | |
*** efried has quit IRC | 20:03 | |
*** efried has joined #openstack-infra | 20:04 | |
efried | clarkb: That seems to have worked. Thanks! | 20:11 |
*** wolverineav has joined #openstack-infra | 20:13 | |
*** roman_g has quit IRC | 20:14 | |
*** armaan has joined #openstack-infra | 20:15 | |
*** wolverin_ has joined #openstack-infra | 20:16 | |
*** wolverineav has quit IRC | 20:16 | |
*** felipemonteiro has quit IRC | 20:20 | |
clarkb | fungi: looks like -dev ran puppet again and nooped | 20:25 |
clarkb | and my browers is still happy with it after a hard refresh | 20:25 |
*** trown is now known as trown|outtypewww | 20:25 | |
*** markvoelker has joined #openstack-infra | 20:25 | |
clarkb | did you want to go ahead and remove the prod server from the emergency file? I am aroudn to help watch it though probably not primary on that as I need to start writing emails | 20:26 |
clarkb | summit is basically an email generator | 20:26 |
*** florianf has quit IRC | 20:34 | |
mnaser | corvus: time to bikeshed but instead of 'winterscale' we could have gone for 'snowtrain' as a reminder of our past 2 ptgs :P | 20:38 |
mnaser | but we do want to keep it seperate, but, i couldn't keep that comment to myself :) | 20:39 |
*** rwsu has joined #openstack-infra | 20:43 | |
corvus | mnaser: i love it. bikeshedding over the codename used to talk about the thing about whose name we're bikeshedding. :) | 20:43 |
mnaser | :D | 20:44 |
mnaser | overall though i'm very +1 with the proposal, it's about time "openstack-infra" becomes its thing | 20:44 |
*** kgiusti has left #openstack-infra | 20:44 | |
mnaser | things like zuul multitenancy and the state we're in means its the perfect time to do that | 20:45 |
*** felipemonteiro has joined #openstack-infra | 20:46 | |
*** hasharAway has quit IRC | 20:51 | |
fungi | clarkb: yep, i'm unleashing the ansibles on sb.o.o again now | 20:52 |
clarkb | cool | 20:52 |
fungi | and done. thanks! | 20:53 |
fungi | #status log storyboard.openstack.org has been removed from the emergency disable list now that storyboard-webclient tarball deployment is fixed | 20:54 |
openstackstatus | fungi: finished logging | 20:54 |
clarkb | fungi: are you going to remove the old tarball too? | 20:54 |
fungi | i think we've merged at least one sb-wc change since puppet was disabled. checking | 20:58 |
fungi | nope, i'm wrong. current deployment seems to be tip of master already | 20:59 |
fungi | so yeah i'll get the old tarball out of the way to make sure redeployment is exercised properly | 20:59 |
fungi | and watching syslog to see when it fires | 21:00 |
*** salv-orlando has joined #openstack-infra | 21:00 | |
*** salv-orlando has quit IRC | 21:01 | |
*** salv-orlando has joined #openstack-infra | 21:01 | |
*** neiloy has quit IRC | 21:06 | |
fungi | looks like it's starting | 21:10 |
*** salv-orlando has quit IRC | 21:13 | |
*** markvoelker has quit IRC | 21:14 | |
*** ssbarnea_ has quit IRC | 21:14 | |
*** sree has joined #openstack-infra | 21:14 | |
*** markvoelker has joined #openstack-infra | 21:14 | |
*** ldnunes has quit IRC | 21:14 | |
*** VW_ has joined #openstack-infra | 21:16 | |
*** markvoelker has quit IRC | 21:18 | |
fungi | seems to have redeployed cleanly, and the webui is still working | 21:19 |
*** sree has quit IRC | 21:19 | |
*** VW has quit IRC | 21:20 | |
*** VW_ has quit IRC | 21:21 | |
*** lifeless has quit IRC | 21:26 | |
*** lifeless_ has joined #openstack-infra | 21:26 | |
ianw | clarkb / fungi : can you read over https://review.openstack.org/#/c/571046/ quickly which adds some notes from our reprepo discussion the other day | 21:27 |
fungi | sure | 21:28 |
ianw | i'm also working on some notes for afs debugging from what i learnt during recent outage | 21:28 |
ianw | and i also found some cool afs contrib utils, and one of them looks like a better basis for getting some stats into statsd and and a grafana dashboard than what i'd previously started | 21:28 |
openstackgerrit | Merged openstack-infra/system-config master: openstack_project: opensuse: Fix OBS repository url https://review.openstack.org/566924 | 21:30 |
*** lifeless has joined #openstack-infra | 21:38 | |
*** lifeless_ has quit IRC | 21:38 | |
*** armaan has quit IRC | 21:39 | |
*** jcoufal_ has quit IRC | 21:41 | |
*** esberglu has quit IRC | 21:43 | |
clarkb | fungi: yay re storyboard. I'm going to read ianw's change now then context switch over to testing the new ansible inventory stuff | 21:48 |
ianw | clarkb: i didn't follow too much of that, but i'm around if i can help on that | 21:49 |
clarkb | ianw: I think we've mostly worked through the problems. Will let you know if new ones pop up | 21:50 |
*** harlowja has joined #openstack-infra | 21:52 | |
fungi | ianw: a few nits/suggestions marked inline, but all in all i'm fine with it as-is | 21:55 |
*** caphrim007_ has joined #openstack-infra | 21:55 | |
*** caphrim007 has quit IRC | 21:59 | |
clarkb | ianw: fungi I've +2'd but not approved. Will let ianw decide if the comments are worth addressing prior to approval (otherwise approving si fine by me) | 21:59 |
openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add some rationale for reprepro https://review.openstack.org/571046 | 22:02 |
ianw | thanks, added some updates. | 22:03 |
ianw | clarkb: hmm, so is the ansible inventory stuff preventing updates on mirror-update? i'd expected hworang's suse update fix to be deployed by now | 22:03 |
clarkb | ok openstack_inventory.py seems to work | 22:03 |
*** lifeless has quit IRC | 22:03 | |
clarkb | ianw: we pinned to old ansible earlier today (maybe 6 hours ago?) which got it running again | 22:03 |
clarkb | ianw: but yes prior to that it would've been possible | 22:04 |
*** lifeless has joined #openstack-infra | 22:04 | |
ianw | hmm, ok | 22:05 |
clarkb | ianw: I just tseted that the new thing would work out of band (I think it will) so will unwip 571263 | 22:05 |
ianw | last run was May 30 21:37:18, maybe it missed and i'm just impatient | 22:06 |
openstackgerrit | Clark Boylan proposed openstack-infra/system-config master: Switch to using new ansible openstack inventory script https://review.openstack.org/571263 | 22:07 |
clarkb | it had a bug I think ^ should be happy now and I tested that the usage of that inventory file works on puppetmaster | 22:07 |
clarkb | mordred: ^ do you want to review that? | 22:07 |
*** kien-ha has joined #openstack-infra | 22:08 | |
clarkb | 15 > 42 == True is my new favorite fact :) | 22:15 |
clarkb | #status log storyboard is deploying latest webclient again after fixing the deployment process around the webclient. | 22:17 |
openstackstatus | clarkb: finished logging | 22:17 |
*** rcernin has joined #openstack-infra | 22:22 | |
*** dave-mccowan has quit IRC | 22:23 | |
*** rlandy|rover is now known as rlandy|rover|bbl | 22:23 | |
*** slaweq_ has quit IRC | 22:30 | |
*** slaweq has joined #openstack-infra | 22:30 | |
ianw | the remote rsync failed on the first attempt at the suse update, running it again | 22:31 |
*** salv-orlando has joined #openstack-infra | 22:32 | |
*** lifeless has quit IRC | 22:33 | |
*** slaweq has quit IRC | 22:34 | |
*** lifeless has joined #openstack-infra | 22:35 | |
*** salv-orlando has quit IRC | 22:37 | |
*** hongbin has quit IRC | 22:38 | |
*** jesslampe has joined #openstack-infra | 22:40 | |
*** rossella_s has quit IRC | 22:41 | |
*** rossella_s has joined #openstack-infra | 22:42 | |
*** r-daneel has quit IRC | 22:42 | |
openstackgerrit | Adam Spiers proposed openstack-infra/irc-meetings master: Remove weekly HA meeting https://review.openstack.org/571326 | 22:44 |
*** wolverin_ has quit IRC | 22:45 | |
*** wolverineav has joined #openstack-infra | 22:45 | |
openstackgerrit | Adam Spiers proposed openstack-infra/irc-meetings master: Remove weekly HA meeting https://review.openstack.org/571326 | 22:46 |
*** nicolasbock has quit IRC | 22:47 | |
openstackgerrit | Merged openstack-infra/system-config master: Add some rationale for reprepro https://review.openstack.org/571046 | 22:48 |
*** mriedem is now known as mriedem_afk | 22:48 | |
*** wolverineav has quit IRC | 22:50 | |
*** rossella_s has quit IRC | 22:51 | |
*** edmondsw has quit IRC | 22:52 | |
*** edmondsw has joined #openstack-infra | 22:53 | |
*** signed8bit is now known as signed8bit_Zzz | 22:53 | |
*** rossella_s has joined #openstack-infra | 22:54 | |
*** edmondsw has quit IRC | 22:57 | |
*** tosky has quit IRC | 23:04 | |
*** dpawlik has joined #openstack-infra | 23:10 | |
*** lifeless has quit IRC | 23:11 | |
*** jesslampe has quit IRC | 23:13 | |
*** jesslampe has joined #openstack-infra | 23:13 | |
tonyb | Can I be added to bootstrappers group for ~10mins to EOL(newton) Kolla and OSA as per http://lists.openstack.org/pipermail/openstack-dev/2018-April/129366.html and http://lists.openstack.org/pipermail/openstack-dev/2018-April/129981.html ? | 23:14 |
*** dpawlik has quit IRC | 23:14 | |
openstackgerrit | Amy Marrich (spotz) proposed openstack-infra/system-config master: Update of codeblock for adding a new list https://review.openstack.org/571329 | 23:14 |
clarkb | tonyb: sure | 23:14 |
tonyb | clarkb: Thanks | 23:15 |
clarkb | tonyb: you are in there now. standard disclaimer and all that | 23:15 |
openstackgerrit | Merged openstack-infra/irc-meetings master: Remove weekly HA meeting https://review.openstack.org/571326 | 23:15 |
tonyb | clarkb: Awesome | 23:15 |
spotz | tonyb You be nice to OSA Newton, it was a good friend:) | 23:16 |
tonyb | spotz: I'm only doing as evrardjp asked ;P but of course I'll do it as kindly as I can :) | 23:16 |
spotz | tonyb hehe | 23:17 |
clarkb | corvus: we both +2'd spotz change at the same time :P I'll approve it | 23:17 |
clarkb | bah we both approved too | 23:17 |
corvus | clarkb: if you do we'll both have approved it :) | 23:17 |
corvus | ha | 23:17 |
spotz | clarkb corvus - thanks! there will be a patch in the next frw days for a new list for the mentoring program:) Wanted to make sure the docs were right as it'll be Ell's first patch:) | 23:18 |
*** wolverineav has joined #openstack-infra | 23:18 | |
clarkb | spotz: thank you for the fix | 23:18 |
spotz | And I'll try to do some reviews myself after I hit the OSA reviews | 23:18 |
*** felipemonteiro has quit IRC | 23:18 | |
spotz | And fix the cinder install guide:) | 23:19 |
*** felipemonteiro has joined #openstack-infra | 23:25 | |
*** markvoelker has joined #openstack-infra | 23:25 | |
*** salv-orlando has joined #openstack-infra | 23:33 | |
*** salv-orlando has quit IRC | 23:34 | |
*** salv-orlando has joined #openstack-infra | 23:34 | |
clarkb | corvus: zuul has been running for almost 3 weeks without a restart and is using less than 3GB resident memory for the scheduler and server total is under 5GB :) | 23:36 |
corvus | \o/ | 23:37 |
clarkb | thinking out loud here we should probably consider reducing the size of that server. Also switching to our zookeeper cluster | 23:39 |
*** salv-orlando has quit IRC | 23:40 | |
clarkb | not urgent but would be nice to free up some resources. | 23:40 |
clarkb | similarly I think we may need to delete the old trusty gerrit server? | 23:40 |
corvus | yep, and i think we have enough data for our baseline now | 23:40 |
* clarkb scribbles a note to followup on the gerrit thing soon | 23:40 | |
*** lifeless has joined #openstack-infra | 23:41 | |
clarkb | a 16GB instance should be plenty big and accomodate the cache being used | 23:41 |
ianw | rsync: change_dir "/opensuse/repositories/Cloud:/OpenStack:/Pike/openSUSE_Leap_15.0" (in pub) failed: No such file or directory (2) | 23:42 |
ianw | clarkb: ^ any particular a priori insight into that? | 23:42 |
clarkb | no, sorry | 23:42 |
ianw | OBS_REPOS=('Virtualization:/containers' 'Cloud:/OpenStack:/Pike' 'Cloud:/OpenStack:/Queens') ... i guess it is assumed all of those are valid for all suse releases | 23:43 |
tonyb | clarkb: Thanks I did the OSA ones but not kolla, and I've removed myself from that group | 23:43 |
*** markvoelker has quit IRC | 23:43 | |
clarkb | ianw: that is probably a good guess, UCA has similar "issues" iirc | 23:43 |
clarkb | tonyb: thanks | 23:43 |
*** markvoelker has joined #openstack-infra | 23:44 | |
clarkb | ianw: http://download.opensuse.org/repositories/Cloud:/OpenStack:/ browesable repos | 23:44 |
clarkb | and yup looks like leap 15 starts with queens so pike is too old | 23:44 |
ianw | hmm, not sure why the mirror has "Staging" "http://download.opensuse.org/repositories/Cloud:/OpenStack:/Queens:/Staging/" | 23:47 |
tonyb | clarkb: Actually can you add me back again, I just worked out what I was doign wrong with kolla | 23:47 |
*** caphrim007_ has quit IRC | 23:47 | |
clarkb | tonyb: sure | 23:47 |
clarkb | ianw: http://download.opensuse.org/repositories/Cloud:/OpenStack:/Queens/ : vs without : | 23:48 |
clarkb | tonyb: done | 23:48 |
*** markvoelker has quit IRC | 23:48 | |
openstackgerrit | Merged openstack-infra/system-config master: Update of codeblock for adding a new list https://review.openstack.org/571329 | 23:50 |
tonyb | clarkb: Thanks, kolla has been tagged all done (for real this time) | 23:54 |
*** rpioso is now known as rpioso|afk | 23:55 | |
*** lifeless has quit IRC | 23:58 | |
*** eharney has joined #openstack-infra | 23:58 | |
*** lifeless has joined #openstack-infra | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!