| clarkb | EmilienM: ok left some comments | 00:04 |
|---|---|---|
| EmilienM | clarkb: thx, I'll take a look | 00:07 |
| clarkb | corvus: ok ze04 is running an executor again I think the init script is working | 00:13 |
| *** xarses_ has joined #openstack-infra | 00:16 | |
| mnaser | fyi | 00:18 |
| mnaser | https://git.centos.org/commitdiff/rpms!kernel.git/6202b1a49906b21f0cdce08b4ff853a65c658750 | 00:18 |
| *** sdague has quit IRC | 00:18 | |
| mriedem | dmsimard: another thing with the logs stuff is that you now have to download the config zips http://logs.openstack.org/67/529867/1/check/tempest-full/23d2919/controller/logs/etc/nova/ | 00:19 |
| mriedem | assuming because devstack-gate's post run code isn't expecting the new directory structure | 00:20 |
| clarkb | mnaser: good to know, are those rpms published yet? | 00:20 |
| mnaser | clarkb: commit was only 90 minutes ago | 00:20 |
| mnaser | i pinged #centos-devel to ask, but im not sure how the internals of the centos builds work | 00:20 |
| clarkb | mnaser: sure but often times all that is staged for once embargo is lifted right? in any case I think our biggest risk is not actually on our centos machines | 00:21 |
| clarkb | its the trusty hypervisors in infra cloud | 00:21 |
| mnaser | well it looks like the embargo was broken | 00:21 |
| clarkb | ya it was | 00:21 |
| mnaser | and a poc is out unfortunately | 00:21 |
| mnaser | im not sure about ubuntu kernels and what their timelines are :( | 00:22 |
| clarkb | we shouldn't run arbitrary code anywhere but single use VMs and transitively on our hypervisors for infra cloud | 00:22 |
| clarkb | mnaser: ya no usn for it yet that I see | 00:22 |
| mnaser | i mean the centos stuff wasnt even announced | 00:23 |
| mnaser | i was just digging around | 00:23 |
| clarkb | that and our laptops | 00:23 |
| clarkb | or wherever you run javascripts and the like | 00:23 |
| clarkb | (though I don't think there is a poc via that vector (yet) | 00:24 |
| *** daidv has quit IRC | 00:24 | |
| mnaser | http://kernel.ubuntu.com/git/ubuntu/ubuntu-trusty.git/log/ | 00:25 |
| mnaser | nothing there unfortunately | 00:25 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/system-config master: Add Puppetlabs mirror for Ubuntu Xenial https://review.openstack.org/531030 | 00:26 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient master: Remove bower and add yarn support https://review.openstack.org/528436 | 00:27 |
| mnaser | https://www.redhat.com/archives/rhsa-announce/2018-January/thread.html | 00:30 |
| mnaser | i guess canonical should announce it anytime now | 00:30 |
| *** claudiub has quit IRC | 00:41 | |
| *** caphrim007 has quit IRC | 00:44 | |
| openstackgerrit | Kendall Nelson proposed openstack-infra/storyboard master: Remove Update tox Step https://review.openstack.org/530427 | 00:46 |
| *** xarses_ has quit IRC | 00:47 | |
| *** gothicmindfood has quit IRC | 00:47 | |
| *** ijw has quit IRC | 00:50 | |
| armax | hi folks, how long does it take for change https://review.openstack.org/#/c/530915/ to become effective? I rechecked on https://review.openstack.org/#/c/530857/ but after over 6 hours the job still seems to run on stable branches, just idle curiosity | 00:50 |
| armax | thanks | 00:50 |
| clarkb | armax: the job being tempest-full? | 00:51 |
| armax | yes | 00:51 |
| armax | clarkb: hi and happy new year! | 00:51 |
| clarkb | armax: hello! | 00:52 |
| clarkb | armax: changes like that should apply immediately when merged as zuul is aware it merged and can apply it immediately | 00:52 |
| clarkb | armax: my guess is something else is making it not work, will look at it | 00:52 |
| armax | I still see it on http://zuulv3.openstack.org/ and change 530857 | 00:52 |
| armax | also, I don’t seem to see the tempest.conf/log files being collected anymore I was tracing andreaf’s recent changes around log collection | 00:53 |
| armax | but couldn’t spot anything obvious | 00:53 |
| clarkb | the regex seems to work as expected (just testing in python interpreter) | 00:55 |
| armax | clarkb: yeah, I was looking at that | 00:55 |
| clarkb | my next guess is that some other variant is applying and overriding the job selection | 00:58 |
| clarkb | I think it logs that stuff for us /me looks | 00:59 |
| *** cuongnv has joined #openstack-infra | 00:59 | |
| armax | clarkb: right, but looking at http://codesearch.openstack.org/?q=tempest-full&i=nope&files=&repos= I see no smoking gun | 01:00 |
| armax | assuming I’d be looking in the right place | 01:00 |
| clarkb | armax: http://logs.openstack.org/57/530857/1/check/tempest-full/6a17ce3/zuul-info/inventory.yaml has the path under inheritance_path | 01:00 |
| armax | ah, that’s good to know | 01:00 |
| clarkb | armax: the top of the list is the base job and then the bottom is the last config for the job | 01:00 |
| armax | pretty cool | 01:01 |
| armax | oh | 01:01 |
| armax | tempest being branchless maybe that’s the source of our problem | 01:01 |
| clarkb | armax: I think I see it | 01:02 |
| clarkb | armax: project-config/projects.yaml line 10348 | 01:02 |
| clarkb | armax: that defines tempest-full there with irrelevant files but no branch matchers | 01:02 |
| clarkb | I think if you add the branch matchers to that too it will do what you expect | 01:02 |
| armax | oh | 01:03 |
| armax | I can give it a go | 01:03 |
| armax | though, I wasn’t expecting it to behave like that | 01:03 |
| armax | patch coming right up | 01:05 |
| armax | I guess I’d have to do the same for the gate part | 01:14 |
| clarkb | ya anywhere you are making a new variant will need it I Think | 01:14 |
| armax | aye | 01:14 |
| clarkb | since this isn't inheritance its a different definition | 01:14 |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config master: Make sure we only run tempest-full on master https://review.openstack.org/531045 | 01:14 |
| *** bandini has quit IRC | 01:14 | |
| *** fanzhang has left #openstack-infra | 01:15 | |
| *** lastmikoi has quit IRC | 01:17 | |
| *** mpjetta has quit IRC | 01:17 | |
| *** Swami has quit IRC | 01:17 | |
| *** bandini has joined #openstack-infra | 01:17 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/system-config master: Add Puppetlabs mirror for Ubuntu Xenial https://review.openstack.org/531030 | 01:19 |
| *** mpjetta has joined #openstack-infra | 01:19 | |
| armax | clarkb: I hope I got it right, thanks for the tip | 01:20 |
| *** aviau has joined #openstack-infra | 01:21 | |
| *** lastmikoi has joined #openstack-infra | 01:23 | |
| corvus | remote: https://review.openstack.org/531046 Only run tempest-full on the master branch | 01:27 |
| corvus | armax, clarkb, andreaf: ^ that may be a simpler alternative | 01:27 |
| armax | corvus: indeed | 01:28 |
| corvus | since the current issue is fundamental to the job, if we can keep the fix as close to that as possible, it should be easier to unwind later | 01:28 |
| corvus | armax: can you make a change Depends-On: the one i just pushed up to verify that fixes it for yoU? | 01:28 |
| armax | corvus: sure | 01:29 |
| armax | corvus: looks like that did the trick | 01:32 |
| armax | I don’t see the job running for 531048 | 01:33 |
| armax | corvus: does this mean that https://review.openstack.org/#/c/530915/1/zuul.d/zuul-legacy-project-templates.yaml could be partially reverted? | 01:34 |
| armax | at least for the parts that touch the branches for tempest-full | 01:34 |
| corvus | armax: yes i think so | 01:49 |
| armax | corvus: OK, let me see if I can earn some brownie points :) | 01:49 |
| *** smatzek has joined #openstack-infra | 01:53 | |
| *** threestrands has joined #openstack-infra | 01:55 | |
| *** smatzek has quit IRC | 01:57 | |
| *** kaisers has quit IRC | 01:58 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config master: Don't run legacy-tempest-dsvm-neutron-full on non-code nova changes https://review.openstack.org/531052 | 02:10 |
| *** smcginnis has quit IRC | 02:13 | |
| *** zhurong has joined #openstack-infra | 02:14 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: license: remove dos line break https://review.openstack.org/531057 | 02:17 |
| mnaser | Do we wanna drop +r? | 02:18 |
| corvus | mnaser: ++ | 02:19 |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/openstack-zuul-jobs master: Restore branch conditional on tempest-full https://review.openstack.org/531058 | 02:19 |
| armax | corvus: ^ | 02:20 |
| *** rlandy|bbl is now known as rlandy | 02:20 | |
| openstackgerrit | shangxdy proposed openstack-infra/gear master: Modify connection timeout process https://review.openstack.org/531059 | 02:22 |
| *** rlandy has quit IRC | 02:23 | |
| *** ijw has joined #openstack-infra | 02:26 | |
| *** rfolco has quit IRC | 02:29 | |
| *** ijw has quit IRC | 02:30 | |
| *** bobh has joined #openstack-infra | 02:36 | |
| *** namnh has joined #openstack-infra | 02:47 | |
| *** bobh has quit IRC | 02:53 | |
| *** RuiChen has joined #openstack-infra | 03:01 | |
| *** kiennt26 has joined #openstack-infra | 03:10 | |
| *** ijw has joined #openstack-infra | 03:14 | |
| *** coolsvap has joined #openstack-infra | 03:26 | |
| yamamoto | is there an equivalent of stats.timers.nodepool.job.xxx these days? | 03:31 |
| *** mriedem has quit IRC | 03:48 | |
| *** xarses_ has joined #openstack-infra | 04:08 | |
| openstackgerrit | Merged openstack-dev/pbr master: Deprecate testr and nose integration https://review.openstack.org/518224 | 04:09 |
| *** lbragstad has quit IRC | 04:11 | |
| *** gyee has quit IRC | 04:29 | |
| *** armax has quit IRC | 04:32 | |
| *** armax has joined #openstack-infra | 04:33 | |
| *** armax has quit IRC | 04:33 | |
| *** armax has joined #openstack-infra | 04:33 | |
| *** armax has quit IRC | 04:34 | |
| *** armax has joined #openstack-infra | 04:34 | |
| *** armax has quit IRC | 04:34 | |
| *** armax has joined #openstack-infra | 04:35 | |
| *** armax has quit IRC | 04:35 | |
| *** udesale has joined #openstack-infra | 04:37 | |
| *** nicolasbock has quit IRC | 04:53 | |
| *** ramishra has joined #openstack-infra | 05:03 | |
| *** yangzhenyu has joined #openstack-infra | 05:08 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove devstack-gate specific tox jobs https://review.openstack.org/528554 | 05:10 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Only copy openafs centos rpms if we built them https://review.openstack.org/531018 | 05:13 |
| *** ijw has quit IRC | 05:13 | |
| *** ramishra has quit IRC | 05:20 | |
| *** pots has quit IRC | 05:21 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove converted devstack-gate tox jobs https://review.openstack.org/528555 | 05:26 |
| *** ramishra has joined #openstack-infra | 05:39 | |
| *** janki has joined #openstack-infra | 05:43 | |
| *** eumel8 has joined #openstack-infra | 05:50 | |
| *** markmcclain has quit IRC | 05:51 | |
| *** markmcclain has joined #openstack-infra | 05:54 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config master: grafana: Update networking-midonet https://review.openstack.org/530220 | 05:57 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/531078 | 06:06 |
| *** swest has joined #openstack-infra | 06:14 | |
| *** armaan has quit IRC | 06:21 | |
| *** hongbin has joined #openstack-infra | 06:22 | |
| *** hongbin has quit IRC | 06:22 | |
| *** dbecker has quit IRC | 06:29 | |
| openstackgerrit | Merged openstack-infra/project-config master: Normalize projects.yaml https://review.openstack.org/531078 | 06:34 |
| *** 07EAAP3JW has joined #openstack-infra | 06:43 | |
| *** 07EAAP3JW has quit IRC | 06:43 | |
| *** hongbin has joined #openstack-infra | 06:43 | |
| *** zhurong has quit IRC | 06:47 | |
| *** zhurong has joined #openstack-infra | 06:52 | |
| *** jaosorior has quit IRC | 07:01 | |
| *** ramishra has quit IRC | 07:10 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Strip \r from build UUID in fingergw https://review.openstack.org/530973 | 07:10 |
| *** jaosorior has joined #openstack-infra | 07:11 | |
| *** ramishra has joined #openstack-infra | 07:12 | |
| *** jaosorior has quit IRC | 07:13 | |
| *** jaosorior has joined #openstack-infra | 07:14 | |
| *** jaosorior has quit IRC | 07:14 | |
| *** jaosorior has joined #openstack-infra | 07:14 | |
| *** sbezverk has quit IRC | 07:21 | |
| *** benj_ has quit IRC | 07:22 | |
| *** threestrands has quit IRC | 07:25 | |
| *** annp has joined #openstack-infra | 07:26 | |
| *** jrist has quit IRC | 07:29 | |
| *** pcaruana has joined #openstack-infra | 07:32 | |
| *** AJaeger has quit IRC | 07:44 | |
| *** AJaeger has joined #openstack-infra | 07:47 | |
| *** jaypipes has quit IRC | 07:53 | |
| *** markvoelker has quit IRC | 07:55 | |
| *** rcernin has quit IRC | 08:04 | |
| *** shardy has joined #openstack-infra | 08:16 | |
| *** sshnaidm|afk is now known as sshnaidm | 08:28 | |
| *** liusheng has quit IRC | 08:30 | |
| *** ramishra has quit IRC | 08:38 | |
| *** dtantsur|afk is now known as dtantsur | 08:49 | |
| *** lucas-afk is now known as lucasagomes | 08:49 | |
| *** mnencia has quit IRC | 08:56 | |
| *** mnencia has joined #openstack-infra | 08:56 | |
| *** jpich has joined #openstack-infra | 09:02 | |
| *** larainema has quit IRC | 09:03 | |
| *** hongbin has quit IRC | 09:03 | |
| *** e0ne has joined #openstack-infra | 09:05 | |
| *** jrist has joined #openstack-infra | 09:13 | |
| *** mnencia has quit IRC | 09:19 | |
| *** mnencia has joined #openstack-infra | 09:20 | |
| *** derekh has joined #openstack-infra | 09:24 | |
| stephenfin | Morning o/ Would anyone be able to advise me on what's happening here with the openstack-tox-functional + privsep test? https://review.openstack.org/#/c/530961/ | 09:33 |
| frickler | stephenfin: iiuc sudo access is removed for tox tests, so starting the privsep daemon fails here http://logs.openstack.org/61/530961/1/check/openstack-tox-functional/b033cc9/job-output.txt.gz#_2018-01-03_16_51_01_314525 | 09:36 |
| stephenfin | frickler: Ah, it's a tox thing | 09:37 |
| * stephenfin initially thought permissions issues, but then realized you'd need sudo for DevStack stuff | 09:37 | |
| *** tosky has joined #openstack-infra | 09:38 | |
| AJaeger | stephenfin: mordred has been working on some tox tests with sudo enabled - best discuss with him the status there and whether this would help your case | 09:38 |
| stephenfin | AJaeger: Sounds like a good call. Yet more work for mordred 🎉 | 09:41 |
| *** kiennt26 has quit IRC | 09:49 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove integrated-gate-nova-net https://review.openstack.org/529858 | 09:55 |
| *** markvoelker has joined #openstack-infra | 09:56 | |
| *** greghaynes has quit IRC | 09:58 | |
| *** sshnaidm is now known as sshnaidm|afk | 10:00 | |
| *** erlon has joined #openstack-infra | 10:00 | |
| *** cuongnv has quit IRC | 10:06 | |
| *** pbourke has quit IRC | 10:10 | |
| *** annp has quit IRC | 10:12 | |
| *** toabctl has quit IRC | 10:16 | |
| *** ijw has joined #openstack-infra | 10:17 | |
| *** ijw has quit IRC | 10:21 | |
| *** toabctl has joined #openstack-infra | 10:23 | |
| danpawlik | does soneome know what is the new kernel for Ubuntu xenial, trusty with a patch for Meltdown and Spectre? | 10:26 |
| *** namnh has quit IRC | 10:27 | |
| *** yangzhenyu has quit IRC | 10:30 | |
| *** markvoelker has quit IRC | 10:30 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy-irc-meetings-tox-ical https://review.openstack.org/528501 | 10:31 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy swift tox jobs https://review.openstack.org/528650 | 10:31 |
| frickler | danpawlik: still pending, see https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SpectreAndMeltdown | 10:46 |
| openstackgerrit | Jens Harbott (frickler) proposed openstack-infra/openstack-zuul-jobs master: Remove obsolete interop jobs https://review.openstack.org/528645 | 10:48 |
| chandankumar | AJaeger: Hello | 10:49 |
| chandankumar | AJaeger: Please point me an example for common template for writting python-tempest-plugin jobs | 10:50 |
| AJaeger | chandankumar: check openstack-zuul-jobs/zuul.d/project-templates.yaml | 10:50 |
| chandankumar | AJaeger: sure! | 10:50 |
| *** markmcclain has quit IRC | 10:51 | |
| chandankumar | AJaeger: i think we can merge this one https://review.openstack.org/#/c/528674/ then i will create the common template and update in one go, what you say? | 10:51 |
| AJaeger | frickler: Remove legacy-irc-meetings-tox-ical was jsut merged - I'll update the introop change | 10:51 |
| AJaeger | chandankumar: works for me | 10:52 |
| frickler | AJaeger: ah, you are right, that one needs to be removed, too, sorry for mixing that up | 10:52 |
| chandankumar | frickler: needs one +2 and +w https://review.openstack.org/#/c/528674/ | 10:52 |
| chandankumar | AJaeger: frickler Thanks :-) | 10:53 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove obsolete interop jobs https://review.openstack.org/528645 | 10:53 |
| AJaeger | frickler: thanks for trying to fix it ;) new version ^ | 10:53 |
| AJaeger | frickler: because of those merge conflicts, I'd like to get those changes in and through our backlog. Thanks for helping to reduce it. Care to review https://review.openstack.org/#/c/530240/ as well, please? | 10:54 |
| *** larainema has joined #openstack-infra | 10:57 | |
| *** markmcclain has joined #openstack-infra | 10:59 | |
| AJaeger | we should ask asettle to add pkovar ... | 11:00 |
| AJaeger | stephenfin: see https://docs.openstack.org/infra/system-config/irc.html#basic-channel-operator-commands | 11:01 |
| AJaeger | wrong channel ;( | 11:01 |
| frickler | uh oh, I think I approved stuff too fast, now zuul starts swapping ... :-( | 11:01 |
| frickler | hope it will get itself together in a bit | 11:01 |
| openstackgerrit | Sam Betts proposed openstack-infra/devstack-gate master: Move location of devstack-gate unstack to post-tempest https://review.openstack.org/530726 | 11:01 |
| frickler | infra-root: zuulv3.o.o web service isn't responding for me and http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=63979&rra_id=all looks like memory usage exploded. I'll try to take a look at the logs on the node | 11:15 |
| openstackgerrit | Adam Spiers proposed openstack-infra/infra-manual master: promote IRC channel creation info into its own section https://review.openstack.org/531133 | 11:16 |
| *** nicolasbock has joined #openstack-infra | 11:17 | |
| frickler | seems this is the patch that got merged last and caused memory usage to increase. not sure whether there is anything specific to that patch, or just that the n-th reconfiguration was too much for zuul https://review.openstack.org/513418 | 11:24 |
| frickler | #status notice zuul seems to have gotten stuck and will probably need a restart, please be patient | 11:26 |
| openstackstatus | frickler: sending notice | 11:26 |
| *** kopecmartin has joined #openstack-infra | 11:26 | |
| -openstackstatus- NOTICE: zuul seems to have gotten stuck and will probably need a restart, please be patient | 11:27 | |
| *** markvoelker has joined #openstack-infra | 11:27 | |
| frickler | infra-root: /me will go and have lunch and give zuul a bit of time to recover, planning to restart in half an hour unless the situation improves | 11:27 |
| *** openstackstatus has quit IRC | 11:28 | |
| *** openstack has quit IRC | 11:28 | |
| *** openstack has joined #openstack-infra | 13:08 | |
| *** ChanServ sets mode: +o openstack | 13:08 | |
| mnaser | bot is late for work today at 8:08 am :> | 13:08 |
| mordred | frickler: fwiw, if you're talking with folks in #freenode, corvus put together a plan yesterday to get us to a better consolidated bot situation | 13:09 |
| AJaeger | mordred: apparently not anymore, see 11:28 the killed by Sigyn ;( | 13:09 |
| *** openstackstatus has joined #openstack-infra | 13:09 | |
| *** ChanServ sets mode: +v openstackstatus | 13:09 | |
| mordred | AJaeger: yah - it's possible it's just the openstack bot that's on the whitelist and not statusbot | 13:09 |
| mordred | \o/ | 13:10 |
| * mordred waves to openstackstatus | 13:10 | |
| openstackgerrit | Dmitry Tantsur proposed openstack-infra/project-config master: Update jobs for ironic-tempest-plugin https://review.openstack.org/529528 | 13:11 |
| dtantsur | AJaeger: removed controversial bit ^^^ | 13:11 |
| AJaeger | dtantsur: not controversial - I just don't know yet how to do it ;( And nobody digged into it yet... | 13:12 |
| * frickler is talking to sigyn admin now, do we have a list of all our bots somewhere? | 13:12 | |
| dtantsur | AJaeger: somebody will have to - it's blocking creating of new projects that publish to pypi | 13:13 |
| openstackgerrit | Dmitry Tantsur proposed openstack-infra/project-config master: Publish ironic-tempest-plugin releases to pypi https://review.openstack.org/531148 | 13:13 |
| dtantsur | and then we can merge this ^^^ | 13:13 |
| AJaeger | dtantsur: LGTM | 13:13 |
| mordred | frickler: openstack, openstackstatus and openstackgerrit - and lemme check what accessbot is using | 13:13 |
| AJaeger | dtantsur: the first one I mean - didn't see the second one yet | 13:13 |
| dtantsur | k | 13:13 |
| dtantsur | the 2nd one can wait | 13:13 |
| AJaeger | dtantsur: do you really need the plugin on pypi? chandankumar, did we put tempest plugins to pypi? | 13:14 |
| mordred | frickler: although the plan corvus wrote up yesterday will consolidate us down to just the openstack bots assocated with the 'openstack' nickserv account | 13:14 |
| AJaeger | dtantsur: we should solve it anyhow - just curious... | 13:14 |
| dtantsur | AJaeger: we currently recommend pip install in our docs :) since it's branchless, I don't see big problems with people using it from pypi in a venv | 13:14 |
| AJaeger | dtantsur: ah, then you need to release it as well... | 13:14 |
| tosky | frickler: re failures on zuul, should we wait a bit before rechecking? | 13:14 |
| dtantsur | AJaeger: in any case, this can wait | 13:15 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Add neutron-vpnaas docs and release notes pub job to zuul layout https://review.openstack.org/522695 | 13:15 |
| frickler | tosky: I think things should be mostly fine now, maybe unless you are making changes to zuul.yaml rechecking should be fine | 13:16 |
| evrardjp | mordred: frickler ptgbot? | 13:16 |
| tosky | frickler: ack, thanks | 13:16 |
| evrardjp | thinking ahead just in case | 13:16 |
| chandankumar | AJaeger: currently we have not putted there but it would be good to put there | 13:16 |
| evrardjp | and happy new year to those I haven't said it yet. | 13:17 |
| frickler | infra-root: so according to the sigyn op we should be fine now bot-wise. just tell them when we invite sigyn to more channels, probably dmsimard did that yesterday? | 13:17 |
| mordred | frickler: accessbot is openstackinfra | 13:18 |
| mordred | frickler: ah - cool | 13:19 |
| Shrews | can an infra-root make my workday have less typing by +3'ing https://review.openstack.org/530918 for me? | 13:20 |
| *** trown|outtypewww is now known as trown | 13:21 | |
| AJaeger | Shrews, infra-root, I think we first need to restart Zuul - I'm seeing too many stuck changes and no move forward | 13:21 |
| Shrews | AJaeger: oh that's fun | 13:21 |
| frickler | AJaeger: oh, indeed, swapping activity has returned, it looked like it had stabilized until 10 minutes ago | 13:22 |
| frickler | still, looking at the time of the day, I'd say we can wait for corvus or maybe fungi to show up and decide | 13:25 |
| openstackgerrit | Merged openstack-infra/project-config master: Added initial jobs for blazar-tempest-plugin https://review.openstack.org/528674 | 13:25 |
| openstackgerrit | Merged openstack-infra/project-config master: Add Zuul project layout for ara-web https://review.openstack.org/528360 | 13:25 |
| openstackgerrit | Merged openstack-infra/project-config master: Add noop job for django_openstack_auth master branch https://review.openstack.org/529321 | 13:25 |
| AJaeger | wow, some merges ^ | 13:25 |
| *** dhill_ has joined #openstack-infra | 13:30 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove legacy job for python-heatclient https://review.openstack.org/513578 | 13:32 |
| *** lucas-hungry is now known as lucasagomes | 13:32 | |
| *** rosmaita has joined #openstack-infra | 13:39 | |
| *** dtantsur is now known as dtantsur|brb | 13:49 | |
| sshnaidm|afk | we see a lot of post_failure in jobs, is it known issue? | 13:58 |
| *** sshnaidm|afk is now known as sshnaidm|mtg | 13:58 | |
| *** bobh has joined #openstack-infra | 13:58 | |
| arxcruz|ruck | mtreinish: around ? | 14:03 |
| arxcruz|ruck | mtreinish: I'm noticing some tripleo jobs are not showing up in openstack-health, wondering if something has changed, in order to have status updated there | 14:04 |
| arxcruz|ruck | we do have the testrepository.subunit file | 14:04 |
| *** bobh has quit IRC | 14:08 | |
| openstackgerrit | Stibbons proposed openstack-dev/pbr master: Support v<semver> version https://review.openstack.org/531161 | 14:18 |
| *** jokke_ has quit IRC | 14:21 | |
| *** hjensas has joined #openstack-infra | 14:22 | |
| *** dtantsur|brb is now known as dtantsur | 14:22 | |
| frickler | sshnaidm|mtg: yes, zuul has some memory issues (again), please bear with us for a bit | 14:25 |
| *** dansmith has quit IRC | 14:25 | |
| *** smatzek has quit IRC | 14:25 | |
| AJaeger | infra-root, anybody in the US already awake? Do you want to restart zuul - or wait and debug? We have so many timeouts that it's unusable IMHO... | 14:26 |
| *** hongbin_ has joined #openstack-infra | 14:27 | |
| *** therve_ is now known as therve | 14:29 | |
| *** mriedem has joined #openstack-infra | 14:29 | |
| *** hongbin_ has quit IRC | 14:29 | |
| *** hongbin has joined #openstack-infra | 14:30 | |
| frickler | AJaeger: o.k., I'm convinced, proceeding to save the queues in preparation for a restart now | 14:30 |
| frickler | infra-root: ^^ | 14:30 |
| *** kgiusti has joined #openstack-infra | 14:31 | |
| *** rosmaita has quit IRC | 14:32 | |
| *** gouthamr has joined #openstack-infra | 14:34 | |
| frickler | infra-root: failed to receive any output for the queues, restarted zuul-scheduler anyway now | 14:37 |
| *** bobh has joined #openstack-infra | 14:38 | |
| *** esberglu has joined #openstack-infra | 14:39 | |
| *** yamamoto has quit IRC | 14:43 | |
| *** yolanda has quit IRC | 14:45 | |
| frickler | o.k., zuul-web seems to have needed a restart, too. /me is curious what the bots will do when I send the next status notice now | 14:45 |
| frickler | #status notice zuul has been restarted, all queues have been reset. please recheck your patches when appropriate | 14:45 |
| openstackstatus | frickler: sending notice | 14:45 |
| *** xarses_ has quit IRC | 14:46 | |
| -openstackstatus- NOTICE: zuul has been restarted, all queues have been reset. please recheck your patches when appropriate | 14:47 | |
| openstackstatus | frickler: finished sending notice | 14:48 |
| *** lbragstad has joined #openstack-infra | 14:54 | |
| *** eharney has joined #openstack-infra | 14:54 | |
| *** ramishra has joined #openstack-infra | 14:56 | |
| *** esberglu has quit IRC | 14:57 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove legacy job for python-heatclient https://review.openstack.org/513578 | 15:00 |
| openstackgerrit | Merged openstack-infra/project-config master: Add neutron-vpnaas docs and release notes pub job to zuul layout https://review.openstack.org/522695 | 15:01 |
| *** yamamoto has joined #openstack-infra | 15:01 | |
| *** smatzek has joined #openstack-infra | 15:01 | |
| *** smatzek has quit IRC | 15:02 | |
| *** smatzek has joined #openstack-infra | 15:02 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/project-config master: Run legacy-tempest-dsvm-nova-libvirt-kvm-apr in devstack experimental https://review.openstack.org/531170 | 15:10 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: WIP Rework log streaming to use logging https://review.openstack.org/531171 | 15:10 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove legacy jobs for python-heatclient https://review.openstack.org/513579 | 15:17 |
| ttx | mordred: just saw your message. I think going through Jean-Daniel is still a good way | 15:21 |
| mtreinish | arxcruz|ruck: if subunit data is getting picked up and written to the subunit2sql db then the jobs will show up in o-h | 15:21 |
| thingee | hey can any infra core do a quick review for the success and thanks bot so I can get a super user post out please https://review.openstack.org/#/c/510699/3 | 15:21 |
| arxcruz|ruck | mtreinish: so, how can I check if it's being picked up? because we had a lot of failures recently that are not showing up in o-h | 15:21 |
| mtreinish | arxcruz|ruck: https://github.com/openstack-infra/system-config/blob/master/modules/openstack_project/files/logstash/jenkins-log-client.yaml#L6-L27 | 15:22 |
| arxcruz|ruck | mtreinish: I assume testrepository.subunit.gz is caught also right ? | 15:23 |
| mtreinish | arxcruz|ruck: yeah it trys the file and the file +.gz | 15:23 |
| arxcruz|ruck | mtreinish: thanks, i'll check on my side | 15:24 |
| *** smatzek has quit IRC | 15:24 | |
| mordred | ttx: ah! thanks for reminding me | 15:25 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck master: Add query for cinder CapacityFilter fail bug 1741275 https://review.openstack.org/531175 | 15:25 |
| openstack | bug 1741275 in Cinder "CapacityFilter intermittently fails in the gate due to insufficient virtual space on the host" [Undecided,Confirmed] https://launchpad.net/bugs/1741275 | 15:25 |
| mordred | infra-root: jean-daniel repsponded, our OVH account should be fixed now | 15:26 |
| dmsimard | do we know what was the problem ? | 15:26 |
| pabelanger | frickler: I am going to try and help on the zuul memory front, will be a good learning experience, I think corvus wanted to do that before bumping to 30GB | 15:26 |
| pabelanger | mordred: ack | 15:26 |
| *** smatzek has joined #openstack-infra | 15:26 | |
| pabelanger | https://review.openstack.org/530945 reverts clouds.yaml removal of OVH | 15:27 |
| *** smatzek_ has joined #openstack-infra | 15:27 | |
| *** smatzek_ has quit IRC | 15:28 | |
| *** smatzek_ has joined #openstack-infra | 15:28 | |
| AJaeger | mordred: did you test that the servers are up again? Can we now merge https://review.openstack.org/#/c/530818/ to enable it? | 15:29 |
| frickler | pabelanger: I just not sure whether it will be possible to make short-term progress there and we have the end of the cycle coming up, so upping the flavor might still be a good intermediate solution | 15:30 |
| AJaeger | pabelanger, dmsimard, could either of you do quick cleanup review for devstack-gate, please? https://review.openstack.org/#/c/528817 | 15:31 |
| *** smatzek has quit IRC | 15:31 | |
| dmsimard | +3 | 15:32 |
| pabelanger | AJaeger: we first need 530945 for OVH revert | 15:32 |
| * frickler goes to enjoy rush hour traffic now, bbl | 15:32 | |
| dmsimard | I'll be going through my review dashboard today btw | 15:32 |
| dmsimard | pabelanger: let's wait on mordred's ack from AJaeger's question | 15:32 |
| *** smatzek_ has quit IRC | 15:33 | |
| AJaeger | thanks, dmsimard | 15:33 |
| fungi | mordred: echoing dmsimard's question... did jd say what went wrong with the account? | 15:33 |
| AJaeger | pabelanger: ok, will wait with +2... | 15:33 |
| openstackgerrit | Merged openstack-infra/system-config master: Update SSH key for shrews https://review.openstack.org/530918 | 15:35 |
| *** smatzek has joined #openstack-infra | 15:36 | |
| *** smatzek has quit IRC | 15:38 | |
| *** AJaeger has quit IRC | 15:40 | |
| *** xarses_ has joined #openstack-infra | 15:42 | |
| *** AJaeger has joined #openstack-infra | 15:44 | |
| *** eharney has quit IRC | 15:46 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck master: Add query for cinder CapacityFilter fail bug 1741275 https://review.openstack.org/531175 | 15:50 |
| openstack | bug 1741275 in Cinder "CapacityFilter intermittently fails in the gate due to insufficient virtual space on the host" [Undecided,Confirmed] https://launchpad.net/bugs/1741275 | 15:50 |
| *** smatzek has joined #openstack-infra | 15:51 | |
| *** smatzek has quit IRC | 15:55 | |
| *** armax has joined #openstack-infra | 15:56 | |
| *** smatzek has joined #openstack-infra | 15:57 | |
| *** smatzek_ has joined #openstack-infra | 16:00 | |
| *** hemna_ has joined #openstack-infra | 16:01 | |
| *** udesale has joined #openstack-infra | 16:01 | |
| *** smatzek has quit IRC | 16:01 | |
| *** smatzek_ has quit IRC | 16:05 | |
| *** smatzek has joined #openstack-infra | 16:06 | |
| *** smatzek has quit IRC | 16:07 | |
| mordred | fungi, dmsimard: he did not - he said they were going to look in to it | 16:08 |
| *** smatzek has joined #openstack-infra | 16:08 | |
| mordred | fungi, dmsimard: also, fwiw, he said to not hesitate to contact him - so I thnk he's still our contact there | 16:08 |
| mordred | AJaeger, dmsimard, pabelanger: I have not verified that anything works yet | 16:09 |
| dmsimard | mordred: so we can enable nodepool ? did you confirm it was okay ? | 16:09 |
| dmsimard | ok | 16:09 |
| mordred | dmsimard: sorry, about 5 items deep in my morning stack atm - http://mirror.bhs1.ovh.openstack.org/pypi/ seems to be working | 16:10 |
| mordred | http://mirror.gra1.ovh.openstack.org/pypi/ not as much | 16:11 |
| dmsimard | possible he fixed bhs1 but not gra1 ? | 16:11 |
| openstackgerrit | Mike Perez proposed openstack-infra/puppet-statusbot master: Add configuration for thanks/success page url https://review.openstack.org/531185 | 16:13 |
| *** udesale has quit IRC | 16:14 | |
| *** edmondsw has joined #openstack-infra | 16:15 | |
| *** smatzek has quit IRC | 16:15 | |
| *** smatzek_ has joined #openstack-infra | 16:15 | |
| *** kopecmartin has quit IRC | 16:16 | |
| *** smatzek_ has quit IRC | 16:16 | |
| openstackgerrit | Mike Perez proposed openstack-infra/system-config master: Set the success/thanks page url for status bot https://review.openstack.org/531187 | 16:16 |
| *** smatzek has joined #openstack-infra | 16:22 | |
| *** snuffkin has quit IRC | 16:22 | |
| *** eharney has joined #openstack-infra | 16:24 | |
| *** nicolasbock has quit IRC | 16:24 | |
| *** nicolasbock has joined #openstack-infra | 16:25 | |
| *** smatzek has quit IRC | 16:27 | |
| openstackgerrit | Mike Perez proposed openstack-infra/statusbot master: Thanks & Success bot provide confirmation site url https://review.openstack.org/510699 | 16:27 |
| *** smatzek has joined #openstack-infra | 16:30 | |
| openstackgerrit | Miguel Lavalle proposed openstack-infra/project-config master: Update Neutron legacy jobs definition for master https://review.openstack.org/530500 | 16:32 |
| *** smatzek_ has joined #openstack-infra | 16:34 | |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/openstack-zuul-jobs master: Skip tempest 35 job for doc and reno only changes https://review.openstack.org/531194 | 16:35 |
| *** smatzek has quit IRC | 16:35 | |
| *** e0ne has quit IRC | 16:36 | |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/openstack-zuul-jobs master: Skip legacy-grenade-dsvm-neutron job for doc and reno only changes https://review.openstack.org/531195 | 16:40 |
| corvus | dmsimard: do you want to remove +r ? | 16:41 |
| *** dtantsur is now known as dtantsur|afk | 16:41 | |
| dmsimard | yes, I can do that | 16:41 |
| dmsimard | testing something first | 16:42 |
| *** ramishra has quit IRC | 16:45 | |
| dmsimard | oh yeah I remember now.. ugh | 16:45 |
| dmsimard | since you can't change modes with chanserv, I have to actually run mlock -r, and then an empty mlock. | 16:45 |
| dmsimard | if you don't see me talking for the next 6 minutes you know why :D | 16:46 |
| *** ChanServ sets mode: -r | 16:48 | |
| *** gcb has quit IRC | 16:49 | |
| *** gcb has joined #openstack-infra | 16:50 | |
| clarkb | dmsimard: re gra1 mirror also possoble they vouldnt recover the instance after whatever happened | 16:52 |
| *** pcaruana has quit IRC | 16:52 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Allow Ansible 2.4 https://review.openstack.org/531009 | 16:52 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Don't remove builds that don't exist https://review.openstack.org/531198 | 16:52 |
| clarkb | we can probably go ahead and reenable bhs1 then sort out gra1 | 16:52 |
| *** smatzek_ has quit IRC | 16:53 | |
| pabelanger | wfm | 16:54 |
| fungi | yeesh, the irc spammers are mostly gone from freenode at the moment, but are picking up activity levels on oftc now | 16:54 |
| dmsimard | yuck .. | 16:55 |
| dmsimard | someone has too much time on their hands | 16:55 |
| fungi | got a few on the gnome community's irc network as well | 16:55 |
| *** sdague has joined #openstack-infra | 16:56 | |
| *** smatzek_ has joined #openstack-infra | 16:56 | |
| *** jpich has quit IRC | 16:56 | |
| *** danpawlik has quit IRC | 16:56 | |
| *** danpawlik has joined #openstack-infra | 16:57 | |
| Shrews | what we need is a paywall | 16:57 |
| Shrews | deposited directly to my bank account, of course | 16:57 |
| dmsimard | mirror01.gra1.ovh.openstack.org shows up as SUSPENDED, I'll try and figure that out | 16:59 |
| openstackgerrit | Merged openstack-infra/devstack-gate master: Remove run-tests and py3-run-tests https://review.openstack.org/528817 | 16:59 |
| *** smatzek_ has quit IRC | 17:00 | |
| dmsimard | we have ourselves a mirror: http://mirror.gra1.ovh.openstack.org/pypi/ | 17:01 |
| pabelanger | dmsimard: clarkb: https://review.openstack.org/530945/ will add back OVH to clouds.yaml so we can start poking at APIs | 17:01 |
| pabelanger | nice | 17:01 |
| dmsimard | uptime is showing up as 116 days o_O | 17:03 |
| fungi | suspended huh? | 17:03 |
| *** esberglu has joined #openstack-infra | 17:03 | |
| dmsimard | so it was really just suspended | 17:04 |
| fungi | neat | 17:04 |
| fungi | resume ftw! | 17:04 |
| dmsimard | I had to do a double take on osc because I expected "unsuspend" but nope | 17:04 |
| clarkb | dmsimard: did you have to reboot it? | 17:06 |
| dmsimard | no, just a resume | 17:06 |
| clarkb | ah | 17:06 |
| dmsimard | hence the 116 day uptime | 17:06 |
| fungi | infra-root: unrelated to the cpu pipelining exploits disclosed today, be careful with recursive chown/chgrp and avoid following symlinks: http://lists.gnu.org/archive/html/coreutils/2017-12/msg00045.html | 17:06 |
| dmsimard | looks like it might have been suspended around Dec 29th/Dec 30th | 17:06 |
| dmsimard | on the topic of cpu exploits, are we planning to upgrade kernels and reboot ? | 17:07 |
| clarkb | dmsimard: yes but no kernels available yet as far sa I can tell | 17:07 |
| clarkb | ubuntu's story is the embargo was supposed to lift on the 9th | 17:07 |
| clarkb | but it lifted a week early and now they are scrambling as a result | 17:08 |
| fungi | dmsimard: we probably should, though i recommend coordinating it in #-incident once packages are up | 17:08 |
| dmsimard | fungi: +1 | 17:08 |
| dmsimard | Just learned that the packages are up for centos so I'll be busy for a bit updating RDO's stuff | 17:10 |
| *** caphrim007 has joined #openstack-infra | 17:10 | |
| dmsimard | I guess we can do the git farm on -infra | 17:10 |
| *** slaweq has joined #openstack-infra | 17:10 | |
| fungi | yeah, rolling restarts on those should be safe if we disable them in haproxy one at a time | 17:11 |
| fungi | restarting git.o.o itself will be disruptive though | 17:11 |
| clarkb | pabelanger wanted to do that anyways for general patching | 17:11 |
| clarkb | (but they are also not servers that I'd worry about given the vulnerability) | 17:12 |
| clarkb | I'm more concerned about my laptop and infracloud | 17:12 |
| fungi | yeah, our biggest exposure is anywhere we run arbitrary payloads. job nodes and zuul executors are my largest worry | 17:13 |
| clarkb | no kernels for either yet :/ | 17:13 |
| fungi | and our personal workstations/laptops/devices of course, right | 17:13 |
| clarkb | https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SpectreAndMeltdown fwiw | 17:14 |
| *** dhill_ has quit IRC | 17:14 | |
| fungi | in other news, hurricane-force winds coupled with a thunderstorm and blizzard is... an interesting combo | 17:14 |
| corvus | fungi: i woke up to a 4.4 today. | 17:14 |
| fungi | nice! | 17:15 |
| corvus | i'm walking distance from the epicenter | 17:15 |
| fungi | i basically didn't sleep last night because the wind was deafening | 17:15 |
| *** slaweq has quit IRC | 17:15 | |
| dmsimard | They're announcing around 16 inches of snow here | 17:15 |
| corvus | everything is awesome! | 17:16 |
| fungi | now if we could just combine them all into a thundersnowcanequake | 17:16 |
| pabelanger | corvus: yah, I can look again today how to rotate out a git server from LB | 17:17 |
| *** gyee has joined #openstack-infra | 17:17 | |
| dmsimard | so I didn't read the meltdown/spectre docs in depth but.. regardless of if we patch our stuff or not -- given we're on virtual machines in different clouds, that means we're still vulnerable if the cloud provider hasn't updated, correct ? | 17:17 |
| corvus | dmsimard: yes, though i'm assuming that's what was happening over december.... has rax mentioned anything publicly yet i wonder? | 17:17 |
| dmsimard | hmm, there was something specific about xen actually, I think only PV was impacted ? /me looks | 17:18 |
| *** spzala has joined #openstack-infra | 17:18 | |
| *** spzala has quit IRC | 17:18 | |
| danpawlik | coreycb: Hi. Do you know what is the state of cannonical team to create new Qemu package for CVE-2017-5715 ? | 17:18 |
| fungi | dmsimard: at least in rackspace, i read the xen advisories and they make it sound like the hypervisors are mostly safe from guest breakout with pvhvm (it's mostly straight pv 64-bit guests you have to worry about being able to address the hypervisor's pagetable) | 17:18 |
| corvus | dmsimard, fungi: nope, apparently that's not what was happening in december | 17:18 |
| clarkb | dmsimard: yes my understanding is that particularly for meltdown (easier to exploit) hypervisors can have their memory read which would expose other guests | 17:18 |
| corvus | https://techcrunch.com/2018/01/03/cloud-infrastructure-vendors-begin-responding-to-chip-kernel-vulnerability/ | 17:18 |
| *** smatzek has joined #openstack-infra | 17:18 | |
| corvus | rax is at the end there | 17:18 |
| clarkb | dmsimard: but short of shutting down all our servers I don't think we can do much there | 17:19 |
| corvus | apparently they weren't cool enough to be part of the nda | 17:19 |
| fungi | yeah | 17:19 |
| dmsimard | yeah.... this sucks | 17:19 |
| clarkb | spectre is a more widespread problem but is much harder to take advantage of | 17:20 |
| danpawlik | or maybe someone else have some contacts with canonical | 17:20 |
| dmsimard | I expected Intel's stock price to tank way more than that | 17:20 |
| fungi | xen advisories mention a hypervisor shim you can insert to safely protect the host from pv guests | 17:20 |
| fungi | so i wouldn't be surprised if rax mitigates that way | 17:20 |
| *** eharney has quit IRC | 17:20 | |
| coreycb | danpawlik: i believe this has the latest status: https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SpectreAndMeltdown | 17:21 |
| *** smatzek has quit IRC | 17:21 | |
| *** smatzek has joined #openstack-infra | 17:21 | |
| corvus | here's an idea: shut everything down and extend winter holidays another week. | 17:21 |
| dmsimard | +1 | 17:21 |
| pabelanger | I could get behind that | 17:21 |
| fungi | corvus: i'll get started by shutting down my systems now ;) | 17:22 |
| dmsimard | alternatively justify 25% lack of productivity due to "performance impact" | 17:22 |
| *** eharney has joined #openstack-infra | 17:22 | |
| fungi | i should go out and play in the snow, except for these 50mph wind gusts (at least we're below hurricane velocities again now) | 17:22 |
| corvus | dmsimard: yeah, you get to use that for the next 10 years too | 17:22 |
| danpawlik | coreycb: ok, thanks. | 17:23 |
| danpawlik | coreycb: its really sad, that suse and red hat has a new package but all patches are confidential | 17:23 |
| fungi | the kernel patches are public | 17:24 |
| clarkb | danpawlik: suse has a new package? my tumbleweed has yet to get 4.14.11 (maybe 4.14.9 got a backported patch?) | 17:24 |
| coreycb | danpawlik: it may have something to do with the embargo, i'm not on the security team though | 17:24 |
| fungi | only the discussion around the kernel patches was embargoed | 17:24 |
| danpawlik | coreycb: and maybe you will know, why debian is telling that package related is just linux (https://security-tracker.debian.org/tracker/CVE-2017-5715), on ubuntu you have (https://people.canonical.com/~ubuntu-security/cve/2017/CVE-2017-5715.html) just kernel but red hat and suse has other opinion: https://www.suse.com/security/cve/CVE-2017-5715/ and | 17:24 |
| danpawlik | https://access.redhat.com/security/cve/cve-2017-5715 | 17:24 |
| danpawlik | hmm | 17:25 |
| danpawlik | fungi: but its partialy related to the kernel | 17:25 |
| danpawlik | fungi: so its possible that emgargo is also on that bug too | 17:25 |
| *** dhill_ has joined #openstack-infra | 17:26 | |
| danpawlik | as I read on the Internet that patch is for: "qemu update to pass the new MSR and CPU flags from the microcode update" | 17:26 |
| fungi | yeah, there are of course other mitigating patches being implemented in, e.g., virtualization systems for similar reasons | 17:26 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: doc: Rework features doc https://review.openstack.org/512248 | 17:27 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/pbr master: doc: Minor rework of usage doc https://review.openstack.org/512249 | 17:27 |
| fungi | it's hard to say that these bugs "affect" any particular piece of software. they're hardware design flaws which need to be mitigated or worked around by software. kernels are in the best position to do most of that | 17:27 |
| persia | Excitingly, there is a decent chance that hypervisors can be made immune to this before hardware, making a strong argument for running sensitive materials on clouds (assuming the substrate maintainers make efforts to control code run on hardware) | 17:27 |
| fungi | persia: totally agree | 17:28 |
| stephenfin | dhellmann, mordred: Fancy shoving these pbr docs patches through? https://review.openstack.org/#/q/topic:doc-rework+(status:open+OR+status:merged) | 17:29 |
| dmsimard | fungi: yeah for Red Hat, there's updates to kernel microcode_ctl qemu-kvm qemu-kvm-rhev libvirt dracut linux_firmware | 17:30 |
| dmsimard | (that's from an openstack perspective on a RHEL deployment) | 17:30 |
| *** stephenfin has left #openstack-infra | 17:30 | |
| *** stephenfin has joined #openstack-infra | 17:30 | |
| EmilienM | clarkb, pabelanger : I would appreciate some review on https://review.openstack.org/#/c/531030/ - thanks | 17:32 |
| EmilienM | clarkb, pabelanger : I'll do centos as well | 17:32 |
| fungi | persia: well, not necessarily clouds, but virtual servers at the very least | 17:32 |
| persia | fungi: Well, yes, but that isn't quite as exciting, except for the Qubes folk. | 17:34 |
| fungi | heh, indeed ;) | 17:34 |
| dmsimard | oh, a demo exploiting meltdown for capturing realtime data.. https://twitter.com/misc0110/status/948706387491786752 | 17:36 |
| *** erlon has quit IRC | 17:36 | |
| clarkb | dmsimard: ya meltdown is much more trivial to exploit | 17:38 |
| fungi | corvus: i particularly enjoyed "we have not yet heard from ibm" | 17:38 |
| clarkb | dmsimard: but is also the attack with patches (kpti) | 17:38 |
| clarkb | now just need new kernels | 17:38 |
| clarkb | do we have a revert of the ovh disable in nodepool yet? | 17:38 |
| *** rlf1 has joined #openstack-infra | 17:39 | |
| *** greghaynes has joined #openstack-infra | 17:42 | |
| coreycb | danpawlik: i'm not sure. #ubuntu-hardened may be a good place for that question. | 17:43 |
| clarkb | mordred: there are quite a few items with your name attached to them on https://etherpad.openstack.org/p/zuulv3-issues anything I can help with as far as moving things along? But also would be good to clear out things that have been fixed | 17:43 |
| clarkb | mordred: the tox with sudo thing came up again too (and is on the etherpad) is that sorted yet? | 17:43 |
| dmsimard | clarkb: I saw a revert of the clouds.yaml, not for nodepool | 17:44 |
| danpawlik | coreycb: thanks. I can't find such irc group to ask so I was thinking that I will ask you / community and someone will know something more | 17:44 |
| danpawlik | join #ubuntu-hardened | 17:44 |
| clarkb | dmsimard: ok I'llget a nodepool revert up now | 17:44 |
| fungi | clarkb: curious if you happen to have seen mriedem's post to -dev about issues serving logs from node-named directories, and what precipitated the regressions there | 17:45 |
| *** lucasagomes is now known as lucas-afk | 17:46 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Revert "Disable OVH" https://review.openstack.org/531202 | 17:46 |
| clarkb | dmsimard: pabelanger fungi mordred ^ use ovh again in nodepool | 17:46 |
| *** slaweq has joined #openstack-infra | 17:47 | |
| clarkb | fungi: os-loganalyze is very specific about what files it will bother annotating since logs come in all the formats. I'm betting the new job changing file paths (and possibly file names?) no longer match os loganalyze's expectations | 17:47 |
| clarkb | fungi: I can take a look though | 17:48 |
| fungi | clarkb: thanks, i'll dig into osla internals then. i looked at the apache vhost config and it didn't appear to treat certain subpaths specially | 17:49 |
| clarkb | andreaf: looking into ^ the first thing I notice is that none of the log files are compressed. That should be addressed a relatively urgent thing to fix | 17:49 |
| *** armax has quit IRC | 17:50 | |
| *** armax has joined #openstack-infra | 17:50 | |
| mordred | clarkb, frickler: I feel like someone was discussing the need for tox without sudo revoked earlier | 17:50 |
| clarkb | fungi: ^ may actually be the problem if the filename osla is looking for include the .gz | 17:50 |
| clarkb | mordred: ya they were in sb | 17:50 |
| clarkb | mordred: for a functional job for os privsep | 17:50 |
| *** sshnaidm|mtg is now known as sshnaidm | 17:51 | |
| clarkb | fungi: hrm no ^.*\.txt(\.gz)?$ is the regex in the wsgi conf and that should match either version | 17:54 |
| mordred | https://review.openstack.org/#/c/512653/ <-- there's a patch for a that (from scanning the etherpad) | 17:55 |
| clarkb | but we enforce the .gz in apache. RewriteRule ^/(.*\.txt\.gz)$ /htmlify/$1 [QSA,L,PT,NS] | 17:55 |
| clarkb | fungi: mriedem andreaf ^ pretty sure that is the issue. If the job starts compressing the log files again it should work | 17:55 |
| jlvillal | Any known issues with unit tests jobs? Or is it just us (Ironic)? | 17:55 |
| jlvillal | Never mind. Seems to be just us. | 17:56 |
| clarkb | jlvillal: zuuls status page builds list should help you filter and see if others are havingtrouble | 17:56 |
| clarkb | but I'm not aware of anything | 17:56 |
| jlvillal | clarkb: Thanks! | 17:57 |
| clarkb | fungi: I'm writing a response to the dev list now unless you have already started one | 17:58 |
| corvus | i think we want the job performing the compression. i'd like for us to get to the point where we can rely on that because it's in a post playbook, and stop having the cron | 17:59 |
| *** Apoorva has joined #openstack-infra | 17:59 | |
| clarkb | corvus: yes absolutely, the cron already has a hard time keeping up so pushing that into the scale out job workers is a big win | 17:59 |
| corvus | hopefully that's something we can put in the new genericized log archiving role | 18:00 |
| *** david-lyle has quit IRC | 18:00 | |
| openstackgerrit | Merged openstack-infra/system-config master: Revert "Remove ovh from clouds.yaml files" https://review.openstack.org/530945 | 18:01 |
| *** david-lyle has joined #openstack-infra | 18:01 | |
| fungi | clarkb: i had not started one, go for it | 18:02 |
| fungi | i had only just started to dig into the problem | 18:02 |
| *** derekh has quit IRC | 18:03 | |
| *** Goneri has joined #openstack-infra | 18:05 | |
| *** slaweq has quit IRC | 18:10 | |
| AJaeger | clarkb: I run 4.14.11 from http://download.opensuse.org/repositories/Kernel:/stable/standard - always latest stable kernel | 18:12 |
| *** jascott1 has joined #openstack-infra | 18:13 | |
| AJaeger | mordred: it was stephenfin | 18:14 |
| *** SumitNaiksatam has joined #openstack-infra | 18:14 | |
| dmsimard | mordred: does http://paste.openstack.org/raw/638205/ ring you a bell ? image is http://paste.openstack.org/show/638215/ | 18:17 |
| AJaeger | mordred: I think we can merge the tox-with-sudo change - after removing the duplicated job, see my comment on https://review.openstack.org/#/c/512653 | 18:18 |
| mordred | dmsimard: no - I'm not sure I've seen a conflict on delete before | 18:18 |
| dmsimard | mordred: haven't tried deleting it manually yet, going to try and reproduce with shade proper | 18:18 |
| mordred | AJaeger: actually - I feel like corvus and I had a brief conversation on this topic right before the break ... | 18:18 |
| openstackgerrit | Xiang Chen proposed openstack-infra/git-review master: fix bug #!/story/2001438 https://review.openstack.org/531209 | 18:19 |
| corvus | mordred: that was so last year... what are we talking about? :) | 18:19 |
| mordred | dmsimard: if you do, can you put shade.simple_logging(http_debug=True) at the top of your script and save the output? the json decode error bothers me, I think there is an error message we're missing | 18:19 |
| mordred | corvus: revoke-sudo and the unittest base job | 18:20 |
| corvus | mordred: oh yes, i believe i said something which caused you to discover a very clever solution to the problem. | 18:20 |
| AJaeger | mordred: perhaps in context of https://review.openstack.org/529662 ? | 18:20 |
| corvus | mordred: you may have written it into the margins of a proof or something. | 18:20 |
| mordred | corvus: yes - that is what I remember too | 18:20 |
| AJaeger | mordred: check 529662, that moves the sudo call... | 18:21 |
| mordred | corvus: but I don't remember the solution or what proof I wroteit in the margins of | 18:21 |
| clarkb | corvus: mordred andreaf mriedem fyi just pushed https://review.openstack.org/531208 to confirm the log annotation problems are related to gzipping | 18:21 |
| clarkb | I expect that the general gzipping of log files will be more involved so didn't start with that | 18:21 |
| mordred | AJaeger: yes! | 18:21 |
| fungi | corvus: i suspect the margins were too small to contain it | 18:21 |
| clarkb | fungi: oh ^ you too | 18:21 |
| mordred | AJaeger, corvus: https://review.openstack.org/529662 is, in fact, what we came up with | 18:21 |
| AJaeger | mordred: so, plan is to merge 529662 and then rework the tox-with-sudo to just have a new run.yaml? | 18:22 |
| mriedem | clarkb: huh, how did that regresss? didn't devstack's post-test host cleanup code do that before? | 18:22 |
| corvus | mordred: you wrote it in the margins of a git commit! clever :) | 18:22 |
| AJaeger | Let me +3 529662. mordred will you rework the tox-with-sudo or shall I? | 18:22 |
| mordred | AJaeger: yup. lemme do that real quick | 18:22 |
| clarkb | mriedem: tempest-full no longer uses devstack-gate so its entirely new cleanup code that wasn't ported properly | 18:22 |
| *** slaweq has joined #openstack-infra | 18:23 | |
| mriedem | ok | 18:23 |
| mriedem | figured it was something like that | 18:23 |
| clarkb | mriedem: basically we ported devstack-gate into zuulv3 native job config and this was something missed | 18:23 |
| fungi | devstack-gate essentially being a legacy framework obsoleted by the available options in zuul v3 | 18:23 |
| openstackgerrit | Xiang Chen proposed openstack-infra/git-review master: fix bug #!/story/2001438 https://review.openstack.org/531209 | 18:23 |
| openstackgerrit | Xiang Chen proposed openstack-infra/git-review master: fix bug #!/story/2001438 https://review.openstack.org/531209 | 18:24 |
| dmsimard | mordred: reproduced with debugging enabled: http://paste.openstack.org/show/638223/ | 18:24 |
| mordred | dmsimard: thanks! | 18:24 |
| openstackgerrit | Xiang Chen proposed openstack-infra/git-review master: fix bug #!/story/2001438 https://review.openstack.org/531209 | 18:24 |
| dmsimard | mordred: looks like the http 409 is text/plain, not JSON | 18:25 |
| dmsimard | "RESP BODY: Omitted, Content-Type is set to text/plain; charset=UTF-8. Only application/json responses have their bodies logged." | 18:25 |
| fungi | thinking about it, i guess we've been deprecating pieces of devstack-gate and moving them into our services for years... who remembers when nodepool was actually some cron'd scripts in devstack-gate? | 18:25 |
| *** dhill_ has quit IRC | 18:25 | |
| *** shardy has quit IRC | 18:25 | |
| AJaeger | team, three quick reviews for job removals in openstack-zuul-jobs: https://review.openstack.org/#/c/528645/ , https://review.openstack.org/#/c/530240/ , https://review.openstack.org/#/c/513579/ , please | 18:25 |
| *** dhill_ has joined #openstack-infra | 18:26 | |
| corvus | fungi: i remember when nodepool was a jenkins job :) | 18:26 |
| corvus | (there was a job to create nodes, and another job to delete them. this design is not advised.) | 18:27 |
| mordred | dmsimard: ok. cool. I thnk there is a patch lurking somewhere to fix that | 18:27 |
| dmsimard | mordred: running openstack image delete yields a slightly more useful error message: Image 4bbd529d-83d3-4d6f-a69e-df7e1e69aa6b could not be deleted because it is in use: The image cannot be deleted because it is in use through the backend store outside of Glance. (HTTP 409) | 18:27 |
| *** slaweq has quit IRC | 18:27 | |
| fungi | corvus: indeed, though the scripts those jobs ran were in the devstack-gate repo was i guess my point | 18:28 |
| mordred | dmsimard: ++ that is the error message we should be producing in shade too | 18:28 |
| corvus | fungi: ah yep :) | 18:28 |
| mordred | that will be fixed by the thing I'm thiking about | 18:28 |
| dmsimard | mordred: I'll file a bug just so I don't forget and can refer to it if need be | 18:28 |
| mordred | dmsimard: so - as for the other thing, *blink-blink* | 18:28 |
| mordred | dmsimard: thanks | 18:29 |
| mordred | dmsimard: I think that may be an issue to raise with the tripleo cloud team | 18:29 |
| *** yamamoto has quit IRC | 18:29 | |
| mordred | (the in-use-by-something-else error) | 18:29 |
| dmsimard | mordred: oh, yeah sure I'll take care of that | 18:29 |
| dmsimard | they upgraded to ocata recently and it was a bumpy ride | 18:30 |
| dmsimard | 2 weeks of headaches :( | 18:30 |
| clarkb | mordred: corvus is stage-output the magical zuul role that should do the gzipping? | 18:30 |
| fungi | dmsimard: but... hooray for being on latest openstack i guess? | 18:32 |
| AJaeger | mordred: shouldn't we move the revoke-sudo in project-config as well? | 18:32 |
| fungi | dmsimard: d'oh, wait, that's pike ;) | 18:32 |
| * dmsimard frowns | 18:33 | |
| *** yamamoto has joined #openstack-infra | 18:33 | |
| dmsimard | ocata actually EOLs like, next month | 18:33 |
| *** rlf1 has quit IRC | 18:33 | |
| clarkb | hrm stage-output claims to already compressing things but that doesn't appear to be happening | 18:34 |
| *** erlon has joined #openstack-infra | 18:34 | |
| *** numans_afk is now known as numans | 18:36 | |
| clarkb | oh! there is a comment in the file I am updating that says it bypasses stage-output | 18:37 |
| AJaeger | mordred: thinking further - no need to change anything for project-config | 18:37 |
| clarkb | so maybe my patch is the actual fix afterall | 18:37 |
| clarkb | corvus: ^ fyi | 18:37 |
| *** yamamoto has quit IRC | 18:37 | |
| corvus | clarkb: mordred and andreaf have been the most involved in the log copying rework. i know the general shape of the end-result, but i don't have the current context. | 18:41 |
| clarkb | corvus: thanks, mordred andreaf can you review https://review.openstack.org/#/c/531208/ to fix mriedem's problems with os-loganalyze? I now think that that fix may actually be the fix until we stop bypassing stage-output with service logs | 18:41 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Move revoke-sudo from unittest pre to job run https://review.openstack.org/529662 | 18:42 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Limit Gerrit SSH API connections to 100 per addr https://review.openstack.org/529712 | 18:43 |
| dmsimard | mordred: fyi image bug if you want to refer to it: https://storyboard.openstack.org/#!/story/2001440 | 18:45 |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Disable OVH" https://review.openstack.org/531202 | 18:47 |
| AJaeger | yeah, another 238 nodes ;) ^ | 18:47 |
| AJaeger | corvus: do we still need the merge-check pipeline or can we remove it from project-config/zuul.d/pipelines.yaml ? | 18:50 |
| dmsimard | AJaeger: if it works | 18:53 |
| dmsimard | should keep an eye on nodepool grafana just in case | 18:53 |
| clarkb | AJaeger: I think we can remove it if we are willing to rely on gerrits merge checking | 18:54 |
| AJaeger | clarkb: we're not using it currently, let me propose the change... | 18:55 |
| *** ijw has joined #openstack-infra | 18:55 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Remove unused merge-check pipeline https://review.openstack.org/531215 | 18:57 |
| AJaeger | clarkb, corvus ^ | 18:57 |
| clarkb | AJaeger: zuul caught an error with it that you'll need to address first | 18:57 |
| AJaeger | clarkb: yeah - seems that we do use it. | 18:58 |
| * AJaeger loves zuul's self-checks | 18:58 | |
| clarkb | but in a trivial way I think we can drop that use of it | 18:58 |
| clarkb | unless that makes the system-required template invalid | 18:58 |
| *** caphrim007_ has joined #openstack-infra | 18:59 | |
| *** Goneri has quit IRC | 19:00 | |
| *** slaweq has joined #openstack-infra | 19:01 | |
| *** pcrews has joined #openstack-infra | 19:01 | |
| *** caphrim007 has quit IRC | 19:03 | |
| AJaeger | clarkb: I fear it will break it. Let's leave it for now... | 19:04 |
| *** slaweq has quit IRC | 19:05 | |
| *** pots has joined #openstack-infra | 19:08 | |
| mnaser | dmsimard: mordred is the in-use glance error by any chance running against an openstack cloud with rbd storage? | 19:10 |
| dmsimard | mnaser: yeah, I know what the 409 is, there is a volume using that image | 19:12 |
| mnaser | ok cool (or a nova instance too) | 19:12 |
| * mnaser goes back to reboot party | 19:12 | |
| clarkb | dmsimard: so you have external tools using nodepool images causing nodepool's image deletions to fail? | 19:13 |
| clarkb | speaking of reboot parties | 19:13 |
| dmsimard | Nope, nodepool boot from volume | 19:13 |
| clarkb | pabelanger: ^ do you want to work through the git backends today? | 19:13 |
| *** slaweq has joined #openstack-infra | 19:14 | |
| eumel8 | clarkb, ianw_pto: It seems we're ready to update translate-dev to Zanata 4.3.3. Awaiting a valid download URL from the Zanata guys and then I will send out a new proposal the next days. | 19:16 |
| clarkb | eumel8: sounds good, this is to address the bugs that were found right? | 19:17 |
| eumel8 | clarkb: yes, the wrong landing page after openid login | 19:18 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove obsolete interop jobs https://review.openstack.org/528645 | 19:19 |
| *** slaweq has quit IRC | 19:19 | |
| clarkb | there are 144 ovh instances in use right now | 19:20 |
| clarkb | elastic-recheck not showing a spike in the pip failures so far | 19:20 |
| eumel8 | clarkb: regarding the wrong accounts in the dev database we decided to drop the complete database and started with a new one. I think it's easier for you instead to puzzle the wrong accounts with the good one. This caused last year after changing the openid provider. | 19:21 |
| clarkb | eumel8: you'll lose the translation history and stuff but thats probbaly ok for the dev instance I guess | 19:22 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: WIP: Use native Zuul v3 bindep fallback tests https://review.openstack.org/530104 | 19:23 |
| eumel8 | clarkb: sure, but nobody can work on the dev server which registered before 2016. It's easier to install some test repos again | 19:23 |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy-grenade-publish-docs https://review.openstack.org/530240 | 19:24 |
| *** jascott1 has quit IRC | 19:26 | |
| *** jascott1 has joined #openstack-infra | 19:26 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: WIP: Use native Zuul v3 bindep fallback tests https://review.openstack.org/530104 | 19:27 |
| *** jascott1 has quit IRC | 19:28 | |
| *** jascott1 has joined #openstack-infra | 19:29 | |
| *** jascott1 has quit IRC | 19:30 | |
| openstackgerrit | Merged openstack-infra/openstack-zuul-jobs master: Remove legacy jobs for python-heatclient https://review.openstack.org/513579 | 19:30 |
| *** jascott1 has joined #openstack-infra | 19:33 | |
| EmilienM | clarkb: thanks for your review on https://review.openstack.org/#/c/531030/ - I'll start the mirror for puppetlabs on centos | 19:34 |
| *** slaweq has joined #openstack-infra | 19:35 | |
| *** yamamoto has joined #openstack-infra | 19:35 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Add native Zuul v3 bindep fallback tests https://review.openstack.org/530104 | 19:38 |
| *** smatzek has quit IRC | 19:39 | |
| *** slaweq has quit IRC | 19:39 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Add native Zuul v3 bindep fallback tests https://review.openstack.org/530104 | 19:40 |
| *** armax has quit IRC | 19:40 | |
| *** yamamoto has quit IRC | 19:42 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Use new bindep tests https://review.openstack.org/531222 | 19:44 |
| dmsimard | TIL certbot uses pbr https://github.com/certbot/certbot/blob/d039106b687f7300a41e11088412f16dd6f023d5/certbot-auto#L997 | 19:46 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: DNM: Test new bindep tests https://review.openstack.org/531225 | 19:47 |
| AJaeger | config-core, could you review https://review.openstack.org/530104 , please? Like to know whether the native bindep test is fine. Once that is merged, we can test it with 531225 and then use by default with 531222 | 19:48 |
| *** sshnaidm is now known as sshnaidm|off | 19:53 | |
| *** eharney has quit IRC | 20:01 | |
| *** jascott1 has quit IRC | 20:02 | |
| *** jascott1 has joined #openstack-infra | 20:02 | |
| *** jascott1 has quit IRC | 20:03 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config master: Limit Gerrit SSH API connections to 100 per addr https://review.openstack.org/529712 | 20:03 |
| *** jascott1 has joined #openstack-infra | 20:04 | |
| clarkb | fungi: can you multiline puppet like that? | 20:04 |
| clarkb | I guess we'll know soon enough | 20:04 |
| *** spzala has joined #openstack-infra | 20:04 | |
| fungi | yeah, i thought we'd done it elsewhere | 20:05 |
| fungi | at any rate, the missing $ characters were a no-go for it | 20:06 |
| fungi | bizarre that assigning collections like that requires you to have $ on the variable being assigned | 20:06 |
| fungi | i feel like i still don't grok some of these syntax nuances in puppet | 20:07 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/system-config master: Add Puppetlabs mirror for CentOS7 https://review.openstack.org/531229 | 20:07 |
| *** Goneri has joined #openstack-infra | 20:07 | |
| *** jascott1 has quit IRC | 20:08 | |
| *** jascott1 has joined #openstack-infra | 20:10 | |
| *** markmcclain has quit IRC | 20:10 | |
| *** markmcclain has joined #openstack-infra | 20:11 | |
| *** jascott1 has quit IRC | 20:13 | |
| *** slaweq has joined #openstack-infra | 20:13 | |
| *** nicolasbock has quit IRC | 20:13 | |
| *** SumitNaiksatam has quit IRC | 20:13 | |
| corvus | it's easy, you always need a $ except when it's not required | 20:14 |
| *** slaweq_ has joined #openstack-infra | 20:15 | |
| clarkb | my ubderstanding of it is you need the $ anywhere it is a variabke and drop the $ if it is a parameter name | 20:16 |
| *** slaweq has quit IRC | 20:18 | |
| fungi | ahh. that sort of makes sense | 20:19 |
| mordred | try one. wait for puppet-lint to complain, then try the other | 20:19 |
| mordred | that's my approach | 20:19 |
| fungi | seems to have been my approach today as well | 20:19 |
| dmsimard | is there a way in nodepool v2 to re-upload an image to a provider ? | 20:25 |
| clarkb | dmsimard: yes run the image-upload command | 20:25 |
| dmsimard | doesn't seem to exist :/ | 20:26 |
| clarkb | did you mean v3? | 20:26 |
| clarkb | v3 doesnt have it but the pre v3 stuff should | 20:27 |
| clarkb | v3 instead attempts to recocnile state more aggressively and upload things when necessary | 20:27 |
| clarkb | so you can force a reupload by deleting older images | 20:27 |
| dmsimard | http://paste.openstack.org/raw/638343/ is the help I get from nodepool -h | 20:28 |
| dmsimard | it's weird cause I remember image-upload existing .. | 20:29 |
| openstackgerrit | James E. Blair proposed openstack-infra/infra-specs master: Add IRC bot improvement spec https://review.openstack.org/531242 | 20:29 |
| corvus | clarkb, fungi, dmsimard, mnaser, mordred: ^ that's the brainstorming etherpad from yesterday dumped into spec form | 20:30 |
| dmsimard | subscribed ty | 20:30 |
| * mnaser adds $self and goes back to reboot party | 20:30 | |
| clarkb | dmsimard: if there is a zookeeper instead of a gearman I believe that is expected | 20:30 |
| clarkb | dmsimard: but if your builders speak gearman instead I would expectn upload command | 20:31 |
| dmsimard | it's kinda weird because we're in this weird mix of things where we're actually still running jenkins and we'll jump straight to zuul v3 | 20:33 |
| dmsimard | I'll figure something out | 20:33 |
| clarkb | the builder + zk work was independent of that though | 20:33 |
| *** jascott1 has joined #openstack-infra | 20:34 | |
| dmsimard | afaict our nodepool builder uses gearman | 20:34 |
| dmsimard | eh, maybe not.. seeing zookeeper stuff in the logs | 20:35 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Don't remove builds that don't exist https://review.openstack.org/531198 | 20:36 |
| dmsimard | I'll just run a new build and see what happens | 20:36 |
| *** jascott1 has quit IRC | 20:38 | |
| *** gagehugo has quit IRC | 20:39 | |
| *** jascott1 has joined #openstack-infra | 20:41 | |
| *** jascott1 has quit IRC | 20:43 | |
| *** jascott1 has joined #openstack-infra | 20:47 | |
| *** jascott1 has quit IRC | 20:50 | |
| *** slaweq has joined #openstack-infra | 20:50 | |
| *** smatzek has joined #openstack-infra | 20:50 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Fix pep8/yamllint errors https://review.openstack.org/530685 | 20:53 |
| *** edmondsw has quit IRC | 20:54 | |
| *** jascott1 has joined #openstack-infra | 20:54 | |
| *** eharney has joined #openstack-infra | 20:56 | |
| *** jascott1 has quit IRC | 20:56 | |
| *** jascott1 has joined #openstack-infra | 20:58 | |
| *** smatzek has quit IRC | 21:05 | |
| *** smatzek has joined #openstack-infra | 21:06 | |
| *** smatzek has quit IRC | 21:10 | |
| *** caphrim007_ has quit IRC | 21:10 | |
| *** caphrim007 has joined #openstack-infra | 21:11 | |
| clarkb | can an infra root be second review on https://review.openstack.org/#/c/516009/ to fix infra publications publishing? Thta will clear out an item from the zuulv3 issues etherpad | 21:17 |
| *** eumel8 has quit IRC | 21:19 | |
| *** edmondsw has joined #openstack-infra | 21:19 | |
| mordred | clarkb: done | 21:19 |
| dmsimard | There will be a brief moment of unavailability of the RDO trunk repositories soon, hopefully the reverse proxy cache will hold us through the reboot for the kernel update. | 21:19 |
| mordred | dmsimard: I believe in the reverse proxy cache | 21:20 |
| openstackgerrit | Merged openstack-infra/publications master: Update make-index for Zuul v3 https://review.openstack.org/516009 | 21:20 |
| dmsimard | mordred: thank you for believing :D | 21:20 |
| clarkb | fungi: http://logs.openstack.org/12/529712/3/check/legacy-infra-puppet-apply-3-ubuntu-trusty/7ad69cd/openstack-infra/system-config/applytest/puppetapplytest01 apply tested your multiline variable assignment and didn't explode so +2 from me | 21:22 |
| *** slaweq has quit IRC | 21:23 | |
| clarkb | https://review.openstack.org/#/c/523937/1 is a simple docs fix too related to storyboard and project creation | 21:24 |
| *** smcginnis has joined #openstack-infra | 21:27 | |
| *** jascott1 has quit IRC | 21:29 | |
| *** jascott1 has joined #openstack-infra | 21:29 | |
| *** slaweq_ has quit IRC | 21:29 | |
| *** jascott1 has quit IRC | 21:30 | |
| *** olaph has quit IRC | 21:30 | |
| *** jascott1 has joined #openstack-infra | 21:30 | |
| *** olaph has joined #openstack-infra | 21:31 | |
| *** slaweq has joined #openstack-infra | 21:32 | |
| mriedem | clarkb: so on https://review.openstack.org/#/c/531208/ the logs look good, | 21:33 |
| mriedem | http://logs.openstack.org/08/531208/1/check/tempest-full/80532f7/controller/logs/screen-n-api.txt.gz?level=WARNING#_Jan_04_19_18_07_552867 | 21:33 |
| mriedem | filtering, colors, all works | 21:33 |
| mriedem | BUT! | 21:33 |
| mriedem | http://logs.openstack.org/08/531208/1/check/tempest-full/80532f7/controller/logs/etc/nova/ still requires a download - but maybe that's a different fix | 21:34 |
| clarkb | mriedem: ya I think thats a separate but related issue | 21:34 |
| *** jascott1 has quit IRC | 21:34 | |
| clarkb | mriedem: separate because this fix won't fix it but related to the porting of devstack-gate and changes in how things are compressed | 21:34 |
| dmsimard | the vhost mime types handles .txt.gz but not .ini.gz and etc | 21:35 |
| *** jascott1 has joined #openstack-infra | 21:35 | |
| dmsimard | in puppet-openstackci | 21:35 |
| dmsimard | we'd need to add .conf.gz, .ini.gz and whatever else | 21:35 |
| mriedem | clarkb: ack; +1 on getting the logs fixed though, that was much more annoying than the config files | 21:35 |
| *** threestrands has joined #openstack-infra | 21:35 | |
| *** threestrands has quit IRC | 21:35 | |
| *** threestrands has joined #openstack-infra | 21:35 | |
| mriedem | mtreinish: sdague: https://review.openstack.org/#/c/531208/ | 21:36 |
| dmsimard | mriedem: http://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/templates/logs.vhost.erb#n24 | 21:36 |
| *** slaweq has quit IRC | 21:37 | |
| *** jascott1 has quit IRC | 21:37 | |
| mriedem | dmsimard: so conf and ini files need to be in that list too? | 21:37 |
| * mriedem doesn't puppet | 21:37 | |
| dmsimard | whatever extension that ought to be text/plain (when decompressed) should be there, yeah | 21:37 |
| dmsimard | there's little to no puppet in that file, it's mostly an apache vhost :) | 21:38 |
| mriedem | heh, i don't do that either | 21:38 |
| mriedem | there isn't much that i do, do | 21:38 |
| *** jascott1 has joined #openstack-infra | 21:38 | |
| dmsimard | you do nova | 21:38 |
| dmsimard | that's good enough :D | 21:38 |
| clarkb | ah ya so I think the change here from devstack-gate to logs is we used to rename the inis to .txt | 21:38 |
| clarkb | which is also another option we could do | 21:39 |
| *** dhill_ has quit IRC | 21:39 | |
| clarkb | but might be more accurate and easier to grok if we .ini and handle that as txt | 21:39 |
| dmsimard | clarkb: either we rename or we add the mimetypes | 21:39 |
| mriedem | looks like it would be at least ini, conf and filters | 21:39 |
| mtreinish | mriedem: +A | 21:39 |
| dmsimard | renaming everything has a cost | 21:39 |
| corvus | i think renaming is weird and we should stop | 21:39 |
| mriedem | can we just hit everything that's under etc/? | 21:39 |
| *** dhill_ has joined #openstack-infra | 21:39 | |
| dmsimard | corvus: +1 | 21:39 |
| *** jcoufal has quit IRC | 21:40 | |
| dmsimard | Trunk RDO repositories have been rebooted after kernel update btw | 21:41 |
| *** dhill_ has quit IRC | 21:41 | |
| *** dhill_ has joined #openstack-infra | 21:42 | |
| *** jascott1 has quit IRC | 21:42 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/openstack-zuul-jobs master: Create artifacts dir regardless of afs rpm builds https://review.openstack.org/531249 | 21:42 |
| clarkb | ok ^ is a fix to my fix for the openafs rpm builds | 21:42 |
| *** sbezverk has joined #openstack-infra | 21:43 | |
| clarkb | reviews much appreciated | 21:43 |
| *** dmellado has quit IRC | 21:43 | |
| clarkb | anyone know if the releasenote build issues with tox siblings is all working happily now? (I kind of assume it does simply because relese notes not working would be a big issue people would be complaining about) | 21:44 |
| dmsimard | btw insight on spectre/meltdown from a qemu perspective https://www.qemu.org/2018/01/04/spectre/ | 21:45 |
| *** bobh has quit IRC | 21:45 | |
| *** wolverineav has joined #openstack-infra | 21:46 | |
| *** dmellado has joined #openstack-infra | 21:46 | |
| *** dhill_ has quit IRC | 21:47 | |
| *** dhill_ has joined #openstack-infra | 21:48 | |
| EmilienM | clarkb: https://review.openstack.org/#/c/531229/ is ready for review when you have time | 21:48 |
| *** dhill_ has quit IRC | 21:49 | |
| *** slaweq_ has joined #openstack-infra | 21:50 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: Support cross-source dependencies https://review.openstack.org/530806 | 21:52 |
| *** dhill_ has joined #openstack-infra | 21:53 | |
| *** dhill_ has quit IRC | 21:54 | |
| *** esberglu has quit IRC | 21:54 | |
| *** slaweq_ has quit IRC | 21:54 | |
| dmsimard | between two kernel updates, reading on the news coming out today.. this from the NY Times: "The worldwide community of coders that oversees the open-source Linux operating system, which runs about 30 percent of computer servers worldwide, has already posted a patch for that operating system." .. only 30% ? | 21:55 |
| *** smatzek has joined #openstack-infra | 21:57 | |
| *** dhill_ has joined #openstack-infra | 21:58 | |
| fungi | dmsimard: you forget that most servers are now serverless | 21:59 |
| dmsimard | oh, is that "the cloud" ? :D | 22:01 |
| fungi | soon it will even be softwareless | 22:01 |
| clarkb | it will just be an aggregate of thought like in the matrix | 22:03 |
| fungi | sounds about right | 22:04 |
| fungi | wake up, neo | 22:04 |
| dmsimard | quick, get keanu reeves | 22:04 |
| fungi | strange things are afoot in the cloud | 22:05 |
| *** slaweq has joined #openstack-infra | 22:05 | |
| *** esberglu has joined #openstack-infra | 22:06 | |
| *** slaweq has quit IRC | 22:10 | |
| *** markmcclain has quit IRC | 22:10 | |
| clarkb | pabelanger: are you around today? Thinking it owuld be a good idea to get the centos updates working and reboots for meltdown in place as soon as possible and know you were interested in how to make haproxy do the right thing around that | 22:11 |
| *** markmcclain has joined #openstack-infra | 22:11 | |
| dmsimard | I'm almost finished patching all of RDO's stuff, I'd take a look but today was pretty rough | 22:12 |
| *** rcernin has joined #openstack-infra | 22:12 | |
| clarkb | I still don't see anything from ubuntu so I think we are in a holding pattern there | 22:13 |
| dmsimard | yeah still no update on the wiki | 22:14 |
| clarkb | dmsimard: its fairly straightforward to do the git backends but there is a process and I know pabelanger wanted to go through it before we all holidayed so wanted to give him the chance | 22:14 |
| clarkb | I'm going to go ahead and disable git08 in haproxy and use it as the first node to be updated | 22:15 |
| *** smatzek has quit IRC | 22:15 | |
| clarkb | dmsimard: you don't happen to know off the top of your head what version of th ekernel we want do you? (centos 7) | 22:18 |
| dmsimard | yeah one sec | 22:18 |
| dmsimard | kernel-3.10.0-693.11.6.el7.x86_64: Linux trunk-primary.rdoproject.org 3.10.0-693.11.6.el7.x86_64 #1 SMP Thu Jan 4 01:06:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | 22:19 |
| *** markmcclain has quit IRC | 22:19 | |
| clarkb | tyty | 22:19 |
| dmsimard | there is another previous dot release within -693 | 22:19 |
| dmsimard | like 693.2.2 for example, just need to make sure it's the .11 | 22:19 |
| clarkb | gotcha | 22:19 |
| dmsimard | gotta love rhel kernel versions | 22:20 |
| *** markmcclain has joined #openstack-infra | 22:20 | |
| clarkb | just waiting for git daemon processes to drop off then will yum update and reboot | 22:21 |
| *** aeng has joined #openstack-infra | 22:21 | |
| *** smatzek has joined #openstack-infra | 22:21 | |
| *** Goneri has quit IRC | 22:21 | |
| *** smatzek has quit IRC | 22:22 | |
| *** smatzek has joined #openstack-infra | 22:23 | |
| dmsimard | where is the haproxy for those ? | 22:23 |
| dmsimard | doesn't seem documented in https://docs.openstack.org/infra/system-config/git.html | 22:24 |
| clarkb | if anyone is wondering haproxy command socket commands are `echo "disable server balance_git_http/git08.openstack.org" | socat /var/lib/haproxy/stats stdio` `echo "disable server balance_git_https/git08.openstack.org" | socat /var/lib/haproxy/stats stdio` and `echo "disable server balance_git_daemon/git08.openstack.org" | socat /var/lib/haproxy/stats stdio` | 22:24 |
| clarkb | dmsimard: git.openstack.org | 22:24 |
| fungi | on git.o.o | 22:24 |
| dmsimard | ah, that was obvious actually | 22:24 |
| clarkb | then when done with git08 rplace disable with enable make sure its happy then go to the next one | 22:24 |
| clarkb | that won't kill existing connections which is why I have to wait forthings to settle on git08 first (but no new connections) | 22:24 |
| dmsimard | ansible has a haproxy module but never tried it http://docs.ansible.com/ansible/latest/haproxy_module.html | 22:25 |
| dmsimard | apparently supports draining | 22:26 |
| *** bobh has joined #openstack-infra | 22:26 | |
| clarkb | yum update is in progress and kernel version in yum output looked good | 22:27 |
| *** smatzek has quit IRC | 22:27 | |
| clarkb | I'll walk through all of the git backends unless someone else wants to get exposure to haproxy too. Then probably just yolo reboot git.o.o later tonight when things go quiet? | 22:28 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: Support cross-source dependencies https://review.openstack.org/530806 | 22:28 |
| clarkb | open to better suggestions re yolo reboot | 22:29 |
| *** jistr has quit IRC | 22:29 | |
| clarkb | rebooting git08 now | 22:30 |
| dmsimard | git.o.o is the backend for review.openstack.org, right ? | 22:30 |
| dmsimard | or review.openstack has it's own git repos ? | 22:30 |
| clarkb | Gerrit is the authoritative canonical source of git repos and it does its own hosting. It then mirrors to our official mirrors, git.openstack.org | 22:31 |
| clarkb | lots of things point at git.o.o to take load off of gerritthough | 22:31 |
| clarkb | Linux git08.openstack.org 3.10.0-693.11.6.el7.x86_64 #1 SMP Thu Jan 4 01:06:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | 22:32 |
| dmsimard | lgtm | 22:32 |
| clarkb | that is the same version thta dmsimard posted above. Apache is running I am adding the server back to the haproxy backends list | 22:32 |
| pabelanger | clarkb: dmsimard: sorry, just back online for a moment, my turn not to feel well, should have mentioned something a little sooner | 22:32 |
| clarkb | pabelanger: no problem just didn't want the opportunity to be wasted. If you are unwell you should work on feeling better andI'm sure we can figure out another time to haproxy later | 22:33 |
| dmsimard | clarkb: reboot takes like a minute if there's nothing silly like an overdue fsck or a selinux relabel | 22:33 |
| dmsimard | clarkb: is there a reason why git.o.o runs centos if it's just for haproxy ? | 22:34 |
| clarkb | dmsimard: we had to use centos for cgit on the backends and I think it was easier to keep everything the same for a logical service rather than mixing | 22:34 |
| pabelanger | clarkb: yah, I am hoping it is only a 24 hour thing, but I can read up on backscroll for haproxy commands after you deal with kernels | 22:34 |
| clarkb | dmsimard: however we could run it on ubuntu or other distros for sure | 22:34 |
| *** ihrachys has quit IRC | 22:34 | |
| *** ihrachys has joined #openstack-infra | 22:35 | |
| dmsimard | also, git.o.o hasn't been updated to centos 7.4 | 22:35 |
| clarkb | dmsimard: oh also it may hvae been that haproxy grew out of load demands later | 22:35 |
| dmsimard | so the update is larger than just a few packages | 22:35 |
| clarkb | dmsimard: 95 packages | 22:35 |
| clarkb | I skimmd them nothing too crazy | 22:35 |
| dmsimard | I see 387 packages on a yum update here | 22:35 |
| corvus | if we think there might be a big enough downtime, we could replace the server | 22:35 |
| clarkb | oh this was git08 at 95 packages | 22:35 |
| clarkb | git.o.o could've drifted different | 22:35 |
| dmsimard | clarkb: ah, yeah, git.o.o hasn't been updated to 7.4 and there are other CVEs fixes that landed in 7.4, we want that updated asap :) | 22:36 |
| corvus | bring up new one, test it, add cname record, watch for issues, drop old cname, delete old server | 22:36 |
| dmsimard | that sounds like a plan | 22:36 |
| clarkb | corvus: I don't expect it to be very long but unexpected fsck could certainly ruin that ssumption | 22:36 |
| clarkb | and ya its a stateless node so doing the move rather than reboot is probably simplest | 22:37 |
| clarkb | I'll continue on with the backends for now | 22:37 |
| dmsimard | I believe there's a systemd/systemctl command to make sure there's no unexpected fsck but I'm always paranoid about that kind of stuff | 22:37 |
| corvus | i have limited time to contribute, so i'm not going to object to yolo reboot :) | 22:37 |
| clarkb | 07 is now disabled, just waiting for it to settle | 22:38 |
| *** Apoorva has quit IRC | 22:38 | |
| *** bobh has quit IRC | 22:38 | |
| *** Apoorva has joined #openstack-infra | 22:39 | |
| corvus | i'm suddenly reminded there seems to be some kind of bug with the mta on those servers; we may be about to get a lot of queued mail. | 22:39 |
| *** wolverineav has quit IRC | 22:39 | |
| dmsimard | clarkb: I don't believe there'd be an unexpected fsck -- there's no check interval on the root partition | 22:40 |
| clarkb | corvus: exciting | 22:41 |
| dmsimard | http://paste.openstack.org/raw/638471/ | 22:41 |
| clarkb | we probably do want to keep an eye on git performance as I expect its quite a bit syscall nound | 22:43 |
| clarkb | *bound | 22:43 |
| clarkb | but I also don't know that we have much choice | 22:43 |
| *** slaweq has joined #openstack-infra | 22:44 | |
| *** nicolasbock has joined #openstack-infra | 22:44 | |
| dmsimard | clarkb: the kernel flag might not be enabled, you can check with /proc/cpuinfo | 22:44 |
| dmsimard | clarkb: it depends on how the hypervisor exposes the cpu, it might not be detected as a vulnerable (and thus the patch isn't applied) | 22:44 |
| clarkb | 07 rebooting now | 22:44 |
| dmsimard | if the patch is applied, you'll have a line in cpuinfo with the insecure bug thing | 22:45 |
| clarkb | dmsimard: I don't think the cpu flags determine it, its any intel cpu | 22:45 |
| clarkb | oh you mean to check it /me looks | 22:45 |
| *** jascott1 has joined #openstack-infra | 22:46 | |
| *** armax has joined #openstack-infra | 22:47 | |
| dmsimard | "bugs : cpu_insecure" is what should be shown if the patch is effective iiuc | 22:47 |
| clarkb | its not, which now begs the question why | 22:47 |
| clarkb | it thinks it has an intel cpu | 22:47 |
| clarkb | or is this the thing like qemu was saying where they need to expose things up thorugh before it knows? | 22:48 |
| dmsimard | let me look at a bare metal centos machine, sec | 22:48 |
| clarkb | 07 is up enabling it in haproxy | 22:48 |
| dmsimard | hmm, I'm not seeing that flag on a bare metal machine either | 22:50 |
| clarkb | dmsimard: maybe the backported patch to centos didn't pull that stuff in? | 22:51 |
| clarkb | or maybe we aren't actually booting with pti :) | 22:51 |
| corvus | https://www.reddit.com/r/sysadmin/comments/7o1769/go_check_your_proccpuinfo_it_will_contain_cpu/ds6xpq8/ | 22:51 |
| corvus | some person on reddit says rhat masks the flag | 22:51 |
| dmsimard | bleh | 22:51 |
| dmsimard | that kind of sucks | 22:51 |
| clarkb | ya thats unfortunate | 22:51 |
| corvus | would be nice to have better confirmation than that, but that seems like a thing worth exploring | 22:51 |
| dmsimard | there has to be a way to see if it's loaded :/ | 22:52 |
| clarkb | I'm continuing to 06, patching in general not a bad thing and until we get to git.o.o isn't impactful | 22:52 |
| dmsimard | sure | 22:52 |
| clarkb | but yes I agree we sould sort out some way of confirming we are patched | 22:52 |
| clarkb | (and not just in the package but in actively running kernel) | 22:52 |
| corvus | https://askubuntu.com/questions/992137/how-to-check-that-kpti-is-enabled-on-my-ubuntu | 22:54 |
| corvus | every one of those things fails | 22:54 |
| clarkb | corvus: even the /proc/config.gz grep? | 22:55 |
| dmsimard | red hat put out a script but it's for ovirt and it does postgresql things.. | 22:55 |
| corvus | clarkb: that path doesn't exist | 22:56 |
| clarkb | ah must be ubuntu things | 22:56 |
| corvus | clarkb: /boot/config-3.10.0-693.11.6.el7.x86_64 does but doesn't have the key | 22:56 |
| dmsimard | I have this: | 22:56 |
| dmsimard | # grep -i isolation /boot/config-3.10.0-693.11.6.el7.x86_64 | 22:56 |
| dmsimard | CONFIG_MEMORY_ISOLATION=y | 22:56 |
| *** edmondsw has quit IRC | 22:56 | |
| dmsimard | not sure if it's the same as the page table thing | 22:56 |
| clarkb | dmsimard: its not CONFIG_PAGE_TABLE_ISOLATION is what we want | 22:57 |
| *** edmondsw has joined #openstack-infra | 22:57 | |
| clarkb | unless during patching things got renamed or merged together | 22:57 |
| dmsimard | yeah that's what I'm wondering | 22:57 |
| dmsimard | this is kernel 3.10 so.. | 22:57 |
| dmsimard | could be something else entirely too | 22:58 |
| corvus | memory_isolation is an old flag | 22:58 |
| clarkb | 06 is rebooting now | 22:58 |
| dmsimard | nothing in dmesg either .. | 22:59 |
| clarkb | anyone know anyone at red hat? >_> | 23:00 |
| dmsimard | lol | 23:01 |
| *** edmondsw has quit IRC | 23:01 | |
| dmsimard | ok I'll go ahead and ask in a secret channel -_- | 23:02 |
| clarkb | I'm on to 05 now | 23:03 |
| *** smatzek has joined #openstack-infra | 23:04 | |
| * dmsimard is back from the secret channel | 23:06 | |
| dmsimard | This gives us what we want: /sys/kernel/debug/x86/pti_enabled | 23:07 |
| *** dave-mccowan has joined #openstack-infra | 23:08 | |
| clarkb | dmsimard: if the file is present that means we are good? | 23:09 |
| dmsimard | it should be present and 1 | 23:09 |
| dmsimard | (do a meow on it) | 23:10 |
| clarkb | oh ya its a 1 thanks | 23:10 |
| *** ijw has quit IRC | 23:10 | |
| clarkb | ok so this did work just not in an obvious way | 23:10 |
| dmsimard | yeah. | 23:10 |
| clarkb | that is reassuring though, thank you for figuring it out /me goe sback to patching | 23:10 |
| dmsimard | I should probably hang out in that secret channel more often, bunch of interesting discussions. | 23:11 |
| clarkb | 05 rebooting now | 23:11 |
| dmsimard | corvus: your suspicion of email spam was correct | 23:12 |
| clarkb | and now on to 04 | 23:13 |
| *** erlon has quit IRC | 23:14 | |
| *** flwang has quit IRC | 23:14 | |
| *** ijw has joined #openstack-infra | 23:15 | |
| fungi | debian has posted their advisory and corresponding kernel packages now | 23:16 |
| dmsimard | Does ubuntu get their kernels from debian ? I'm not up to date on the relationship between debian and ubuntu anymore | 23:17 |
| *** hongbin has quit IRC | 23:18 | |
| clarkb | I want to say the kernel effectively forks off when ubuntu grabs it from debian | 23:18 |
| clarkb | I'm going to go ahead and start on 03 in parallel | 23:19 |
| *** slaweq has quit IRC | 23:19 | |
| *** smatzek has quit IRC | 23:19 | |
| *** smatzek has joined #openstack-infra | 23:19 | |
| *** jistr has joined #openstack-infra | 23:19 | |
| clarkb | no usn updates | 23:21 |
| *** smatzek has quit IRC | 23:21 | |
| fungi | yeah, debian and ubuntu work on their kernel packages somewhat indepndently | 23:21 |
| *** smatzek has joined #openstack-infra | 23:21 | |
| clarkb | if they don't get them up today those of you up early in the morning should check and get ansible to apply them then we can work on rolling reboots through the day | 23:21 |
| clarkb | actually if they get them up before 0600 then automated updates will install it iirc | 23:22 |
| fungi | as clarkb indicates, i too think ubuntu inherits a kernel from debian/testing at the time the next ubuntu release is stabilized and then forks from there | 23:22 |
| fungi | like most packages in ubuntu | 23:22 |
| fungi | but they _may_ be treating the kernel specially, since ubuntu tends to enable a lot of additional bits | 23:23 |
| fungi | so not certain | 23:23 |
| *** flwang has joined #openstack-infra | 23:24 | |
| *** smatzek has quit IRC | 23:26 | |
| clarkb | well thats exceptionally curious | 23:28 |
| clarkb | git05 got 693-11.1 not 693-11.6 | 23:28 |
| clarkb | and does not show pti enabled | 23:28 |
| clarkb | it thinks it is up to date too | 23:28 |
| dmsimard | yay.. I think I just finished the last important update on our infra | 23:29 |
| dmsimard | s/our/rdo/ | 23:29 |
| clarkb | I'm going to finish up 04-01 then go back and figure out hwat is up with 05 | 23:29 |
| clarkb | unless someone wnts to look into 05 | 23:29 |
| dmsimard | I'll take a look at 05 | 23:29 |
| openstackgerrit | Stibbons proposed openstack-dev/pbr master: Support v<semver> version https://review.openstack.org/531262 | 23:30 |
| clarkb | dmsimard: maybe it is pointing at stale mirrors? | 23:30 |
| *** dave-mccowan has quit IRC | 23:31 | |
| openstackgerrit | Stibbons proposed openstack-dev/pbr master: Support v<semver> version https://review.openstack.org/531161 | 23:32 |
| clarkb | 03 and 04 are done and have pti enabled | 23:33 |
| clarkb | doing 01 and 02 now | 23:33 |
| dmsimard | clarkb: yeah, stale mirror.. doing a yum clean all and a yum update yields the update we want.. is git05 still out of haproxy ? | 23:33 |
| clarkb | dmsimard: it is not, I will remove it and let you know | 23:34 |
| dmsimard | ok | 23:34 |
| clarkb | dmsimard: it is out now | 23:34 |
| dmsimard | ok I'll reboot after the update completes | 23:34 |
| dmsimard | When ubuntu updates are out, it'd be cool to check if "/sys/kernel/debug/x86/pti_enabled" is also a thing on Ubuntu so we could have a quick ansible task that checks which machines we have updated and which are still pending | 23:35 |
| dmsimard | since I guess we still have a mix of 14.04 and 16.04 | 23:36 |
| clarkb | +1 | 23:36 |
| fungi | yup | 23:36 |
| fungi | great idea, dmsimard | 23:36 |
| fungi | though we could probably || chain the shell commands in the task | 23:37 |
| fungi | if the methods of checking between them do have to differ | 23:37 |
| dmsimard | eh, either that or a proper playbook that detects centos/14.04/16.04 | 23:37 |
| dmsimard | hopefully they're all the same | 23:37 |
| *** tosky has quit IRC | 23:39 | |
| *** mtreinish has quit IRC | 23:39 | |
| dmsimard | clarkb: git05 is up to date and pti enalbed | 23:39 |
| clarkb | dmsimard: thanks | 23:39 |
| clarkb | I will reenable it in haproxy now | 23:40 |
| clarkb | done | 23:40 |
| clarkb | still waiting on yum on 01 and 02 | 23:40 |
| clarkb | then I'll double check all 8 are patched and haproxy has them all then we can think about a yolo reboot on 08 | 23:41 |
| dmsimard | I'm going to take a break and eat some nutrients | 23:41 |
| clarkb | er not on 08, git.o.o | 23:41 |
| *** mtreinish has joined #openstack-infra | 23:42 | |
| openstackgerrit | Stibbons proposed openstack-dev/pbr master: Support v<semver> version https://review.openstack.org/531161 | 23:45 |
| clarkb | 02 and 01 reenabled in haproxy | 23:45 |
| openstackgerrit | Stibbons proposed openstack-dev/pbr master: Support v<semver> version https://review.openstack.org/531161 | 23:46 |
| *** rkukura has quit IRC | 23:46 | |
| clarkb | confirmed all 8 have pti enabled | 23:47 |
| *** rkukura has joined #openstack-infra | 23:47 | |
| *** slaweq has joined #openstack-infra | 23:49 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP: Support cross-source dependencies https://review.openstack.org/530806 | 23:49 |
| clarkb | and haproxy show stat doesn't show any in MAINT mode | 23:49 |
| clarkb | any other infra-roots around for possible yolo update and reboot of git.o.o? | 23:49 |
| clarkb | the alternative is to build a new server and update dns records. The two downsides to this are the hour long dns record ttls and potential corporate firewall rule problems | 23:50 |
| clarkb | but downside to yolo reboot is any jobs fetching git content during the couple minutes it takes to reboot will have a sad | 23:50 |
| openstackgerrit | Mike Perez proposed openstack-infra/puppet-statusbot master: Add configuration for thanks/success page url https://review.openstack.org/531185 | 23:50 |
| corvus | clarkb: have we accepted git.o.o as a service that needs special firewall rules? | 23:51 |
| corvus | i mean, it's all standard ports | 23:51 |
| corvus | clarkb: i have to run now and can't help today, but can pitch in some tomorrow | 23:52 |
| clarkb | corvus: thats a good point http(s) should all be proxied at $corp | 23:52 |
| clarkb | whereas 29418 on gerrit is special | 23:52 |
| mtreinish | clarkb: tempest also pulls the openstack logo from git.o.o for glance image create from remote location | 23:53 |
| *** slaweq has quit IRC | 23:53 | |
| mtreinish | so if any runs hitting those tests during the reboot window will fail | 23:53 |
| corvus | clarkb: it does have 9418, but heck, we've talked about dropping that altogether. i'm not sure it's worth worrying about. | 23:53 |
| clarkb | ya | 23:53 |
| *** jistr has quit IRC | 23:54 | |
| clarkb | I think I will go ahead and just build a new server | 23:54 |
| corvus | mtreinish: i like the overloading of the term image | 23:54 |
| clarkb | I'm going to update the dns ttls to 5 minute snow as step 9 | 23:54 |
| clarkb | er step 0 | 23:54 |
| *** jistr has joined #openstack-infra | 23:55 | |
| mtreinish | corvus: heh, yeah I always like that too | 23:55 |
| clarkb | its more cloudy to do it this way too | 23:55 |
| *** stakeda has joined #openstack-infra | 23:55 | |
| clarkb | dns updated, off to boot a new server now | 23:56 |
| *** nicolasbock has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!