| *** nicolasbock has quit IRC | 00:01 | |
| *** yamamoto has joined #openstack-infra | 00:02 | |
| *** rkukura has quit IRC | 00:12 | |
| *** weifan has quit IRC | 00:15 | |
| *** weifan has joined #openstack-infra | 00:16 | |
| *** weifan has quit IRC | 00:20 | |
| *** markvoelker has quit IRC | 00:24 | |
| *** _erlon_ has quit IRC | 00:29 | |
| *** Adri2000 has quit IRC | 00:40 | |
| *** mriedem has quit IRC | 00:44 | |
| *** Adri2000 has joined #openstack-infra | 00:45 | |
| *** bobh has joined #openstack-infra | 00:58 | |
| *** yamamoto has quit IRC | 01:03 | |
| *** lseki has quit IRC | 01:10 | |
| *** zhurong has joined #openstack-infra | 01:12 | |
| *** hwoarang has quit IRC | 01:17 | |
| *** gyee has quit IRC | 01:19 | |
| ianw | we should probably change the "source" link in the topic | 01:19 |
|---|---|---|
| *** hwoarang has joined #openstack-infra | 01:22 | |
| *** jamesmcarthur has quit IRC | 01:22 | |
| *** jamesmcarthur has joined #openstack-infra | 01:31 | |
| *** jamesmcarthur has quit IRC | 01:43 | |
| *** jamesmcarthur has joined #openstack-infra | 01:44 | |
| *** michael-beaver has quit IRC | 01:48 | |
| *** bobh has quit IRC | 01:54 | |
| *** jamesmcarthur has quit IRC | 02:00 | |
| *** jamesmcarthur has joined #openstack-infra | 02:00 | |
| *** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://opendev.org/opendev/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/" | 02:01 | |
| fungi | ianw: how's that? | 02:01 |
| *** Goneri has quit IRC | 02:05 | |
| *** rlandy has quit IRC | 02:12 | |
| *** hongbin has joined #openstack-infra | 02:17 | |
| *** jamesmcarthur has quit IRC | 02:39 | |
| *** jamesmcarthur has joined #openstack-infra | 02:40 | |
| *** jamesmcarthur has quit IRC | 02:43 | |
| *** jamesmcarthur has joined #openstack-infra | 02:43 | |
| *** yamamoto has joined #openstack-infra | 02:51 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: docs: add cleanup-run documentation https://review.opendev.org/662147 | 02:55 |
| *** whoami-rajat has joined #openstack-infra | 03:14 | |
| *** happyhemant has quit IRC | 03:20 | |
| *** gregoryo has joined #openstack-infra | 03:36 | |
| *** igordc has quit IRC | 03:39 | |
| *** jamesmcarthur has quit IRC | 03:45 | |
| *** jamesmcarthur has joined #openstack-infra | 03:46 | |
| *** jamesmcarthur has quit IRC | 03:49 | |
| *** jamesmcarthur_ has joined #openstack-infra | 03:49 | |
| *** hwoarang has quit IRC | 03:59 | |
| *** hwoarang has joined #openstack-infra | 04:01 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Parallelize github event processing https://review.opendev.org/662818 | 04:03 |
| *** ramishra has joined #openstack-infra | 04:05 | |
| *** igordc has joined #openstack-infra | 04:06 | |
| *** hongbin has quit IRC | 04:17 | |
| *** haleyb has quit IRC | 04:22 | |
| *** armax has quit IRC | 04:22 | |
| *** udesale has joined #openstack-infra | 04:25 | |
| *** cyberpear has joined #openstack-infra | 04:51 | |
| openstackgerrit | Merged zuul/zuul master: git: only list heads and tags references https://review.opendev.org/661185 | 05:11 |
| *** pcaruana has joined #openstack-infra | 05:12 | |
| *** pcaruana has quit IRC | 05:23 | |
| *** pcaruana has joined #openstack-infra | 05:28 | |
| *** ccamacho has quit IRC | 05:28 | |
| *** hwoarang has quit IRC | 05:28 | |
| *** raukadah is now known as chandankumar | 05:29 | |
| *** jamesmcarthur_ has quit IRC | 05:29 | |
| *** hwoarang has joined #openstack-infra | 05:30 | |
| *** ykarel|away has joined #openstack-infra | 05:31 | |
| *** ykarel|away is now known as ykarel | 05:34 | |
| *** e0ne has joined #openstack-infra | 05:43 | |
| *** jtomasek has joined #openstack-infra | 05:44 | |
| *** igordc has quit IRC | 05:51 | |
| *** kjackal has quit IRC | 05:52 | |
| *** kopecmartin|off is now known as kopecmartin | 06:07 | |
| *** lpetrut has joined #openstack-infra | 06:10 | |
| *** slaweq has joined #openstack-infra | 06:13 | |
| *** dpawlik has joined #openstack-infra | 06:13 | |
| openstackgerrit | caoyuan proposed openstack/os-testr master: Update hacking version https://review.opendev.org/641384 | 06:14 |
| evrardjp_ | corvus: sorry I went to bed early. I suppose this is what you prefer? https://review.opendev.org/#/c/662817/1 | 06:14 |
| evrardjp_ | technically gather_facts are done implicitly, unless disabled by the conf/env | 06:15 |
| evrardjp_ | which I am not sure this patch would fix things | 06:15 |
| *** pgaxatte has joined #openstack-infra | 06:15 | |
| *** ykarel has quit IRC | 06:16 | |
| *** dtantsur|afk is now known as dtantsur | 06:16 | |
| *** ykarel has joined #openstack-infra | 06:17 | |
| *** kjackal has joined #openstack-infra | 06:24 | |
| *** gfidente|afk has quit IRC | 06:32 | |
| *** jbadiapa has joined #openstack-infra | 06:41 | |
| *** yamamoto has quit IRC | 06:41 | |
| *** evrardjp_ is now known as evrardjp | 06:43 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] backups https://review.opendev.org/662657 | 06:44 |
| *** ykarel is now known as ykarel|afk | 06:48 | |
| *** kjackal_v2 has joined #openstack-infra | 06:51 | |
| *** kjackal has quit IRC | 06:51 | |
| ianw | AJaeger: if you get a minute -- https://review.opendev.org/#/c/662670 -- ever seen this doubling of TOC entries due to (seemingly) a sub-TOC on another page? | 06:54 |
| AJaeger | ianw: really strange | 06:57 |
| AJaeger | maybe stephenfin has an idea ^ | 06:57 |
| *** ianychoi has quit IRC | 06:57 | |
| *** ianychoi has joined #openstack-infra | 06:58 | |
| *** yamamoto has joined #openstack-infra | 06:59 | |
| *** ccamacho has joined #openstack-infra | 07:04 | |
| *** ccamacho has quit IRC | 07:04 | |
| *** ccamacho has joined #openstack-infra | 07:04 | |
| *** iurygregory has quit IRC | 07:06 | |
| evrardjp | pabelanger: are you there by any chance? | 07:08 |
| evrardjp | (it's to talk about the gather_facts which didn't happen implicitly in promote pipeline) | 07:09 |
| *** e0ne has quit IRC | 07:09 | |
| *** ginopc has joined #openstack-infra | 07:14 | |
| *** gregoryo has quit IRC | 07:15 | |
| *** rpittau|afk is now known as rpittau | 07:15 | |
| *** slaweq has quit IRC | 07:16 | |
| *** rossella_s has joined #openstack-infra | 07:17 | |
| *** ykarel|afk is now known as ykarel | 07:18 | |
| *** yamamoto has quit IRC | 07:20 | |
| *** yamamoto has joined #openstack-infra | 07:21 | |
| *** gfidente has joined #openstack-infra | 07:23 | |
| lpetrut | Hi. We were thinking about renaming the #openstack-hyper-v channel to #openstack-windows. We think that its scope is too narrow at the moment, being more or less tied to Hyper-V/Nova discussions. There are people interested in deploying other Openstack services on Windows as well (such as Cinder, Glance, Neutron, etc) or running Windows guests (on top of KVM as well). We think this would better cover the scope of the "winstackers" team. | 07:23 |
| lpetrut | Looking at this http://codesearch.openstack.org/?q=openstack-hyper-v, I guess we'd only have to update /project-config, /governance and /system-config, right? Does anyone know how can we enable IRC logs? The current channel is not logged at the moment. | 07:23 |
| *** ginopc has quit IRC | 07:23 | |
| *** ginopc has joined #openstack-infra | 07:23 | |
| *** tkajinam has quit IRC | 07:24 | |
| *** tkajinam has joined #openstack-infra | 07:24 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/nodepool master: nodepool: improve static driver request handling for multi labels https://review.opendev.org/662954 | 07:25 |
| AJaeger | lpetrut: http://docs.openstack.org/infra/system-config/irc.html should cover all of your questions | 07:26 |
| lpetrut | that's great, thanks for the link | 07:26 |
| *** adriant has quit IRC | 07:29 | |
| *** yamamoto has quit IRC | 07:30 | |
| *** zbr has joined #openstack-infra | 07:31 | |
| *** iokiwi has quit IRC | 07:31 | |
| *** markvoelker has joined #openstack-infra | 07:32 | |
| *** yamamoto has joined #openstack-infra | 07:33 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Evaluate CODEOWNERS settings during canMerge check https://review.opendev.org/644557 | 07:33 |
| *** yamamoto has quit IRC | 07:39 | |
| *** slaweq has joined #openstack-infra | 07:40 | |
| *** yamamoto has joined #openstack-infra | 07:42 | |
| *** jpich has joined #openstack-infra | 07:47 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] backups https://review.opendev.org/662657 | 07:48 |
| *** pkopec has joined #openstack-infra | 07:49 | |
| *** jpena|off is now known as jpena | 07:51 | |
| *** iurygregory has joined #openstack-infra | 07:52 | |
| *** xek has joined #openstack-infra | 07:52 | |
| *** tosky has joined #openstack-infra | 07:58 | |
| *** markvoelker has quit IRC | 08:05 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Evaluate CODEOWNERS settings during canMerge check https://review.opendev.org/644557 | 08:08 |
| *** lucasagomes has joined #openstack-infra | 08:12 | |
| *** dchen has quit IRC | 08:17 | |
| *** iokiwi has joined #openstack-infra | 08:18 | |
| *** adriant has joined #openstack-infra | 08:18 | |
| *** factor has joined #openstack-infra | 08:18 | |
| *** jaosorior has joined #openstack-infra | 08:21 | |
| slaweq | hi infra-root | 08:21 |
| slaweq | can You take a look at https://review.opendev.org/#/c/643733/ and tell me what You think about it? thx in advance | 08:21 |
| *** priteau has joined #openstack-infra | 08:23 | |
| *** tkajinam has quit IRC | 08:24 | |
| *** hrw has left #openstack-infra | 08:31 | |
| *** e0ne has joined #openstack-infra | 08:32 | |
| *** ravikumark815 has joined #openstack-infra | 08:36 | |
| *** hwoarang has quit IRC | 08:37 | |
| *** janki has joined #openstack-infra | 08:39 | |
| *** ykarel is now known as ykarel|lunch | 08:40 | |
| *** ravikumark815 has left #openstack-infra | 08:40 | |
| *** iurygregory is now known as iurygregory_emea | 08:40 | |
| *** iurygregory_emea is now known as iury_emeahackfes | 08:40 | |
| *** hwoarang has joined #openstack-infra | 08:43 | |
| *** derekh has joined #openstack-infra | 08:44 | |
| *** pkopec has quit IRC | 08:48 | |
| *** spsurya has joined #openstack-infra | 08:51 | |
| *** lennyb has quit IRC | 08:53 | |
| *** tesseract has joined #openstack-infra | 08:56 | |
| *** factor has quit IRC | 09:06 | |
| *** factor has joined #openstack-infra | 09:06 | |
| *** rcernin has quit IRC | 09:07 | |
| jamespage | morning | 09:07 |
| *** factor has quit IRC | 09:07 | |
| *** factor has joined #openstack-infra | 09:07 | |
| jamespage | can gerrit/zuul deal with circular depends? I have a set of neutron reviews that need an associated update into neutron-tempest-plugin, but -tempest-plugin also needs the neutron reviews across multiple stable series to pass check/gate | 09:08 |
| *** factor has quit IRC | 09:08 | |
| *** factor has joined #openstack-infra | 09:09 | |
| *** factor has quit IRC | 09:10 | |
| *** ykarel|lunch is now known as ykarel | 09:10 | |
| *** factor has joined #openstack-infra | 09:10 | |
| *** factor has quit IRC | 09:11 | |
| *** factor has joined #openstack-infra | 09:12 | |
| *** pkopec has joined #openstack-infra | 09:12 | |
| *** lennyb has joined #openstack-infra | 09:12 | |
| slaweq | jamespage: hi, I don't think it will work like that | 09:12 |
| jamespage | no that's kinda where I had got to - I think I'll need to temporarily skip the tempest test and use a Depends-On from the reviews in neutron | 09:13 |
| jamespage | and then re-enable after all of the neutron reviews have landed | 09:13 |
| slaweq | jamespage: exactly, I was just going to write the same :) | 09:13 |
| *** factor has quit IRC | 09:15 | |
| *** rpittau is now known as rpittau|reboot | 09:17 | |
| jamespage | slaweq: thanks for confirming - I'll rework on that basis | 09:18 |
| *** rpittau|reboot is now known as rpittau | 09:21 | |
| openstackgerrit | Merged zuul/zuul master: test_v3: replace while loop with iterate_timeout https://review.opendev.org/662112 | 09:28 |
| *** panda is now known as panda|ruck | 09:38 | |
| *** tesseract has quit IRC | 09:40 | |
| *** jaosorior has quit IRC | 09:41 | |
| *** owalsh_ has joined #openstack-infra | 09:48 | |
| *** owalsh has quit IRC | 09:49 | |
| *** mugsie_ is now known as mugsie | 09:52 | |
| *** hwoarang has quit IRC | 09:54 | |
| *** ociuhandu has joined #openstack-infra | 09:54 | |
| *** hwoarang has joined #openstack-infra | 09:55 | |
| *** kjackal_v2 has quit IRC | 10:00 | |
| *** markvoelker has joined #openstack-infra | 10:02 | |
| *** happyhemant has joined #openstack-infra | 10:13 | |
| *** kjackal has joined #openstack-infra | 10:19 | |
| *** owalsh has joined #openstack-infra | 10:19 | |
| *** owalsh_ has quit IRC | 10:22 | |
| *** slaweq has quit IRC | 10:30 | |
| *** markvoelker has quit IRC | 10:36 | |
| *** yamamoto has quit IRC | 10:36 | |
| *** yamamoto has joined #openstack-infra | 10:37 | |
| *** yamamoto has quit IRC | 10:38 | |
| *** nicolasbock has joined #openstack-infra | 10:39 | |
| *** yoctozepto_ is now known as yoctozepto | 10:42 | |
| *** nicolasbock has quit IRC | 10:47 | |
| *** iury_emeahackfes is now known as iurygregory | 10:56 | |
| *** slaweq has joined #openstack-infra | 11:01 | |
| *** beagles has quit IRC | 11:03 | |
| *** hwoarang has quit IRC | 11:06 | |
| *** jaosorior has joined #openstack-infra | 11:07 | |
| *** hwoarang has joined #openstack-infra | 11:08 | |
| *** Goneri has joined #openstack-infra | 11:10 | |
| *** dpawlik has quit IRC | 11:12 | |
| *** jcoufal has joined #openstack-infra | 11:16 | |
| *** yamamoto has joined #openstack-infra | 11:21 | |
| *** janki has quit IRC | 11:30 | |
| *** markvoelker has joined #openstack-infra | 11:33 | |
| *** jpena is now known as jpena|lunch | 11:34 | |
| *** rh-jelabarre has joined #openstack-infra | 11:42 | |
| *** dpawlik has joined #openstack-infra | 11:43 | |
| *** yolanda has quit IRC | 11:44 | |
| *** panda|ruck is now known as panda|ruck|eat | 11:45 | |
| *** ccamacho has quit IRC | 11:45 | |
| *** dchen has joined #openstack-infra | 11:46 | |
| *** panda|ruck|eat is now known as panda|ruck | 11:51 | |
| *** udesale has quit IRC | 11:52 | |
| *** udesale has joined #openstack-infra | 11:52 | |
| *** yamamoto has quit IRC | 11:58 | |
| *** yamamoto has joined #openstack-infra | 12:00 | |
| *** jaosorior has quit IRC | 12:00 | |
| *** b3nt_pin has joined #openstack-infra | 12:02 | |
| *** markvoelker has quit IRC | 12:05 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Mount tmpfs on ansible tmp dir https://review.opendev.org/663015 | 12:10 |
| *** yamamoto has quit IRC | 12:13 | |
| *** mugsie is now known as mugsie_ | 12:18 | |
| *** mugsie_ is now known as mugsie | 12:18 | |
| *** rlandy has joined #openstack-infra | 12:30 | |
| *** dchen has quit IRC | 12:31 | |
| *** jpena|lunch is now known as jpena | 12:32 | |
| *** priteau has quit IRC | 12:34 | |
| *** panda|ruck is now known as panda|ruck|eat | 12:37 | |
| *** Goneri has quit IRC | 12:37 | |
| *** spsurya has quit IRC | 12:40 | |
| *** rlandy has quit IRC | 12:49 | |
| *** tdasilva has joined #openstack-infra | 12:50 | |
| *** aakarsh has quit IRC | 12:50 | |
| *** yamamoto has joined #openstack-infra | 12:51 | |
| *** panda|ruck|eat is now known as panda|ruck | 12:51 | |
| *** priteau has joined #openstack-infra | 12:57 | |
| *** tdasilva has quit IRC | 12:58 | |
| *** yamamoto has quit IRC | 12:59 | |
| *** yamamoto has joined #openstack-infra | 12:59 | |
| *** jamesmcarthur has joined #openstack-infra | 13:00 | |
| *** rfarr has joined #openstack-infra | 13:01 | |
| *** ekultails has joined #openstack-infra | 13:03 | |
| *** tdasilva has joined #openstack-infra | 13:03 | |
| *** priteau has quit IRC | 13:04 | |
| smcginnis | lpetrut: +1 on the rename. | 13:09 |
| *** rlandy has joined #openstack-infra | 13:17 | |
| *** mriedem has joined #openstack-infra | 13:17 | |
| *** Goneri has joined #openstack-infra | 13:18 | |
| *** diablo_rojo has joined #openstack-infra | 13:18 | |
| *** b3nt_pin is now known as beagles | 13:19 | |
| *** jamesmcarthur has quit IRC | 13:20 | |
| *** aakarsh has joined #openstack-infra | 13:20 | |
| *** jamesmcarthur has joined #openstack-infra | 13:20 | |
| *** hwoarang has quit IRC | 13:21 | |
| *** diablo_rojo has quit IRC | 13:22 | |
| *** diablo_rojo has joined #openstack-infra | 13:23 | |
| *** jamesmcarthur has quit IRC | 13:25 | |
| *** hwoarang has joined #openstack-infra | 13:27 | |
| *** rlandy_ has joined #openstack-infra | 13:29 | |
| *** rlandy has quit IRC | 13:30 | |
| *** aaronsheffield has joined #openstack-infra | 13:30 | |
| clarkb | jamespage: slaweq yup no cycles | 13:32 |
| jamespage | clarkb: ack | 13:33 |
| openstackgerrit | Merged opendev/irc-meetings master: Hold kolla meetings in #openstack-kolla https://review.opendev.org/662801 | 13:33 |
| openstackgerrit | Merged opendev/irc-meetings master: Change Placement meeting to Office Hour https://review.opendev.org/662814 | 13:33 |
| *** rlandy_ is now known as rlandy | 13:34 | |
| *** lpetrut has quit IRC | 13:43 | |
| *** liuyulong has joined #openstack-infra | 13:53 | |
| slaweq | clarkb: hi, can You take a look at https://review.opendev.org/#/c/643733/ and tell me what You think about it maybe? | 13:54 |
| clarkb | slaweq: I would look at how devstack does it. It serializes it in journald's export format which allows you yo use journalct to query it once converted back to native format | 13:57 |
| clarkb | also I found xz compresses journals much better than gz if you want to keep the size down | 13:57 |
| clarkb | its a little more work to consume it but way more flexible | 13:58 |
| slaweq | clarkb: so basically it is already in devstack.journal.xz.gz file, but in binary format | 14:00 |
| slaweq | so it's harder to check as You need to download it and use journalctl tool to check it | 14:00 |
| *** eharney has joined #openstack-infra | 14:00 | |
| clarkb | yes but then you can query specific units and time ranges and so on | 14:00 |
| *** amorin has joined #openstack-infra | 14:01 | |
| clarkb | if you want logs for tthose specific services in human readable format I would dump just those units logs maybe | 14:01 |
| *** lseki has joined #openstack-infra | 14:01 | |
| *** haleyb has joined #openstack-infra | 14:03 | |
| *** rpittau is now known as rpittau|afk | 14:05 | |
| *** jcoufal_ has joined #openstack-infra | 14:09 | |
| *** jcoufal has quit IRC | 14:13 | |
| *** mriedem is now known as mriedem_away | 14:14 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: [DNM] admin REST API: docker-compose PoC, frontend https://review.opendev.org/643536 | 14:17 |
| *** zbr has quit IRC | 14:21 | |
| *** zbr has joined #openstack-infra | 14:22 | |
| *** pkopec has quit IRC | 14:27 | |
| *** pkopec has joined #openstack-infra | 14:27 | |
| *** ykarel is now known as ykarel|away | 14:30 | |
| *** jcoufal_ has quit IRC | 14:31 | |
| evrardjp | pabelanger: could you ping me when you're here? I need your infinite ansible + zuul knowledge | 14:31 |
| zbr | is anyone aware of some rax mirroring issues because I keep get failures. Also http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22Mirror%20sync%20in%20progress%5C%22 looks interesting | 14:31 |
| clarkb | zbr: can you link to an example? there were problems yesterday with a specific mirro running out of disk in its cache | 14:33 |
| zbr | clarkb: sure, http://logs.openstack.org/54/650054/10/check/openstack-ansible-deploy-aio_metal-ubuntu-bionic/23239f2/job-output.txt.gz#_2019-06-04_12_44_05_775592 | 14:33 |
| clarkb | ok that is the mirror | 14:33 |
| clarkb | I wonder if the cache was corrupted by the running out of disk | 14:33 |
| clarkb | hrm but that is from the afs side which is not using the apache cache | 14:34 |
| clarkb | likely a different issue then | 14:34 |
| *** armax has joined #openstack-infra | 14:35 | |
| clarkb | 906361 is the size in bytes that I get for that file now | 14:35 |
| clarkb | and seems to be a valid gzip file | 14:35 |
| clarkb | and that was ~2 hours ago, long enough that the afs ubuntu mirror could've updated since | 14:36 |
| * clarkb looks at logstash query | 14:36 | |
| zbr | clarkb: incomplete rsyncs? clearly this happened for days, is not a one-off. | 14:36 |
| fungi | the ubuntu packages aren't mirrored vai rsync | 14:37 |
| clarkb | right we should always have a valid mirror due to reprepro which makes this odd | 14:37 |
| clarkb | is it possible you aren't doing an apt-get update first and this is a sync issue due to that? | 14:37 |
| fungi | they're mirrored with a tool called reprepro which fetches new packages, builds new indices, deletes old packages and then we vos release the volume it was updating to atomically update the content in all caches | 14:37 |
| fungi | also we delay the package deletions to the next run actually, in case you're working from an older index | 14:38 |
| clarkb | happens on a variety of jobs some of which I'm fairly certain do an update first (the tox with bindep jobs) | 14:38 |
| fungi | so the order of operations is actually delete old packages left behind in the previous run, fetch new packages, build indices, vos release | 14:39 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Return python artifact records to Zuul https://review.opendev.org/663053 | 14:39 |
| clarkb | oh that happens during apt-get update, weird | 14:39 |
| clarkb | also always seems to be bionic-updates | 14:39 |
| fungi | also if the packages are incomplete/corrupt, reprepro should fail in which case we won't vos release and so the broken mirror content doesn't get published | 14:40 |
| fungi | which makes this error all the more strange | 14:40 |
| clarkb | is it possible the afs cache has cached bad data? | 14:40 |
| fungi | clarkb: are all the failures for the same file? | 14:41 |
| fungi | and do they all report the same effective and expected size? | 14:41 |
| clarkb | fungi: no some are for bionic-updates/main and others bionic-updates/universe. All on Packages.gz for the respective repo though | 14:42 |
| fungi | and all are for mirror.dfw.rax? | 14:42 |
| clarkb | yes they all seem to be from that mirror | 14:42 |
| fungi | when does it seem to have started? | 14:43 |
| clarkb | fwiw I confirm that the size for bionic-updates/universe is 1290787 if fetched here | 14:43 |
| clarkb | and seems to be a valid gzip file | 14:43 |
| clarkb | fungi: about 24 hours ago | 14:43 |
| clarkb | little over that ~25-26? | 14:44 |
| clarkb | could be related to the lvm/ext4 updates maybe? | 14:44 |
| clarkb | is there a safe way to clear out afs cache? | 14:44 |
| clarkb | corvus: ^ | 14:44 |
| fungi | fs flush or fs flushvolume i think? | 14:45 |
| fungi | was looking into cache coherency docs for afs just now | 14:45 |
| clarkb | fs flush -path | 14:45 |
| clarkb | I think we should try that next | 14:46 |
| corvus | that looks promising. haven't used it myself. | 14:46 |
| corvus | and if we get another error, fs flushvolume. | 14:46 |
| pabelanger | evrardjp: sure, have a few mins | 14:46 |
| evrardjp | woot | 14:46 |
| fungi | this isn't the rebuilt mirror, is it? still has openstack.org in the instance name and looks like it was built in january of last year | 14:46 |
| evrardjp | pabelanger: so. let me show you a patch: | 14:47 |
| clarkb | fungi: this is the new mirror | 14:47 |
| fungi | weirf | 14:47 |
| fungi | er, weird | 14:47 |
| clarkb | fungi: http://logs.openstack.org/37/663037/2/check/openstack-tox-pep8/2aae794/job-output.txt#_2019-06-04_14_30_48_500082 says opendev.org. Its the one corvus did fs updates on yesterday | 14:47 |
| evrardjp | pabelanger: I wanted to achieve this: https://review.opendev.org/#/c/659578/ | 14:47 |
| clarkb | I'm going to run `fs flush -path /var/cache/openafs` | 14:47 |
| clarkb | on the mirror | 14:47 |
| *** yamamoto has quit IRC | 14:48 | |
| fungi | oh, right, i'm logging into mirror.dfw.rax.openSTACK.org and it's not a cname to the replacement | 14:48 |
| evrardjp | pabelanger: as you can see, the build/upload worked fine with the right date. | 14:48 |
| evrardjp | The promote failed because ansible_date_time isn't defined, because facts aren't gathered on the promote pipeline | 14:48 |
| clarkb | fs: Error flushing file /var/cache/openafs: Invalid argument | 14:48 |
| fungi | looks like the kernel started reporting apache2 segfaults yesterday at 11:16:21z | 14:49 |
| pabelanger | evrardjp: Hmm, this looks to because it ran on zuul-executor, which we don't collect facts by default (to avoid leaking data) | 14:49 |
| *** owalsh has quit IRC | 14:49 | |
| evrardjp | pabelanger: we discussed this with corvus yesterday | 14:49 |
| corvus | clarkb: i think you want the afs path of the file | 14:49 |
| clarkb | I think the path is meant to be a /afs path | 14:49 |
| clarkb | corvus: ya trying that next :) | 14:49 |
| fungi | ahh, the segfaults are likely related to the full proxy cache, so that's expected | 14:49 |
| evrardjp | and we came with either the approach to change the promote docker image role, or to gather_facts for making that happen | 14:50 |
| fungi | nothing in dmesg since the ext4 resize | 14:50 |
| clarkb | `fs flush -path /afs/openstack.org/mirror/ubuntu` did not error | 14:50 |
| evrardjp | see also: https://review.opendev.org/#/c/662817/1/playbooks/docker-image/promote.yaml or https://review.opendev.org/#/c/662828/ | 14:50 |
| clarkb | bionic updates for universe packages.gz do not differ before and after the flush on that path | 14:51 |
| pabelanger | evrardjp: I haven't been too involved with the new docker build image process, but trying to see why we allow an upload from test node, but do it on executor for promote pipeline. I guess one uses a password and other doesn't | 14:51 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Return javascript content artifact records to Zuul https://review.opendev.org/663056 | 14:51 |
| clarkb | maybe it was the index itself that reported the wrong size though? | 14:51 |
| evrardjp | as you can see, the latest isn't really favoured by corvus, and I kinda join him there (let's omit the race condition, so it's not really a big deal for me, and I knew that) | 14:51 |
| clarkb | in which case it may still be happy after the flush | 14:51 |
| pabelanger | evrardjp: also, I am not sure gather_facts: true will actually collect anything, because we prime fact cache with empty jsonfile | 14:52 |
| evrardjp | pabelanger: well the upload really happens in the build/upload | 14:52 |
| pabelanger | so, that might be a noop | 14:52 |
| evrardjp | the promote only fetches the manifest of previous upload, and ensure things are well tagged (basically ensures things are running after things have merged succesfully) | 14:52 |
| evrardjp | pabelanger: exactly! | 14:52 |
| evrardjp | which is exactly where you step in :D | 14:53 |
| pabelanger | evrardjp: yah, so my first question is why didn't we do that on nodepool node? if we did upload there | 14:53 |
| *** yamamoto has joined #openstack-infra | 14:53 | |
| evrardjp | that's a lot of consumption of resources for just a curl? | 14:53 |
| *** yamamoto has quit IRC | 14:53 | |
| pabelanger | yah, that could be it | 14:53 |
| pabelanger | so, if we forced a fact refresh, are we okay with that | 14:54 |
| *** ykarel_ has joined #openstack-infra | 14:54 | |
| *** yamamoto has joined #openstack-infra | 14:54 | |
| corvus | promote is meant to be instantaneous and run on the executors. | 14:54 |
| pabelanger | that is my first question here | 14:54 |
| fungi | clarkb: was it still broken a few minutes ago? looks like the most recent vos release for that volume was 2019-06-04T14:49:33z | 14:54 |
| fungi | so, like, 5 minutes ago | 14:54 |
| *** yamamoto has quit IRC | 14:55 | |
| evrardjp | pabelanger: what do you mean? | 14:55 |
| *** yamamoto has joined #openstack-infra | 14:55 | |
| *** yamamoto has quit IRC | 14:55 | |
| clarkb | fungi: logstash says it was still erroring at 14:50 | 14:55 |
| clarkb | I'm now fs flushing specific files to see if doing it on a path doesn't fully recurse | 14:56 |
| evrardjp | pabelanger: so here my concern is that by default if ansible doesn't get explicitly asked to gather_facts, it must do it automatically, so my patch is basically useless. I am just wondering what I should do to make this happen | 14:56 |
| pabelanger | evrardjp: there is a question about if we want to gather facts at all on an executor, because we don't today to reduce the info we leak about them | 14:56 |
| *** owalsh has joined #openstack-infra | 14:56 | |
| *** ykarel|away has quit IRC | 14:56 | |
| evrardjp | I see, so you mean partially leak info | 14:56 |
| clarkb | fungi: any idea where apt gets the other byte size count from? | 14:56 |
| pabelanger | setup: filter='ansible_date_time' should just return the specific info | 14:57 |
| *** lpetrut has joined #openstack-infra | 14:57 | |
| evrardjp | where is the code that makes it not gather facts or makes the fact gathered an empty json? | 14:57 |
| pabelanger | but I am not sure if that will do anything because of the fact cache | 14:57 |
| corvus | pabelanger: i don't want to change zuul's default, but maybe it would be okay to explicitly gather facts -- if that's possible. | 14:57 |
| pabelanger | evrardjp: 1 sec | 14:57 |
| corvus | if that setup line works, that would be ideal :) | 14:57 |
| evrardjp | setup is technically forcing the refresh of the fact cache ? | 14:57 |
| pabelanger | corvus: yah, I think we need to test | 14:57 |
| evrardjp | sorry, I meant... setup is technically forcing the refresh of the fact cache | 14:58 |
| evrardjp | so that would make sense | 14:58 |
| clarkb | fungi: if the error rate decreases now after my fs flushes on the specific paths I think we should flush volume to ensure the entire volume is happy | 14:58 |
| pabelanger | evrardjp: https://opendev.org/zuul/zuul/src/branch/master/zuul/executor/server.py#L405 | 14:58 |
| clarkb | need a few minutes to look at logstash data to know though | 14:58 |
| fungi | clarkb: http://mirror.dfw.rax.opendev.org/ubuntu/dists/bionic-updates/Release | 14:58 |
| *** jcoufal has joined #openstack-infra | 14:58 | |
| pabelanger | evrardjp: so, you should be able to try to reproduce locally but setting up fact cache like that for localhost | 14:59 |
| pabelanger | then see if setup task works | 14:59 |
| evrardjp | I am doing that | 14:59 |
| evrardjp | it works | 14:59 |
| clarkb | fungi: ok so lets flush taht file too I guess | 14:59 |
| evrardjp | woot nice | 14:59 |
| pabelanger | Yay | 14:59 |
| evrardjp | I mean | 14:59 |
| corvus | fungi, clarkb: may as well flush the volume if you're planning on doing it anyway :) | 14:59 |
| evrardjp | I tried locally with my non-restricted environment that I can get that info | 14:59 |
| evrardjp | not in zuul's env | 14:59 |
| clarkb | corvus: k | 15:00 |
| *** chandankumar is now known as raukadah | 15:00 | |
| pabelanger | evrardjp: k, next would be to setup fact cache way we did in zuul | 15:01 |
| evrardjp | I am doing this right now :) | 15:01 |
| evrardjp | step by step :) | 15:01 |
| clarkb | `fs flushvolume -path /afs/openstack.org/mirror/ubuntu` just run | 15:01 |
| clarkb | (timestamps in irc should help cross check against logstash) | 15:01 |
| pabelanger | evrardjp: I have to run to next meeting | 15:02 |
| pabelanger | will review in a bit | 15:02 |
| evrardjp | yeah, will reping you. I have a meeting too, thanks for the timecheck there ;) | 15:02 |
| clarkb | early logstash returns show that may have been the fix | 15:03 |
| fungi | clarkb: the error zbr linked was for main/binary-amd64/Packages.gz... i can wget it and its sha256sum matches what's in the release file | 15:03 |
| clarkb | fungi: ya logstash says it stopped around 14:56 wheich is when I starte fs flushing those paths | 15:03 |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Add jobs to publish artifacts to tarballs.opendev.org https://review.opendev.org/662877 | 15:03 |
| clarkb | fungi: should wait a bit to make sure that there aren't any stragglers or slow indexing | 15:03 |
| clarkb | but so far is looking good | 15:03 |
| clarkb | zbr: thank you for pointing that out. I think there is a good chance that the fs work that was done to fix the apache cache impacted the afs cache in an unexpected way | 15:05 |
| *** dpawlik has quit IRC | 15:05 | |
| corvus | clarkb: any ideas how? | 15:06 |
| clarkb | corvus: not really :/ weird interaction between block device and fs layers in kernel? | 15:06 |
| clarkb | I guess we can see if dmesg has any complaints | 15:07 |
| corvus | that was just lvextend+resize2fs. i would be very surprised. | 15:07 |
| clarkb | nothing from dmesg | 15:07 |
| clarkb | if one of the caches were also shared with / and that filled up I would suspect something in state somewhere on disk there but we had separate mount points for each cache | 15:08 |
| *** michael-beaver has joined #openstack-infra | 15:08 | |
| fungi | actually, the time it started would suggest it coincided with the apache segfaults in dmesg, a few hours before the resize | 15:08 |
| fungi | volume/fs resize happened around 14:20z | 15:08 |
| fungi | apache segfaults started at 11:16z | 15:09 |
| clarkb | could apache segfaulting in the middle of a read maybe cause afs cache to half populate? | 15:09 |
| fungi | could it be there's nothing wrong with afs but apache is unhappy and needs a restart? | 15:09 |
| clarkb | fungi: I haven't touched apache, but the problem seems to have stopped after flushing the afs cache | 15:09 |
| clarkb | ~13 minutes now without a hit | 15:10 |
| fungi | well, if we see it recur i guess we can investigate that route | 15:10 |
| fungi | what logstash query were you using? | 15:10 |
| clarkb | syscalls do happen within the context of a running process so it is possible I suppose that running process segfaulting during afs syscalls could have impacted it ? | 15:11 |
| *** jamesmcarthur has joined #openstack-infra | 15:11 | |
| clarkb | as a side note most of these failures seemed to happen in pre-run | 15:12 |
| clarkb | so we should have retried most of them | 15:12 |
| *** eernst has joined #openstack-infra | 15:13 | |
| *** ykarel_ is now known as ykarel | 15:14 | |
| *** yamamoto has joined #openstack-infra | 15:15 | |
| *** yamamoto has quit IRC | 15:15 | |
| *** yamamoto has joined #openstack-infra | 15:15 | |
| *** eernst has quit IRC | 15:16 | |
| *** eernst has joined #openstack-infra | 15:16 | |
| *** dpawlik has joined #openstack-infra | 15:18 | |
| *** yamamoto has quit IRC | 15:20 | |
| *** dpawlik has quit IRC | 15:22 | |
| *** parasitid has quit IRC | 15:28 | |
| *** lpetrut has quit IRC | 15:29 | |
| *** e0ne has quit IRC | 15:32 | |
| clarkb | fungi: sorry missed that question http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22Mirror%20sync%20in%20progress%5C%22 which zbr provided earlier | 15:33 |
| clarkb | no new hits yet | 15:33 |
| *** armax has quit IRC | 15:34 | |
| *** gyee has joined #openstack-infra | 15:35 | |
| zbr | clarkb: so you flushed afs cache and it no longer appears? | 15:36 |
| clarkb | zbr: correct | 15:37 |
| *** spsurya has joined #openstack-infra | 15:40 | |
| rosmaita | fungi: when you have a few minutes, let me know if i addressed your concern: https://review.opendev.org/#/c/655451/ | 15:41 |
| *** bnemec has quit IRC | 15:44 | |
| *** ykarel has quit IRC | 15:45 | |
| *** jistr is now known as jistr|call | 15:45 | |
| *** jistr|call is now known as jistr | 15:45 | |
| *** bnemec has joined #openstack-infra | 15:46 | |
| clarkb | follow up on the nb03 arm64 image builder situation. All images have been rebuilt in the last ~15 hours including the buster image | 15:46 |
| clarkb | if hrw comes back around looking for buster images it is there now | 15:46 |
| *** hamzy_ has quit IRC | 15:48 | |
| evrardjp | pabelanger: mmm I am editing my fact cache settings, pointing to a certain location, so I see my ansible fact cache setting works. I edited the localhost file to have the same content as the line it's pointing. and things are sitll working out of magic | 15:51 |
| *** cousin_luigi has joined #openstack-infra | 15:56 | |
| cousin_luigi | Greetings. | 15:56 |
| *** ccamacho has joined #openstack-infra | 15:57 | |
| *** iurygregory has quit IRC | 15:59 | |
| *** lucasagomes has quit IRC | 16:00 | |
| clarkb | fungi: part of the raeson we haven't been able to remove the shim is people won't update the jobs :P | 16:01 |
| clarkb | fungi: though I think at the pTG we mostly decided to pull the rug out from under them https://review.opendev.org/#/c/656195/ is the chagne for that I think | 16:01 |
| clarkb | if the legacy jobs were gone the shim would be gone | 16:02 |
| fungi | rosmaita: you're too late, i approved it ~20 minutes before you asked | 16:02 |
| clarkb | er I guess that rug pull is only for bindep side | 16:02 |
| clarkb | we should also rug pull z-c shim | 16:02 |
| fungi | yep, that was what i was getting at | 16:03 |
| AJaeger | clarkb, for 656195 it also will break python36 etc jobs for repos that don't have bindep.txt files | 16:03 |
| fungi | clarkb: same scenario though, i agree | 16:03 |
| clarkb | AJaeger: yes | 16:03 |
| clarkb | AJaeger: we need to send an email and give people a few weeks to prepare | 16:03 |
| clarkb | AJaeger: but then we should merge it and continue to march forward | 16:03 |
| fungi | AJaeger: what specifically will be broken in the python36 jobs? | 16:03 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Return python artifact records to Zuul https://review.opendev.org/663053 | 16:04 |
| clarkb | fungi: they won't have python3 installed | 16:04 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Return javascript content artifact records to Zuul https://review.opendev.org/663056 | 16:04 |
| clarkb | orany other deps | 16:04 |
| fungi | clarkb: really? i thought python3.6 was installed by default in bionic | 16:04 |
| fungi | do our images not include it? | 16:04 |
| *** pkopec has quit IRC | 16:04 | |
| clarkb | fungi: oh thats a good point sopython3 may be a bad example | 16:04 |
| clarkb | all their other system deps like libssl-dev then | 16:04 |
| AJaeger | fungi, any kind of job that needs one of the packages in there - like python3 or mariadb, won't it? | 16:04 |
| rosmaita | fungi: ty! | 16:05 |
| *** sreejithp has joined #openstack-infra | 16:05 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: DNM: test zuul-jobs artifact changes Depends-On: https://review.opendev.org/663056 https://review.opendev.org/663081 | 16:06 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: DNM: test zuul-jobs artifact changes https://review.opendev.org/663081 | 16:06 |
| zbr | clarkb: fungi: i am considering making a presentation about how to use molecule for testing ansible roles for Shanghai summit. would you find the subject interesting? | 16:09 |
| clarkb | zbr: maybe? I think we have foudn our new tests that actually run our full ansible run with puppet too to be the most effective testing for us | 16:09 |
| clarkb | but we also aren't writing general roles for general consumption usually | 16:10 |
| *** dtantsur is now known as dtantsur|afk | 16:11 | |
| fungi | yeah, our main need is to satisfy ourselves that we won't break critical service interactions | 16:11 |
| zbr | clarkb: the big difference is that using molecule the user could test changes made to a single role locally almost instantly via tox, on multiple platforms in parallel. current testing approach does require user to raise CR which would be run by zuul. | 16:11 |
| *** tosky has quit IRC | 16:12 | |
| clarkb | zbr: ya I have no interest in running tests like that locally | 16:12 |
| clarkb | (I don't trust config managemtn tools on my local machines that are not my local machine config management tools :) ) | 16:13 |
| clarkb | so I have to spin up a Vm for that and unless I need to debug unexpected behavior zuul is great for that :) | 16:13 |
| zbr | well... maybe user locally term was incorrect: i wanted to say outside ci, on docker containers, VMs, or clouds, molecule can use any of these as providers. | 16:13 |
| zbr | nothing ever is running locally, it would be.... insane | 16:14 |
| *** ykarel has joined #openstack-infra | 16:15 | |
| fungi | at that point, all that's missing to turn it into a ci system is a scheduler ;) | 16:15 |
| zbr | haha, good one | 16:15 |
| *** ykarel is now known as ykarel|away | 16:15 | |
| *** pgaxatte has quit IRC | 16:16 | |
| clarkb | I think if our focus switched to writing roles that were supposed to be on galaxy and used across a wide variety of systems then molecule would make a lot of sense | 16:16 |
| clarkb | but our experience trying to do that in the puppet world is that it is a lot of effort for almost no return so we've transitioned away from that to writing roles for what we need | 16:17 |
| clarkb | and those are constrained to a small enough set of systems and assumptions that our run the world testing works great | 16:17 |
| *** mriedem_away is now known as mriedem | 16:20 | |
| *** ramishra has quit IRC | 16:20 | |
| *** xek has quit IRC | 16:20 | |
| *** xek_ has joined #openstack-infra | 16:20 | |
| *** ykarel|away has quit IRC | 16:23 | |
| evrardjp | pabelanger: so I couldn't really reproduce this locally. So I edited the file differently, and made that non read/writable by ansible, so that fact caching fails, and then did the filter on date_time, and that works in my playbook | 16:24 |
| evrardjp | however, I had to explicitly say that the gather_facts: was smart | 16:25 |
| evrardjp | (which corresponds to L406 on the link you gave me) | 16:25 |
| *** lpetrut has joined #openstack-infra | 16:27 | |
| *** jcoufal has quit IRC | 16:29 | |
| *** jcoufal has joined #openstack-infra | 16:29 | |
| openstackgerrit | Jean-Philippe Evrard proposed zuul/zuul-jobs master: Explicitly store facts for promote https://review.opendev.org/662817 | 16:30 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: DNM: test zuul-jobs artifact changes https://review.opendev.org/663081 | 16:31 |
| *** jpich has quit IRC | 16:32 | |
| openstackgerrit | Jean-Philippe Evrard proposed zuul/zuul-jobs master: Explicitly store date facts for promote https://review.opendev.org/662817 | 16:32 |
| *** mattw4 has joined #openstack-infra | 16:33 | |
| evrardjp | pabelanger: I added you to the review. I will work on that tomorrow based on your comments (if any). | 16:33 |
| evrardjp | thanks for your help, clarkb pabelanger and corvus! | 16:34 |
| evrardjp | (and patience, that was awesome!) | 16:34 |
| corvus | evrardjp: thank you! | 16:34 |
| *** mattw4 has quit IRC | 16:37 | |
| *** xek_ has quit IRC | 16:37 | |
| *** kjackal has quit IRC | 16:43 | |
| dmsimard | ara 1.0 has officially been released and tagged, please let me know if anyone notices fallout | 16:52 |
| dmsimard | I also plan on helping with upgrades in the near future, ofc | 16:53 |
| openstackgerrit | Merged openstack/devstack-gate master: Pin ARA to <1.0.0 https://review.opendev.org/662746 | 16:53 |
| *** lpetrut has quit IRC | 16:55 | |
| openstackgerrit | Adam Coldrick proposed opendev/storyboard master: Add a 'security' flag to Teams https://review.opendev.org/640823 | 16:55 |
| openstackgerrit | Adam Coldrick proposed opendev/storyboard master: Add a table mapping Teams to Projects https://review.opendev.org/641718 | 16:55 |
| openstackgerrit | Adam Coldrick proposed opendev/storyboard master: Add an endpoint for getting Projects related to a Team https://review.opendev.org/641719 | 16:55 |
| openstackgerrit | Adam Coldrick proposed opendev/storyboard master: Allow Teams to be filtered by project_id https://review.opendev.org/641720 | 16:55 |
| openstackgerrit | Adam Coldrick proposed opendev/storyboard master: Add a security flag to Stories https://review.opendev.org/642056 | 16:55 |
| openstackgerrit | Adam Coldrick proposed opendev/storyboard master: Add RESTful endpoints for Story permissions https://review.opendev.org/642057 | 16:55 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Revert "Create zuul/web/static on demand" https://review.opendev.org/663099 | 16:57 |
| *** pkopec has joined #openstack-infra | 16:57 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: DNM: test zuul-jobs artifact changes https://review.opendev.org/663081 | 16:57 |
| *** mattw4 has joined #openstack-infra | 16:58 | |
| *** weifan has joined #openstack-infra | 17:03 | |
| *** derekh has quit IRC | 17:06 | |
| *** cousin_luigi has left #openstack-infra | 17:09 | |
| *** jpena is now known as jpena|off | 17:09 | |
| *** hamzy has joined #openstack-infra | 17:10 | |
| *** ykarel|away has joined #openstack-infra | 17:12 | |
| *** ykarel|away has quit IRC | 17:17 | |
| *** kopecmartin is now known as kopecmartin|off | 17:18 | |
| *** adriant has quit IRC | 17:20 | |
| *** iokiwi has quit IRC | 17:21 | |
| *** gfidente is now known as gfidente|afk | 17:21 | |
| *** whoami-rajat has quit IRC | 17:24 | |
| clarkb | dmsimard: is there a tldr on what changes you have to make to use it? | 17:27 |
| dmsimard | clarkb: recording data is the same -- pip install ara (or pip install ara[server] if recording data locally) and export ANSIBLE_CALLBACK_PLUGINS=$(python -m ara.setup.callback_plugins) | 17:28 |
| *** ociuhandu has quit IRC | 17:28 | |
| dmsimard | the web interface is completely decoupled now and is a stateless consumer of the api | 17:28 |
| *** pkopec has quit IRC | 17:29 | |
| clarkb | dmsimard: does that mean you can't compile a static site anymore? | 17:29 |
| dmsimard | not yet | 17:29 |
| dmsimard | the web interface is somewhat similar to zuul-web, it compiles to static css/js/html but expects an API to pull in data | 17:30 |
| dmsimard | an equivalent to static html export and/or the sqlite wsgi middleware is work for >1.0 | 17:31 |
| clarkb | gotcha | 17:31 |
| dmsimard | but hey, playbook names and labels are in, something that I remember mordred and corvus wanting for a long time: https://ara.readthedocs.io/en/feature-1.0/playbook-names-and-labels.html | 17:33 |
| dmsimard | sorry it took so long :/ | 17:33 |
| clarkb | fungi: were y ou going to push the z-c compat shim removal from base? | 17:37 |
| clarkb | (your email implies you might but wasn't sure) | 17:37 |
| *** ricolin has joined #openstack-infra | 17:38 | |
| *** gyee has quit IRC | 17:41 | |
| *** udesale has quit IRC | 17:46 | |
| fungi | clarkb: i think as long as doing so doesn't conflict with any base-test experiments already underway | 17:46 |
| clarkb | I am not aware of any | 17:46 |
| fungi | like for the bindep fallback removal. i guess that doesn't actually need base job changes? | 17:46 |
| *** jamesmcarthur has quit IRC | 17:46 | |
| clarkb | it does not | 17:46 |
| fungi | looks like we merely take it out of our main site-variables list | 17:47 |
| clarkb | we remove the global var and let people override if they want | 17:47 |
| mordred | dmsimard: woot! | 17:47 |
| *** yamamoto has joined #openstack-infra | 17:47 | |
| *** eernst has quit IRC | 17:47 | |
| fungi | i can't remember, does the bindep role have a graceful fallback behavior if there is no bindep.txt file and no bindep_fallback value? | 17:47 |
| clarkb | fungi: that way you can still say "this job has a bindep file overhere not part of the base dir in $repo | 17:47 |
| fungi | like, basically be a no-op? | 17:48 |
| fungi | just making sure we're not requiring projects to add an empty bindep.txt or somethnig | 17:48 |
| clarkb | fungi: tasks/main.yaml guards for when bindep_file is defined | 17:49 |
| clarkb | fungi: so it should be safe | 17:49 |
| fungi | cool, because running bindep when there is no file supplied exits nonzero | 17:50 |
| *** adriant has joined #openstack-infra | 17:50 | |
| *** iokiwi has joined #openstack-infra | 17:51 | |
| clarkb | that is in zuul/zuul-jobs/roles/bindep fwiw | 17:51 |
| clarkb | I'm fairly certain it is safe as is given my read of it | 17:52 |
| fungi | oh, huh... seems to use https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/bindep/tasks/find.yaml which basically duplicates bindep's own config finding behaviors | 17:52 |
| *** yamamoto has quit IRC | 17:52 | |
| clarkb | ya that sets bindep_file and if unset we don't do the package installs or make an install command | 17:52 |
| fungi | i guess in service of making bindep_file a fallback choice rather than an override, yeah | 17:52 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Revert "Revert "Create zuul/web/static on demand"" https://review.opendev.org/663108 | 17:54 |
| clarkb | looking at train calendar we could probably make those changes in the next ~4 weeks after this week | 17:54 |
| clarkb | maybe pick a week and go for it (I'm out 17-20 though) | 17:54 |
| fungi | so... the "Define bindep_run fact" task seems to only fire if the find tasks located a bindep file or a bindep_file fallback is specified. however in the packages tasks we unconditionally seem to do "Get list of packages to install from bindep" with a command using the result. what's going to happen there when bindep_run is undefined? | 17:56 |
| *** ginopc has quit IRC | 17:56 | |
| *** jcoufal has quit IRC | 17:57 | |
| *** jcoufal has joined #openstack-infra | 17:58 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Return javascript content artifact records to Zuul https://review.opendev.org/663056 | 17:59 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Return javascript content artifact records to Zuul https://review.opendev.org/663056 | 18:00 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Return python artifact records to Zuul https://review.opendev.org/663053 | 18:00 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: DNM: test zuul-jobs artifact changes https://review.opendev.org/663081 | 18:01 |
| *** rkukura has joined #openstack-infra | 18:06 | |
| *** diablo_rojo has quit IRC | 18:12 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Return python artifact records to Zuul https://review.opendev.org/663053 | 18:13 |
| clarkb | fungi: we also skip the package tasks if bindep_file is undefined | 18:19 |
| clarkb | there is a when on that include in main.yaml | 18:19 |
| fungi | oh, yep, so there is! concern withdrawn | 18:20 |
| *** tdasilva has quit IRC | 18:21 | |
| fungi | +2 from me then | 18:21 |
| *** spsurya has quit IRC | 18:24 | |
| *** whoami-rajat has joined #openstack-infra | 18:25 | |
| * clarkb eats before meeting | 18:27 | |
| *** diablo_rojo has joined #openstack-infra | 18:31 | |
| *** ociuhandu has joined #openstack-infra | 18:34 | |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul-jobs master: Use password lookup for run-buildset-registry role https://review.opendev.org/663119 | 18:43 |
| *** ociuhandu has quit IRC | 18:43 | |
| *** weifan has quit IRC | 18:44 | |
| *** weifan has joined #openstack-infra | 18:44 | |
| *** weifan has quit IRC | 18:45 | |
| fungi | evrardjp: clarkb: ^ that came out of the discussions yesterday about ansible facts | 18:46 |
| fungi | depends on getting that plugin exposed by zuul, which is still wip, but this is the next step in seeing if it will work | 18:47 |
| clarkb | \o/ | 18:48 |
| *** weifan has joined #openstack-infra | 18:49 | |
| *** weifan has quit IRC | 18:51 | |
| *** kjackal has joined #openstack-infra | 18:52 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Return python artifact records to Zuul https://review.opendev.org/663053 | 18:53 |
| *** e0ne has joined #openstack-infra | 18:54 | |
| *** e0ne has quit IRC | 18:55 | |
| clarkb | friendly reminder our weekly meeting begins in a few minutes. Join us in #openstack-meeting | 18:56 |
| mnaser | 2019-06-04 14:56:34.347479 | ubuntu-bionic | E: Failed to fetch http://mirror.dfw.rax.opendev.org/ubuntu/dists/bionic-updates/main/binary-amd64/Packages.gz File has unexpected size (906361 != 907417). Mirror sync in progress? [IP: 2001:4800:7819:105:be76:4eff:fe04:9b8a 80] | 19:00 |
| mnaser | im assume this is safe to recheck? | 19:00 |
| openstackgerrit | Merged zuul/zuul master: Revert "Create zuul/web/static on demand" https://review.opendev.org/663099 | 19:00 |
| clarkb | mnaser: yes we think we fixed that today by clearing the afs cache on that mirror | 19:00 |
| mnaser | cool | 19:00 |
| clarkb | logstash shows no new instances since | 19:00 |
| *** ricolin has quit IRC | 19:10 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] backups https://review.opendev.org/662657 | 19:11 |
| *** e0ne has joined #openstack-infra | 19:29 | |
| *** e0ne has quit IRC | 19:30 | |
| *** raissa has joined #openstack-infra | 19:31 | |
| *** weifan has joined #openstack-infra | 19:33 | |
| *** eernst has joined #openstack-infra | 19:33 | |
| *** jcoufal has quit IRC | 19:33 | |
| *** weifan has quit IRC | 19:38 | |
| *** eernst has quit IRC | 19:38 | |
| *** imacdonn has quit IRC | 19:38 | |
| *** weifan has joined #openstack-infra | 19:39 | |
| *** eernst has joined #openstack-infra | 19:39 | |
| *** altlogbot_2 has joined #openstack-infra | 19:39 | |
| *** jamesmcarthur has joined #openstack-infra | 19:42 | |
| *** slaweq has quit IRC | 19:43 | |
| *** eernst has quit IRC | 19:44 | |
| *** slaweq has joined #openstack-infra | 19:45 | |
| *** jamesmcarthur has quit IRC | 19:45 | |
| *** jamesmcarthur has joined #openstack-infra | 19:45 | |
| *** imacdonn has joined #openstack-infra | 19:53 | |
| *** raissa has quit IRC | 19:59 | |
| mordred | clarkb: I approved the first nodepool builders patch | 19:59 |
| clarkb | mordred: woot | 19:59 |
| *** dave-mccowan has joined #openstack-infra | 20:00 | |
| clarkb | I need a short break after the meeting but them I'm back to it and will start working through my todo list as well as keeping an eye on nodepool | 20:02 |
| *** raissa has joined #openstack-infra | 20:02 | |
| *** raissa has quit IRC | 20:02 | |
| openstackgerrit | Merged opendev/puppet-mediawiki master: More transitional Xenial package name replacements https://review.opendev.org/661848 | 20:05 |
| openstackgerrit | Jeremy Stanley proposed zuul/nodepool master: Update os-client-config references to openstacksdk https://review.opendev.org/663131 | 20:05 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] backups https://review.opendev.org/662657 | 20:05 |
| openstackgerrit | Merged opendev/system-config master: Add control plane clouds to nodepool builder clouds.yaml https://review.opendev.org/640027 | 20:15 |
| *** gfidente|afk has quit IRC | 20:21 | |
| *** weifan has quit IRC | 20:23 | |
| *** e0ne has joined #openstack-infra | 20:23 | |
| openstackgerrit | Jeremy Stanley proposed opendev/base-jobs master: Test removal of fetch-zuul-cloner https://review.opendev.org/663135 | 20:25 |
| *** Lucas_Gray has joined #openstack-infra | 20:25 | |
| *** dave-mccowan has quit IRC | 20:27 | |
| *** Lucas_Gray has quit IRC | 20:27 | |
| clarkb | and back | 20:29 |
| *** Lucas_Gray has joined #openstack-infra | 20:29 | |
| fungi | so i guess now we need to decide what dnm changes we want to depend on that ^ | 20:30 |
| openstackgerrit | Clark Boylan proposed opendev/infra-specs master: Mark opendev-gerrit spec completed https://review.opendev.org/663136 | 20:30 |
| fungi | well, not depend in the zuul depends-on sense | 20:30 |
| fungi | but switch to base-test once that merges | 20:30 |
| clarkb | fungi: I would do a zuulv3 tempest job, a nova/neutron unittest jobs, and maybe a requirements job? I think that will get you like 95% coverage | 20:31 |
| *** hamzy has quit IRC | 20:31 | |
| clarkb | infra-root https://review.opendev.org/663136 is an easy review and marks opdnev-gerrit spec completed if you have a quick moment | 20:31 |
| fungi | also some legacy jobs, grenade in particulat | 20:31 |
| fungi | particuler | 20:31 |
| fungi | gah | 20:31 |
| * fungi gives up on typing | 20:31 | |
| diablo_rojo | lol | 20:32 |
| clarkb | fungi: we aren't removing it from them are we? | 20:32 |
| clarkb | fungi: those will still use the shim | 20:32 |
| clarkb | (they require it) | 20:32 |
| clarkb | I guess you want to make sure we haven't accidentally removed it from those jobs? | 20:33 |
| fungi | yes | 20:33 |
| fungi | or more precisely, that they're actually using the bits we think they are in legacy-base | 20:33 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] backups https://review.opendev.org/662657 | 20:33 |
| fungi | because right now this role is in both base and legacy-base, so in theory we're doing it twice for those jobs? | 20:34 |
| *** sthussey has joined #openstack-infra | 20:34 | |
| fungi | or do legacy-base jobs not inherit from base? | 20:34 |
| clarkb | legacy-base is a separate base no shared inheritance iirc | 20:35 |
| fungi | right, okay, that's why it's not running twice | 20:37 |
| *** eharney has quit IRC | 20:41 | |
| *** e0ne has quit IRC | 20:41 | |
| *** e0ne has joined #openstack-infra | 20:42 | |
| *** e0ne has quit IRC | 20:43 | |
| *** dklyle has quit IRC | 20:49 | |
| *** dklyle has joined #openstack-infra | 20:49 | |
| fungi | and with that, i should probably go wrestle my yard into submission while the weather is being reasonably mild | 20:51 |
| *** whoami-rajat has quit IRC | 20:54 | |
| *** mattw4 has quit IRC | 20:55 | |
| *** weifan has joined #openstack-infra | 20:57 | |
| *** mattw4 has joined #openstack-infra | 20:58 | |
| *** jamesmcarthur has quit IRC | 21:00 | |
| *** jamesmcarthur has joined #openstack-infra | 21:00 | |
| *** Goneri has quit IRC | 21:02 | |
| *** jamesmcarthur has quit IRC | 21:04 | |
| *** pcaruana has quit IRC | 21:07 | |
| *** Qiming has quit IRC | 21:11 | |
| *** Qiming has joined #openstack-infra | 21:14 | |
| *** jamesmcarthur has joined #openstack-infra | 21:14 | |
| *** kjackal has quit IRC | 21:16 | |
| *** jamesmcarthur has quit IRC | 21:18 | |
| *** ekultails has quit IRC | 21:19 | |
| *** jamesmcarthur has joined #openstack-infra | 21:20 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Add db backups to gitea https://review.opendev.org/663146 | 21:20 |
| clarkb | corvus: ^ that may simplify your local backups as well | 21:21 |
| clarkb | you don't need multiple commands because things are dumped to stdout so you can exec stdout into gzip | 21:21 |
| clarkb | I'm missing a " | 21:22 |
| clarkb | one sec | 21:22 |
| *** mriedem is now known as mriedem_away | 21:22 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Add db backups to gitea https://review.opendev.org/663146 | 21:22 |
| clarkb | mordred: fwiw the clouds.yaml update seems to have been ok on nb01 | 21:23 |
| clarkb | at least checking the file. Haven't checked if it has uploaded any images since | 21:24 |
| clarkb | it is building an image now so should do uploads soonish | 21:24 |
| clarkb | fungi: +2 on the opendev base jobs fetch-zuul-cloner removal | 21:25 |
| clarkb | fungi: should I start working to draft the email for these flag days? | 21:25 |
| *** jamesmcarthur has quit IRC | 21:28 | |
| fungi | clarkb: not a bad idea. feel free to steal prose from my last message to the other thread | 21:28 |
| fungi | it was a mini teaser announcement in lieu of anything more official | 21:29 |
| *** hwoarang has quit IRC | 21:29 | |
| *** hwoarang has joined #openstack-infra | 21:32 | |
| clarkb | writing an announcement email reminds me of https://review.opendev.org/#/c/661843/ anyone else want to review that one to add a service-announce@opendev.org list? | 21:36 |
| fungi | ooh! | 21:36 |
| fungi | this would be a great thing to christen it with | 21:37 |
| fungi | i missed that got formally proposed, only caught the conversation and probably didn't pay close enough attention | 21:37 |
| *** aakarsh has quit IRC | 21:37 | |
| fungi | make it so | 21:38 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] backups https://review.opendev.org/662657 | 21:38 |
| *** jamesmcarthur has joined #openstack-infra | 21:49 | |
| *** yamamoto has joined #openstack-infra | 21:50 | |
| *** jamesmcarthur has quit IRC | 21:51 | |
| *** eernst has joined #openstack-infra | 21:51 | |
| clarkb | infra-root https://etherpad.openstack.org/p/9NRyYVT2g5 is draft email | 21:52 |
| clarkb | I think we need a chagne that removes z-c from base as well as base-test (we can WIP it) so that I can include that in the email too | 21:52 |
| clarkb | fungi: ^ | 21:52 |
| openstackgerrit | Merged zuul/nodepool master: Update os-client-config references to openstacksdk https://review.opendev.org/663131 | 21:55 |
| *** yamamoto has quit IRC | 21:55 | |
| openstackgerrit | Merged opendev/system-config master: Add opendev service-announce list https://review.opendev.org/661843 | 21:55 |
| fungi | sure, on it | 22:00 |
| fungi | just a sec | 22:00 |
| fungi | on another yard break anyway | 22:00 |
| clarkb | yay mowing | 22:01 |
| fungi | clarkb: on the announcement, i wonder if we should avoid use of the word "legacy" except when talking about jobs which descend from legacy-base, for increased clarity of messaging there | 22:02 |
| clarkb | fungi: how about those edits | 22:02 |
| fungi | maybe say "deprecated" (though people will say "why doesn't it print a deprecation notice?" and we'll proceed to point them to dozens of mailing list messages over the past two years) | 22:02 |
| fungi | yeah, i think what you have there now works | 22:03 |
| clarkb | ya I think emphasizing that this stuff comes from the land before time helps | 22:04 |
| *** jamesmcarthur has joined #openstack-infra | 22:04 | |
| clarkb | aka zuulv2 | 22:04 |
| *** jtomasek has quit IRC | 22:05 | |
| openstackgerrit | Jeremy Stanley proposed opendev/base-jobs master: Remove fetch-zuul-cloner https://review.opendev.org/663151 | 22:06 |
| pabelanger | fungi: added some extra info to your run down about zuul-cloner-removal | 22:06 |
| pabelanger | but good right up, pretty much what happened | 22:06 |
| pabelanger | write* | 22:06 |
| fungi | thanks pabelanger!!! | 22:07 |
| *** eernst has quit IRC | 22:07 | |
| *** slaweq has quit IRC | 22:07 | |
| openstackgerrit | Merged openstack/devstack-gate master: Note status of CINDER_PERIODIC_INTERVAL setting https://review.opendev.org/655451 | 22:08 |
| *** mattw4 has quit IRC | 22:08 | |
| clarkb | hrm I bet tools/tox_install.sh has to be updated to use the /home/zuul/src/* paths | 22:09 |
| clarkb | that should be straightforward to address | 22:09 |
| *** slaweq has joined #openstack-infra | 22:11 | |
| pabelanger | yah, I can't remember 100%, but I also think there was a workflow issue locally when people did tools/tox_install.sh too, maybe AJaeger knows more | 22:12 |
| pabelanger | but +1, looking forward to having it removed | 22:13 |
| clarkb | can probably do a if ! [ -d $HOME/src/$repo ] ; then git clone https://opendev.org/$repo ; done check | 22:13 |
| *** jamesmcarthur has quit IRC | 22:16 | |
| *** sreejithp has quit IRC | 22:19 | |
| *** rfarr has quit IRC | 22:20 | |
| mordred | clarkb: heya - after approving that first clouds.yaml change, I proceeded to be on the phone for 2 hours ... anything break? | 22:22 |
| mordred | clarkb: oh - I see you have reported it seems to be pok | 22:22 |
| mordred | ok | 22:22 |
| mordred | I'll go ahead and approve the next change | 22:22 |
| clarkb | mordred: ya I haven't seen aything go sideways but don't thnik we've done any uploads yet | 22:24 |
| *** slaweq has quit IRC | 22:24 | |
| *** mattw4 has joined #openstack-infra | 22:24 | |
| clarkb | I think we've had a few failed image builds though which I haven't looked at yet | 22:24 |
| corvus | clarkb: i made 2 minor edits to the etherpad: one to clarify what zuul we're talking about (opendev), and the second is that i don't think we need to send this to the zuul-discuss list, as i don't believe the zuul project uses any legacy jobs (and sending it to that list would probably only confuse people) | 22:25 |
| clarkb | Could not open project list url: 'https://opendev.org/openstack/project-config/raw/gerrit/projects.yaml' | 22:25 |
| clarkb | corvus: thanks! | 22:26 |
| clarkb | I'm going to debug (and hopefully fix the above image building error now) | 22:26 |
| fungi | clarkb: yeah, i think a copy to openstack-discuss is likely plenty. i doubt airship, kata or starlingx have any legacy jobs either | 22:26 |
| fungi | but also sending to service-announce is good for paving the way for our future of not caring so much ;) | 22:27 |
| clarkb | wfm | 22:27 |
| clarkb | fwiw that url works in my browser | 22:27 |
| clarkb | I can wget it on nb02 | 22:27 |
| clarkb | so maybe not a persistent error | 22:27 |
| fungi | or maybe we have a bad backend? | 22:28 |
| clarkb | well the backend should be the same | 22:29 |
| clarkb | er same for the wget on nb02 | 22:29 |
| clarkb | fungi: pabelanger any reason to not approve https://review.opendev.org/#/c/663135/ now and get the ball rolling? | 22:31 |
| clarkb | I'll send out the email once that merges I think | 22:31 |
| fungi | clarkb: ahh, good point, if the pool remains static, the source hash should as well | 22:32 |
| pabelanger | clarkb: yah, we should be safe to land | 22:32 |
| clarkb | ok I have approved it | 22:32 |
| *** slaweq has joined #openstack-infra | 22:34 | |
| fungi | pabelanger: i *think* over the ensuing couple of years tonyb, prometheanfire and others working on openstack requirements made great strides to rewrite, replace and backport standard solutions to constraints handling in official openstack projects, so it may go more smoothly this time around | 22:37 |
| *** aakarsh has joined #openstack-infra | 22:37 | |
| prometheanfire | ohai? | 22:38 |
| fungi | i'm poking around in master branches of large projects and am at least only turning up zuul-cloner references in playbooks/legacy/... so *hopefully* those are just being used by legacy jobs | 22:39 |
| clarkb | I've subscribed to the service-announce ml | 22:39 |
| clarkb | you should too :) | 22:39 |
| fungi | ooh, so i should! | 22:39 |
| fungi | so should we all | 22:39 |
| openstackgerrit | Merged opendev/base-jobs master: Test removal of fetch-zuul-cloner https://review.opendev.org/663135 | 22:40 |
| pabelanger | fungi: progress! | 22:40 |
| pabelanger | also, I think tripleo is going to be the biggest user of zuul-cloner, the last I looked they still rely on it a lot | 22:40 |
| pabelanger | however, I also think they new specific base jobs too | 22:41 |
| clarkb | everyone happy with the email etherpad? I'll send it to openstack-discuss and service-announce@lists.opendev.org shortly | 22:41 |
| pabelanger | clarkb:+1 | 22:41 |
| fungi | yeah, and tripleo jobs can always add that role explicitly if they need it | 22:41 |
| clarkb | pabelanger: yes they are also the biggest complainers about it (via zbr) | 22:41 |
| mordred | infra-root: approved the change to build images for the first two control plane clouds | 22:41 |
| fungi | once it's gone from the base job, they can also make a derivative version of that role which doesn't chmod the repos, if that's what they want | 22:42 |
| clarkb | mordred: yay | 22:43 |
| *** dave-mccowan has joined #openstack-infra | 22:44 | |
| *** eernst has joined #openstack-infra | 22:44 | |
| clarkb | ok sending emails now | 22:45 |
| *** slaweq has quit IRC | 22:47 | |
| fungi | i wonder if, in a that's-so-meta sort of way, we need to announce in a number of places the creation of the service-announce ml | 22:50 |
| mordred | fungi: yes. although I do not think we need to announce it on the service-announce ml | 22:51 |
| *** tkajinam has joined #openstack-infra | 22:51 | |
| clarkb | ya I think we should start advertising it to people | 22:51 |
| clarkb | for openstack and starlingx I think we can send emails to their mailing lists | 22:53 |
| clarkb | maybe zuul too or just say hi over the cube wall for zuul :) | 22:53 |
| fungi | mordred: ooh, i hadn't thought of that! | 22:53 |
| clarkb | then for airship we might need to have hogepodge figure out how to get that in the right places (their mailing list is not very well used) | 22:53 |
| fungi | maybe if people sent important things to their mailing list, they might feel more compelled to start using it | 22:54 |
| corvus | fungi: beware of the leopard | 22:55 |
| openstackgerrit | Merged openstack/project-config master: Build ubuntu bionic minimal images for control plane https://review.opendev.org/640044 | 22:56 |
| *** dave-mccowan has quit IRC | 22:56 | |
| *** rh-jelabarre has quit IRC | 22:56 | |
| *** dave-mccowan has joined #openstack-infra | 22:57 | |
| *** rh-jelabarre has joined #openstack-infra | 22:57 | |
| fungi | corvus: how do you know what's on the door to my disused lavatory?!? | 22:58 |
| corvus | in the storyboard slow query log i have found a single query which is over 138KiB long. it "only" took 51 seconds to run. | 22:59 |
| fungi | WOW | 22:59 |
| fungi | just... wow | 22:59 |
| fungi | i'm surprised it could be parsed in 51 seconds | 23:00 |
| fungi | much less executed | 23:02 |
| mordred | the mysql lexer and parser are actually quite good hand-crafted madness | 23:02 |
| corvus | fungi: do you think there's any sensitive info in these? | 23:02 |
| fungi | it seems unlikely. i didn't notice any when spot-checking, at least | 23:02 |
| fungi | the sensitive info would be in where clauses i suppose? but i doubt we query on api tokens and session keys | 23:03 |
| fungi | or on full content of sensitive things in private stories | 23:03 |
| corvus | the word 'token' does not appear in the log | 23:04 |
| fungi | the sensitive info would be in the query results | 23:04 |
| fungi | so unlikely | 23:04 |
| *** slaweq has joined #openstack-infra | 23:04 | |
| corvus | there are 15.4k queries logged, they de-duplicate to 5588 unique queries (though i suspect if we factor out parameterization we'll get even fewer) | 23:06 |
| fungi | might also make sense to rank them based on execution time multiplied by occurrence? | 23:07 |
| mordred | corvus: wget http://percona.com/get/pt-query-digest ; bash pt-query-digest /path/to/slow-query-log.sql> digest-output.txt | 23:07 |
| fungi | an occasional really-long-running query is probably less of a target than a very-frequently-run moderately-slow query | 23:07 |
| corvus | yeah; also examined_rows-returned_rows would be interesting (as a measure of inefficiency) | 23:07 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Add db backups to gitea https://review.opendev.org/663146 | 23:07 |
| fungi | ahh, yeah, that makes sense | 23:08 |
| corvus | mordred: what you don't think i should rewrite that in python?! :) thank you. | 23:08 |
| mordred | :) | 23:08 |
| corvus | mordred: though that does seem to be a perl script not a bash script | 23:09 |
| mordred | oh - piddle | 23:09 |
| mordred | chmod +x pt-query-digest :) | 23:09 |
| corvus | it is chugging | 23:10 |
| mordred | corvus: if you want to get crazy, you can do filters, like this: | 23:10 |
| mordred | --filter '($event->{Row_ratio} = $event->{Rows_sent} / ($event->{Rows_examined})) && 1' | 23:10 |
| mordred | which will add a ratio of rows sent to rows examined | 23:10 |
| corvus | mordred: if the program runs long enough, it outputs a progress and ETA :) | 23:10 |
| mordred | https://www.percona.com/doc/percona-toolkit/2.2/pt-query-digest.html | 23:10 |
| corvus | mordred, fungi: i think what would be maximially helpful would be to put the raw query log somewhere, along with the output of this program. there's probably enough to get started on improvements right away, but it may also be useful for the folks digging in to this to run their own analisys with different args. | 23:12 |
| mordred | ++ | 23:12 |
| corvus | i'll stick them in afs and we can share a files.o.o link | 23:12 |
| mordred | people can also get them from AFS that way | 23:14 |
| corvus | true | 23:14 |
| clarkb | compeltely unrelated but I've discovered the utility of firefox containers (different than linux containers but similar in that they partition sets of tabs off from each other) | 23:15 |
| *** slaweq has quit IRC | 23:17 | |
| *** Lucas_Gray has quit IRC | 23:17 | |
| *** dchen has joined #openstack-infra | 23:18 | |
| tonyb | I thought we got rid of the last reason for tox_install.sh so if you find people using it then please point them out so we can invesitage | 23:20 |
| *** rcernin has joined #openstack-infra | 23:20 | |
| clarkb | tonyb: excellent news and will do | 23:21 |
| *** dchen has quit IRC | 23:21 | |
| *** dchen has joined #openstack-infra | 23:22 | |
| *** lseki has quit IRC | 23:22 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] backups https://review.opendev.org/662657 | 23:27 |
| *** gyee has joined #openstack-infra | 23:29 | |
| hogepodge | clarkb: we have a meeting this thursday where I'll mention it | 23:32 |
| clarkb | fungi: our puppet jobs use the z-c shim if you want more test cases | 23:37 |
| clarkb | also xenial updates now doing the bionic updates errors? | 23:37 |
| clarkb | is something wrong with afs or with reprepro? | 23:38 |
| ianw | E: Failed to fetch http://mirror.dfw.rax.opendev.org/ubuntu/dists/bionic-security/universe/binary-amd64/Packages.gz File has unexpected size (414749 != 415430). Mirror sync in progress? [IP: 2001:4800:7819:105:be76:4eff:fe04:9b8a 80] | 23:38 |
| clarkb | http://logs.openstack.org/46/663146/3/check/openstackci-beaker-puppet-4/e4939c7/job-output.txt.gz#_2019-06-04_23_35_38_494535 xenial errors too | 23:38 |
| clarkb | these were the type of errors I corrected earlier with the flush volume | 23:38 |
| ianw | is that related to the prior afs issues | 23:39 |
| ianw | ... jinx ... | 23:39 |
| ianw | are we thinking afs client issues? | 23:39 |
| clarkb | possible, this is a bionic host right? | 23:39 |
| clarkb | I'm going to flushvolume as that seemed to help previously | 23:39 |
| openstackgerrit | Merged opendev/infra-specs master: Mark opendev-gerrit spec completed https://review.opendev.org/663136 | 23:40 |
| clarkb | flushvolume done | 23:40 |
| ianw | yeah, this is 1.8 i think, as opposed to 1.6 | 23:40 |
| *** yamamoto has joined #openstack-infra | 23:41 | |
| ianw | 1.8.0~pre5 | 23:42 |
| ianw | upstream is up to 1.8.3 | 23:42 |
| clarkb | odd that it didn't start until yseterayish | 23:42 |
| clarkb | *yesterdayish | 23:43 |
| clarkb | typing is hard | 23:43 |
| clarkb | or maybe it was happening previously and we didn't notice? | 23:43 |
| *** sthussey has quit IRC | 23:43 | |
| clarkb | ianw: if the flushvolume doesn't help I did run fs flush on specific paths last time prior to the flush volume | 23:44 |
| clarkb | ianw: root's shell history on that host has the details | 23:44 |
| ianw | i'm wondering if the openafs client role has a history on a long-running server? | 23:44 |
| ianw | the wheel builds are ephemeral servers ... is it used anywhere else? | 23:45 |
| clarkb | maybe when we had specific wheel build servers? but I think those were puppet at the time | 23:45 |
| ianw | probably worth an audit to double check it's idempotence, before we blame openafs :) | 23:45 |
| clarkb | ++ | 23:46 |
| *** eernst has quit IRC | 23:46 | |
| clarkb | http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22Mirror%20sync%20in%20progress%5C%22 for tracking | 23:47 |
| ianw | let me find that request in the logs | 23:48 |
| clarkb | 16:45:46 is the most recent hit | 23:48 |
| ianw | 2019-06-04 23:35:52.738695 for Packages.gz was the one i watched | 23:49 |
| clarkb | er that is pacific time | 23:49 |
| clarkb | there is a :48 hit now too | 23:49 |
| clarkb | so flushvolume may not have helped | 23:49 |
| clarkb | I would've thought the flushvolume a super set of the fs flush but maybe it isn't. I guess I should issue the specific flushes too | 23:50 |
| *** slaweq has joined #openstack-infra | 23:50 | |
| fungi | so... we're only seeing it on one client right? | 23:51 |
| fungi | hard to blame the source volume if so | 23:51 |
| clarkb | fungi: yes | 23:52 |
| fungi | has apache been restarted since the proxy ran out of disk and started segfaulting? could it just be some borked apache threads/memory? | 23:52 |
| ianw | 2001:4800:7819:105:be76:4eff:fe04:3a3d - - [04/Jun/2019:23:35:51 +0000] "GET /ubuntu/dists/bionic-security/universe/binary-amd64/Packages.gz HTTP/1.1" 200 414749 - "-" "Debian APT-HTTP/1.3 (1.6.10)" | 23:52 |
| *** lseki has joined #openstack-infra | 23:53 | |
| ianw | that was it serving the request; it didn't error or otherwise think there was anything wrong with that file, afaics | 23:53 |
| ianw | root 1133 1 0 May24 ? 00:00:32 /usr/sbin/apache2 -k start ... i guess it wasn't a full "stop/start"? | 23:54 |
| ianw | i'll do a full restart | 23:58 |
| fungi | nothing new in dmesg or the apache error log, so it likely won't help | 23:58 |
| clarkb | ok I've fs flushed the paths I could see that were sad as well as the root disk Release files | 23:59 |
| clarkb | now to see if that helps or was a fluke last time | 23:59 |
| clarkb | I'm going to have to pop out soon | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!