Wednesday, 2018-11-21

*** jamesmcarthur has quit IRC00:00
openstackgerritIan Wienand proposed openstack/diskimage-builder master: simple-init: allow for NetworkManager support  https://review.openstack.org/61912000:07
*** jamesmcarthur has joined #openstack-infra00:07
*** tosky has quit IRC00:12
*** armax has joined #openstack-infra00:13
*** mriedem has quit IRC00:17
openstackgerritIan Wienand proposed openstack/diskimage-builder master: simple-init: allow for NetworkManager support  https://review.openstack.org/61912000:19
*** vinaykns has quit IRC00:21
*** yamamoto has quit IRC00:34
*** gfidente has quit IRC00:34
*** longkb has joined #openstack-infra00:38
*** longkb has quit IRC00:38
openstackgerritIan Wienand proposed openstack/diskimage-builder master: simple-init: allow for NetworkManager support  https://review.openstack.org/61912000:41
*** scas has quit IRC00:44
*** hamzy has joined #openstack-infra00:44
openstackgerritIan Wienand proposed openstack-infra/glean master: [wip] NetworkManager support  https://review.openstack.org/61896400:49
openstackgerritIan Wienand proposed openstack/diskimage-builder master: simple-init: allow for NetworkManager support  https://review.openstack.org/61912000:57
*** jamesmcarthur has quit IRC01:05
*** jamesmcarthur has joined #openstack-infra01:08
openstackgerritIan Wienand proposed openstack-infra/glean master: [wip] NetworkManager support  https://review.openstack.org/61896401:14
*** jamesmcarthur has quit IRC01:17
*** jamesmcarthur has joined #openstack-infra01:18
*** eernst has joined #openstack-infra01:19
*** eernst has quit IRC01:21
*** jamesmcarthur has quit IRC01:22
*** roman_g has quit IRC01:26
*** eernst has joined #openstack-infra01:40
*** eernst has quit IRC01:45
*** jamesmcarthur has joined #openstack-infra01:56
*** mtreinish has quit IRC02:13
*** scas has joined #openstack-infra02:17
*** mrsoul has joined #openstack-infra02:30
*** ramishra has quit IRC02:41
openstackgerritIan Wienand proposed openstack-infra/nodepool master: Add Fedora 29 testing  https://review.openstack.org/61867102:49
*** anteaya has quit IRC02:51
*** fuentess has quit IRC03:40
*** bhavikdbavishi has joined #openstack-infra03:58
*** bhavikdbavishi1 has joined #openstack-infra04:01
*** bhavikdbavishi has quit IRC04:03
*** bhavikdbavishi1 is now known as bhavikdbavishi04:03
*** yamamoto has joined #openstack-infra04:05
openstackgerritIan Wienand proposed openstack-infra/nodepool master: Add Fedora 29 testing  https://review.openstack.org/61867104:11
openstackgerritIan Wienand proposed openstack-infra/nodepool master: Add Fedora 29 testing  https://review.openstack.org/61867104:12
*** psachin has joined #openstack-infra04:14
*** ykarel|away has joined #openstack-infra04:15
*** ykarel|away is now known as ykarel04:15
*** udesale has joined #openstack-infra04:17
*** rlandy has quit IRC04:31
*** janki has joined #openstack-infra04:42
*** yamamoto has quit IRC05:20
openstackgerritMerged openstack-infra/project-config master: Retire openstack-ansible-os_monasca-ui  https://review.openstack.org/61732105:21
openstackgerritIan Wienand proposed openstack/diskimage-builder master: simple-init: allow for NetworkManager support  https://review.openstack.org/61912005:21
openstackgerritIan Wienand proposed openstack-infra/nodepool master: Add Fedora 29 testing  https://review.openstack.org/61867105:23
*** yamamoto has joined #openstack-infra05:24
*** chandankumar has joined #openstack-infra05:36
*** chandankumar is now known as chkumar|ruck05:37
*** yamamoto has quit IRC05:47
*** yamamoto has joined #openstack-infra05:51
*** d0ugal has quit IRC06:05
*** yamamoto has quit IRC06:07
*** annp has joined #openstack-infra06:17
openstackgerritTobias Henkel proposed openstack-infra/zuul master: DNM: Test zuul gear keepalive  https://review.openstack.org/61915006:22
*** noama has joined #openstack-infra06:22
*** apetrich has joined #openstack-infra06:40
*** yamamoto has joined #openstack-infra06:41
*** hjensas has joined #openstack-infra06:48
*** yboaron_ has joined #openstack-infra06:49
*** ifat_afek has joined #openstack-infra06:53
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: executor: add support for generic build resource  https://review.openstack.org/57066807:05
openstackgerritIan Wienand proposed openstack-infra/glean master: [wip] NetworkManager support  https://review.openstack.org/61896407:06
*** rfolco has quit IRC07:06
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: Implement zookeeper-auth  https://review.openstack.org/61915507:13
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: Implement zookeeper-auth  https://review.openstack.org/61915607:13
*** ahosam has joined #openstack-infra07:14
*** rcernin has quit IRC07:26
*** quiquell|off is now known as quiquell07:28
openstackgerritMerged openstack-infra/nodepool master: Remove nodepool-k8s-functional and install-nodepool roles  https://review.openstack.org/61640907:43
*** ykarel is now known as ykarel|lunch07:45
*** quiquell is now known as quiquell|brb07:53
*** bhavikdbavishi has quit IRC07:53
*** ccamacho has joined #openstack-infra07:59
*** jtomasek has joined #openstack-infra07:59
openstackgerritzhufl proposed openstack/diskimage-builder master: Add missing ws separator between words  https://review.openstack.org/61916908:02
*** ginopc has joined #openstack-infra08:03
*** sean-k-mooney has quit IRC08:06
*** yboaron_ has quit IRC08:10
*** quiquell|brb is now known as quiquell08:15
*** rpittau has joined #openstack-infra08:19
*** yamamoto has quit IRC08:24
*** bhavikdbavishi has joined #openstack-infra08:32
*** shardy has joined #openstack-infra08:35
*** shardy has quit IRC08:39
*** dtantsur|afk is now known as dtantsur08:42
*** bobh has joined #openstack-infra08:43
*** eumel8 has joined #openstack-infra08:45
*** yamamoto has joined #openstack-infra08:47
*** ralonsoh has joined #openstack-infra08:50
*** ykarel|lunch is now known as ykarel08:52
*** jpena|off is now known as jpena08:52
openstackgerritIan Wienand proposed openstack-infra/glean master: [wip] NetworkManager support  https://review.openstack.org/61896408:56
*** tosky has joined #openstack-infra08:57
openstackgerritTobias Henkel proposed openstack-infra/zuul master: DNM: Test zuul gear keepalive  https://review.openstack.org/61915009:02
*** ccamacho has quit IRC09:02
*** e0ne has joined #openstack-infra09:03
*** ccamacho has joined #openstack-infra09:04
*** yboaron_ has joined #openstack-infra09:05
*** markmcd has joined #openstack-infra09:06
*** rfolco has joined #openstack-infra09:09
*** d0ugal has joined #openstack-infra09:11
*** chkumar|ruck has quit IRC09:18
*** chkumar|ruck has joined #openstack-infra09:18
*** rossella_s has joined #openstack-infra09:19
*** bhavikdbavishi has quit IRC09:22
*** d0ugal has quit IRC09:22
*** bhavikdbavishi1 has joined #openstack-infra09:22
*** roman_g has joined #openstack-infra09:23
*** bhavikdbavishi1 is now known as bhavikdbavishi09:24
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: Implement zookeeper-auth  https://review.openstack.org/61915609:25
*** jaosorior has quit IRC09:26
*** d0ugal has joined #openstack-infra09:26
*** derekh has joined #openstack-infra09:30
*** yboaron_ has quit IRC09:35
*** ahosam has quit IRC09:36
*** yboaron_ has joined #openstack-infra09:36
*** ahosam has joined #openstack-infra09:36
*** ahosam has quit IRC09:40
*** pcaruana has joined #openstack-infra09:48
*** kjackal has joined #openstack-infra09:49
*** shardy has joined #openstack-infra09:51
openstackgerritKartikeya Jain proposed openstack/diskimage-builder master: Adding support for SLES 15 in element 'sles'  https://review.openstack.org/61918609:53
*** yamamoto has quit IRC09:55
*** bobh has quit IRC10:01
*** sshnaidm|afk is now known as sshnaidm10:02
*** jaosorior has joined #openstack-infra10:03
*** ahosam has joined #openstack-infra10:03
openstackgerritRafael Folco proposed openstack-infra/elastic-recheck master: Categorize error mounting image volumes due to libpod bug[1]  https://review.openstack.org/61905910:06
*** Emine has joined #openstack-infra10:10
*** yboaron_ has quit IRC10:15
*** yboaron_ has joined #openstack-infra10:16
*** slivik has joined #openstack-infra10:17
*** yamamoto has joined #openstack-infra10:22
openstackgerritRafael Folco proposed openstack-infra/elastic-recheck master: Categorize internal libpod error  https://review.openstack.org/61919710:24
*** yamamoto has quit IRC10:28
openstackgerritRafael Folco proposed openstack-infra/elastic-recheck master: Categorize internal libpod error  https://review.openstack.org/61919710:29
*** yamamoto has joined #openstack-infra10:30
*** Emine has quit IRC10:31
*** priteau has joined #openstack-infra10:37
*** ahosam has quit IRC10:43
openstackgerritChandan Kumar proposed openstack-infra/elastic-recheck master: Add query for ssh timeout in tempest scenario for bug 1802971  https://review.openstack.org/61757910:48
openstackbug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged] https://launchpad.net/bugs/180297110:48
*** ahosam has joined #openstack-infra10:51
*** shardy has quit IRC10:52
*** shardy has joined #openstack-infra10:53
openstackgerritChandan Kumar proposed openstack-infra/elastic-recheck master: Add query for ssh timeout in tempest scenario for bug 1802971  https://review.openstack.org/61757910:53
openstackbug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged] https://launchpad.net/bugs/180297110:53
*** Emine has joined #openstack-infra10:54
*** ahosam has quit IRC10:55
*** ifat_afek has quit IRC10:58
*** kjackal has quit IRC10:59
*** kjackal_v2 has joined #openstack-infra10:59
*** sambetts_ has joined #openstack-infra11:19
*** sambetts_ is now known as sambetts|afk11:21
*** kjackal_v2 has quit IRC11:25
*** kjackal has joined #openstack-infra11:25
*** udesale has quit IRC11:28
*** kjackal has quit IRC11:30
*** kjackal_v2 has joined #openstack-infra11:30
*** aperevalov has quit IRC11:31
*** aojea has joined #openstack-infra11:39
*** xek has quit IRC11:39
*** BOKALDO_ has joined #openstack-infra11:40
*** yamamoto has quit IRC11:45
openstackgerritJean-Philippe Evrard proposed openstack-infra/infra-manual master: Replace mailing list  https://review.openstack.org/61921611:46
AJaegerfungi, why do I get a thread that is directed to "openstack@lists.openstack.org" via openstack-discuss? See http://lists.openstack.org/pipermail/openstack-discuss/2018-November/000102.html and the rest in the thread. Was that a BCC? OR some miscommunication?11:46
AJaegers/miscommunication/misconfiguration/11:46
*** sean-k-mooney has joined #openstack-infra11:47
ssbarnea|bkp2any logstash guru around? i need help on https://ask.openstack.org/en/question/117469/how-to-use-and-in-logstash-queries/11:48
*** mugsie has joined #openstack-infra11:49
*** njohnston has quit IRC11:49
*** njohnston_ has joined #openstack-infra11:49
*** janki has quit IRC11:50
*** janki has joined #openstack-infra11:56
fricklerAJaeger: iiuc mails to the old lists are getting automatically forwarded to -discuss for the transition now11:56
fricklerAJaeger: the idea being that you can unsubscribe from the old lists now without losing anything11:56
AJaegerfrickler: taht went to openstack - old list is openstack-dev!11:57
fricklerAJaeger: there are three old lists that all move to discuss, openstack, openstack-dev and openstack-ops11:58
* AJaeger reads fungi's announcement again now...11:58
fricklerAJaeger: oh, four in fact, openstack-sigs too11:59
AJaegerOh indeed - frickler, you're right. I missed openstack, only had the others on the radar.11:59
AJaegerthanks, frickler !11:59
AJaegerfungi, ignore me - frickler explained everything, all looks fine ;)11:59
*** psachin has quit IRC12:01
*** evrardjp has joined #openstack-infra12:01
*** ifat_afek has joined #openstack-infra12:01
*** bhavikdbavishi1 has joined #openstack-infra12:02
*** bhavikdbavishi has quit IRC12:03
*** bhavikdbavishi1 is now known as bhavikdbavishi12:03
*** xek has joined #openstack-infra12:04
*** yamamoto has joined #openstack-infra12:07
*** janki has quit IRC12:12
*** sshnaidm is now known as sshnaidm|afk12:15
*** bhavikdbavishi has quit IRC12:25
*** yamamoto has quit IRC12:36
*** jpena is now known as jpena|lunch12:39
aspiershrm, is there some inconsistency regarding whether IRC meetings are prefixed with "openstack-"? http://eavesdrop.openstack.org/meetings/12:45
*** e0ne has quit IRC12:45
aspierse.g. there is nova and openstack-nova12:45
*** aojea has quit IRC12:45
openstackgerritJose Luis Franco proposed openstack-infra/project-config master: ansible-role-* projects switch to non-pypi jobs  https://review.openstack.org/61797412:46
*** dtantsur is now known as dtantsur|brb12:47
*** cdent has joined #openstack-infra12:50
evrardjphey, do we have a best practice for .gitignore files? I have seen like many things in different projects, and I remember I got a series of patches to change details in OSA, I just wondering if at some point we decided on a convention12:50
*** ramishra has joined #openstack-infra12:51
cdentpeople have remarkably strong opinions on .gitignore files. which surprises me12:51
evrardjpI don't really care?12:52
cdentme too12:52
evrardjpok will take yours then :p12:52
evrardjpyou won the argument!12:52
cdentgo me!12:52
aspiersif only there was a git command for figuring out how .gitignore rules apply to individual files ... ;-)12:54
evrardjpaspiers: show off12:54
aspiershaha12:54
aspiersSCNR12:54
evrardjpI had to urbandictionnary SCNR. Totally worth it.12:55
aspiers:)12:55
evrardjpaspiers: on that link is what people use to start the meeting12:58
aspierswhich link?12:58
evrardjpwhat should be used in what's on eavesdrop12:58
evrardjphttp://eavesdrop.openstack.org/meetings/12:58
aspiersoh yeah, I know12:58
evrardjpthat's why you have x things for OSA, because I cannot remember12:58
aspiersI started the "self-healing" meeting this morning, which is why I asked12:58
aspiersbut openstack-self-healing already exists because of how I submitted the IRC meeting to Gerrit12:59
evrardjpok12:59
aspiershaha12:59
*** jcoufal has joined #openstack-infra12:59
aspiersmaybe there should be some validation done by the bot12:59
evrardjpseems #startmeeting self_healing12:59
evrardjpshould be enough12:59
aspierserrr, careful!12:59
evrardjpyeah12:59
aspiersyou almost started another meeting :)12:59
aspiersactually I used hyphen but it converted to _12:59
evrardjpis that new?13:00
aspiersdunno13:00
cdentspace, dash and underscore all mean the same thing to startmeeting13:01
cdentI just posted a making the most of zuul question to the ML: http://lists.openstack.org/pipermail/openstack-discuss/2018-November/000105.html13:01
evrardjpcdent: that's interesting, why do we have openstack-ansible and openstack_ansible in said meetings/ folder then? :)13:02
cdentwell if they ended up in the same folder they must mean the same thing ?13:02
*** e0ne has joined #openstack-infra13:02
cdentor do you mean different folders within there?13:03
*** e0ne has quit IRC13:03
evrardjpdifferent folders withing meetings/13:03
evrardjpat least different folders within meetings exist13:03
*** e0ne has joined #openstack-infra13:03
evrardjpbut hey I prefer your question13:03
cdentMaybe it changed some time after 201613:04
evrardjpI should maybe ask to move all the files in one location, then use said location in the bot13:04
evrardjpcdent: so your question is basically what are the base jobs definitions for your use case, right?13:05
cdentevrardjp: pretty much yeah. or anything else to make it little13:06
openstackgerritJakub Sliva proposed openstack-infra/project-config master: Create repository for ceilometer-dashboard plugin  https://review.openstack.org/61923513:06
*** yamamoto has joined #openstack-infra13:06
evrardjpcdent: well OSA jobs for example only inherit from base, then we have a top level one to checkout our basillion of repos and do minimal stuff (not running for documentation only changes for example)13:07
evrardjpthen we inherit from that pseudo top level13:07
evrardjpso technically you can do whatever you want, but that's not the answer you were looking for, right? :p13:07
cdentwell. I'm hoping for slightly more guidance on the "make sure the logs I want to capture are" part13:08
evrardjpmaybe that part could help: https://github.com/openstack/openstack-ansible/blob/master/zuul.d/playbooks/post.yml#L30-L64  ?13:10
evrardjpbasically https://github.com/openstack/openstack-ansible/blob/master/zuul.d/playbooks/post.yml#L57-L5813:10
*** rh-jelabarre has joined #openstack-infra13:10
evrardjpin fact... I agree with you. I will check what's in base to see if there is a convention already13:11
*** zul has joined #openstack-infra13:13
evrardjpmaybe a good question for #zuul13:16
*** yamamoto has quit IRC13:18
* cdent nods13:18
*** kjackal_v2 has quit IRC13:24
*** kjackal has joined #openstack-infra13:25
*** ifat_afek has quit IRC13:30
*** rlandy has joined #openstack-infra13:37
fungievrardjp: cdent: probably closest thing we have to a project-wide consensus on .gitignore files is the result of the discussion starting at http://lists.openstack.org/pipermail/openstack-dev/2018-August/133393.html and really boils down to "don't add cruft from personal tools to .gitignore" because otherwise we're going to have $random_newcomer blasting 1k repos with a change to add a .gitignore line13:39
fungifor some random editor or ide only they're using. easier to just say that class of things is what you should configure your dev environment to ignore rather than trying to configure it in every project to which you might contribute13:39
logan-cdent: I think if you inherit base its on your job to ship the logs back to the executor from the test node (ie the OSA example evrardjp provided)13:39
cdentlogan-: is there something between base and devstack that has the log magic built in?13:40
fungievrardjp: cdent: aspiers: problem with relocating meeting logs is that the htmlified copies are pre-baked with urls embedding the original paths so would all need some careful stream editing to update for a new location13:42
*** jpena|lunch is now known as jpena13:42
evrardjpfungi: yeah I remember that conversation, thanks for the link :D13:42
logan-cdent: guessing its this https://github.com/openstack-dev/devstack/blob/master/roles/fetch-devstack-log-dir/tasks/main.yaml13:43
evrardjpfungi: for the meeting folder, it's fine like this then :p13:43
*** boden has joined #openstack-infra13:43
fungieven the text copies are going to make at least one mention of the old url in fact13:44
evrardjplogan-: hasn't the base jobs in openstack by default an upload? I'd say so, but I haven't managed to find exactly the same role name, and moved to somehting else :p13:44
evrardjpfungi: yeah, too much effort for not many results13:44
logan-evrardjp: i don't think so, because not every job runs on a test node requiring log uploads13:45
logan-like jobs running on the executor13:45
logan-you could put together a base-with-logs job and inherit that instead13:46
evrardjp yeah13:47
evrardjpbecause I have the impression everyone is doing that sync task, and because of the "onion" layer of pre/post jobs, it would be simpler to be consistent everywhere13:48
*** mriedem has joined #openstack-infra13:49
logan-hmm http://zuul.openstack.org/job/base says base does copy logs though13:49
fungievrardjp: what the infra team did when they changed their meeting name was just add a url to the earlier meeting logs location in a note on the persistent agenda: https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Previous_meetings13:51
fungithough we had even earlier irc meetings still, there was no meetbot for them at the time13:52
fungiwe didn't start logging the infra meeting with meetbot until august 201213:52
evrardjpfungi: that's what I did in OSA too13:53
openstackgerritMarkus Hosch proposed openstack-infra/zuul master: Add testcase to show inconsistency during merging  https://review.openstack.org/61924513:53
evrardjpaspiers: ^13:53
*** quiquell is now known as quiquell|brb13:53
logan-ah the base job's log copying only means it copies logs from the executor to logs.openstack.org https://github.com/openstack-infra/project-config/blob/master/playbooks/base/post-logs.yaml -- the logs have to be shipped to the executor by child jobs currently13:54
*** janki has joined #openstack-infra13:59
*** trown|outtypewww is now known as trown14:02
cdentthanks for digging into that logan- , very helpful14:02
cdentI've got a wip going with a more minimal setup now. I expect it fail greatly, but we'll see14:02
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Display ref instead of NA for time triggered items  https://review.openstack.org/61925214:03
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Add testcase to show inconsistency during merging  https://review.openstack.org/61924514:06
*** ykarel is now known as ykarel|away14:06
*** dtantsur|brb is now known as dtantsur14:07
*** pbourke has quit IRC14:07
*** fuentess has joined #openstack-infra14:07
*** pbourke has joined #openstack-infra14:09
*** ifat_afek has joined #openstack-infra14:09
*** kgiusti has joined #openstack-infra14:11
*** chkumar|ruck has quit IRC14:12
*** Douhet has quit IRC14:13
*** quiquell|brb is now known as quiquell14:19
*** bobh has joined #openstack-infra14:21
*** sthussey has joined #openstack-infra14:22
openstackgerritLars Kellogg-Stedman proposed openstack-infra/glean master: changes for alpine compatibility  https://review.openstack.org/61925814:23
fungissbarnea|bkp2: can you point to an actual job log that you expected to match your logstash query?14:23
openstackgerritQuique Llorente proposed openstack-infra/zuul master: Prevent exception at executor  https://review.openstack.org/61798414:24
*** jaosorior has quit IRC14:25
*** Douhet has joined #openstack-infra14:26
*** njohnston_ has quit IRC14:29
*** njohnston has joined #openstack-infra14:29
*** quiquell is now known as quiquell|brb14:32
*** agopi|brb has quit IRC14:37
*** udesale has joined #openstack-infra14:39
*** quiquell|brb is now known as quiquell14:48
*** ykarel|away has quit IRC14:55
*** Douhet has quit IRC14:56
*** Douhet has joined #openstack-infra15:04
fungi#status log rolled back garbled summit feedback pad using: wget -qO- 'http://localhost:9001/api/1.2.11/restoreRevision?apikey='$(cat /opt/etherpad-lite/etherpad-lite/APIKEY.txt)'&padID=BER-Feedback-Session&rev=5564'15:05
openstackstatusfungi: finished logging15:06
*** agopi has joined #openstack-infra15:08
ssbarnea|bkp2fungi: in fact I am unable to find "newer" builds with the same error. See: https://review.openstack.org/#/c/608968/15:09
ssbarnea|bkp2this CR was created more than a month ago, and meanwhile the error no longer appears in logs. Even the specific occurences from the LP bug were recycled.15:10
*** ykarel|away has joined #openstack-infra15:10
*** ykarel|away is now known as ykarel15:10
ssbarnea|bkp2it seems that time needed to review a change elastic-recheck may make it impossible to test it because log rotation, on both logstash and the fileserver.15:11
ssbarnea|bkp2what should we do in this case? close both bug and CR with - unable to reproduce?15:11
*** chandankumar has joined #openstack-infra15:11
ssbarnea|bkp2i think that we do need retention of >60 days on logs but probably we don't have the disk for it.15:12
ssbarnea|bkp2that specific issue is not of concern right bow, I am more interested about how we should deal with these.15:14
*** Emine has quit IRC15:17
*** Emine has joined #openstack-infra15:18
corvuslogan-, cdent: i think pabelanger picked up on a WIP effort started by mordred to update the base job to sync from ~/logs on all remote hosts automatically, so that jobs wouldn't need to do sync tasks themselves; i don't have links to emails/changes for that handy; pabelanger knows the latest status, but i could probably dig up more info if needed.15:19
cdent✔15:19
*** chandankumar has quit IRC15:22
*** chandankumar has joined #openstack-infra15:24
*** jaosorior has joined #openstack-infra15:24
fungissbarnea|bkp2: we probably could add extra volumes, but it's already something like 6tb of storage sharded across the cluster to get us ~10 days retention (i think)15:28
fungiperformance might dictate that we add more cluster members if we're adding more data, and that's where things get tough because each cluster member is a 60gb ram flavor15:28
fungithat logstash backend is by far the most expensive (to our donors) service we operate besides nodepool15:29
fungiespecially if you take the 20(?) logstash workers into account15:30
ssbarnea|bkp2fungi: at least this explain why I had the impression that the query didn't work: i was looking at a log file which was like 10-12 old and logstash was failing to return anything when window size was 14 days.15:32
ssbarnea|bkp2is not clear anywhere when did the cut-off happen.15:32
ssbarnea|bkp2anyway at this moment I am more concerned about retension on logs.openstack.org than on logstash.15:33
ssbarnea|bkp2for logstash I have low (retention) expectation.15:33
corvuswe should be able to increase raw log storage retention when we switch to swift15:34
ssbarnea|bkp2this also means that we are ok to close CRs made on elastic-search which do not identify anything, I can even try to document that.15:34
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Fix manual dequeue of github items  https://review.openstack.org/61927215:34
corvus(which we could do at any point, but if we delay it a bit more, we might be able to have a better experience)15:34
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Fix manual dequeue of github items  https://review.openstack.org/61927215:35
ssbarnea|bkp2corvus: i have big personal interest in improving how we deal with log files, keep this in mind as I am more than willing to help there.15:35
corvusssbarnea|bkp2: will do, thanks15:37
*** chandankumar has quit IRC15:39
*** quiquell is now known as quiquell|off15:39
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Display ref instead of NA for time triggered items  https://review.openstack.org/61925215:43
fungiyeah, we certainly could benefit from much longer raw log storage retention, but that's not really searchable in a sane way. increasing the retention for logstash.o.o is probably not something we're going to find a way to do unless we can reduce the amount of data we're actually logging15:43
*** ramishra has quit IRC15:45
ssbarnea|bkp2fungi: regarding log browsing please check https://docs.google.com/document/d/1TM2_USBVq55Xqk7WHbnsd-mrQthDTzl0rMq6-4MwbdY/edit# -- i started to write on it some time ago in order not to forget my idea.15:48
clarkbfungi: ya its 6TB total storage with about 5TB useable (so that we can handle the loss of a single node) and we tend to use a good chunk of it15:50
clarkbthe other constraint is you need enough memory to load queries and we often run up against that with available memory on the nodes15:51
*** ykarel is now known as ykarel|away15:52
clarkbwe already filter all debug logs out and still end up with a billion records a day15:52
*** noama has quit IRC15:52
clarkbfungi: how did you determine the revision to rollback to on the feedback etherpad? did you get the contents for each rev going back in time until the bad data went away?15:54
clarkbwonder if there is a smarter method15:54
*** slivik has quit IRC15:57
*** udesale has quit IRC15:58
fungiclarkb: someone else did that and i simply trusted them (after checking that revision looked sane)15:59
clarkbah15:59
fungiwith access to the api you can query pad contents by revision and also revision diffs i think, so that could be a future option in such cases15:59
fricklerclarkb: fungi: you can use the timeslider (clock button on the top right) https://etherpad.openstack.org/p/BER-Feedback-Session/timeslider#580216:11
clarkbfrickler: ah that shows the rev number got it16:11
fungiyeah, someone had provided me the direct timeslider url and i just fed the number from the url into the rev parameter16:14
*** diablo_rojo has joined #openstack-infra16:16
*** agopi is now known as agopi|food16:17
*** jamesmcarthur has quit IRC16:17
*** jamesmcarthur_ has joined #openstack-infra16:18
*** bobh has quit IRC16:19
*** ykarel|away has quit IRC16:27
*** janki has quit IRC16:36
*** bhavikdbavishi has joined #openstack-infra16:40
*** yboaron_ has quit IRC16:43
openstackgerritMerged openstack-infra/system-config master: fedora mirror : drop -p from rsync  https://review.openstack.org/61850416:47
*** pcaruana has quit IRC16:50
*** efried_back_mon has quit IRC16:51
*** agopi|food is now known as agopi16:51
*** xek_ has joined #openstack-infra16:54
clarkbcorvus: jamesmcarthur_ sent email with the info the registrar wants (and seems hopeful that providing this may make them happy)16:55
jamesmcarthur_we shall see clarkb:16:57
*** xek has quit IRC16:57
corvusclarkb, jamesmcarthur_: great, thanks.  i'll get that together in a jiffy16:57
clarkbcorvus: thank you!16:57
jamesmcarthur_corvus: danke16:58
clarkbianw: it looks like you have a bunch of +2's on the bandersnatch retirement change. Let me know if I can help with the pypi afs volume cleanup16:58
*** shardy has quit IRC17:01
*** e0ne has quit IRC17:02
*** Emine has quit IRC17:05
openstackgerritClark Boylan proposed openstack-infra/glean master: [wip] NetworkManager support  https://review.openstack.org/61896417:06
*** shardy has joined #openstack-infra17:07
clarkbif anyone else is interested in future rhel/fedora glean support ^ and its parents are the changes for you17:08
*** ifat_afek has quit IRC17:13
openstackgerritMerged openstack-infra/project-config master: Fix ansible-role-cloud-launcher docs  https://review.openstack.org/61602917:13
*** sambetts|afk has quit IRC17:14
*** betherly has joined #openstack-infra17:15
*** yamamoto has joined #openstack-infra17:16
*** bhavikdbavishi has quit IRC17:16
*** xek_ is now known as xek17:18
*** bhavikdbavishi has joined #openstack-infra17:19
*** yamamoto has quit IRC17:20
*** betherly has quit IRC17:21
openstackgerritClark Boylan proposed openstack/diskimage-builder master: package-installs: provide for skip from env var  https://review.openstack.org/61911917:21
openstackgerritClark Boylan proposed openstack/diskimage-builder master: simple-init: allow for NetworkManager support  https://review.openstack.org/61912017:21
*** bhavikdbavishi has quit IRC17:24
rfolcoclarkb, quick question, is there any mechanism to indicate that a job is deprecated ? I was thinking in a custom message on emit-job-header or something... the issue is that I have deprecated jobs being referred in multiple projects and multiple branches. So the list of changes to remove is huge before actually removing the job definition. Any thoughts ?17:24
clarkbianw: left a couple of thoughts on those dib changes (the new patchsets only fixed pep8)17:24
clarkbrfolco: emitting a warning message via ansible is probably the thing you can do today17:25
clarkbrfolco: then you will be able to query on logstash for that message to dig up where it is used17:25
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources master: Added new endpoint to get my presentations per role and per summit:  https://review.openstack.org/61932117:25
openstackgerritMerged openstack-infra/openstackid-resources master: Added new endpoint to get my presentations per role and per summit:  https://review.openstack.org/61932117:27
rfolcoclarkb, cool, I was just thinking about emit-job-header, but that one I would need a good justification I think17:27
clarkbrfolco: also for the stable branhces you may not want to change the jobs there. At least typically openstack lets stable jobs sit as they were and then once the stable branch goes away or the testing itself no longer works the test is removed entirely17:28
rfolcoclarkb, coz tripleo-ci (where my job stands) is not branched... many projects are branched and have in their zuul config these old branch jobs (ocata)...17:30
rfolcounless I am missing anything, zuul syntax check looks for all branches for untrusted-projects17:31
clarkbrfolco: it does, but if the jobs are already there they should be correct syntax right? Mostly just saying that changing jobs on stable branches is often more effort than it is worth17:32
AJaegerrfolco: also each job should have a description - If I deprecate jobs, I'll add that information to the description.17:32
*** ginopc has quit IRC17:35
rfolcoI am blocked in removing the job definition from my zuul config because many of these branched projects refer to that old job name and don't pass ci for old branches like ocata17:35
rfolcoI wish zuul could follow branches keyword for syntax check as well, not for where running the job17:36
fungii'm not sure what you mean with that wish17:37
corvusjamesmcarthur_, clarkb, fungi: ohhhhh i think i get it.  i think the registrar wants ds records and not the dnskey record.  (gandi asks for dnskey and generates ds records for you).  shouldn't be a problem, it's just a bit more work.17:39
rfolcofungi, like telling zuul which branches you want to ignore for syntax check17:39
corvusjamesmcarthur_, clarkb, fungi: the dnskey record is certainly "more characters" than the ds record :)17:39
fungicorvus: i concur17:39
fungirfolco: oh, zuul doesn't ignore any branches. it loads its configuration from all branches of a project because it might at some point be asked to test a commit for one of them17:40
clarkbrfolco: the issue is you would braek those branches because the jobs are on an unbranched repo17:40
clarkbrfolco: it is an error to remove the jobs from the unbranched repo while still using the jobs in the branched repos17:41
*** jaosorior has quit IRC17:41
clarkbrfolco: if you converted to branched config for the job defs you could hide that debt away in older branches17:41
clarkbbut potentially add more work when you sync between them17:41
clarkbcorvus: aha17:41
*** jamesmcarthur_ has quit IRC17:41
openstackgerritKaio Kassiano Moura Oliveira proposed openstack-infra/project-config master: Retire openstack-ansible-os_monasca-ui  https://review.openstack.org/61721217:42
toskynot sure if I should ask here or on -tc or somewhere else: now that we are moving to openstack-discuss, can I update the author-email field for all sahara repositories to point to it instead of openstack-dev,17:42
*** jamesmcarthur has joined #openstack-infra17:42
toskyor should I wait for a set of coordinated patches?17:42
tosky(update in setup.cfg)17:43
fungitosky: i don't expect any coordinated patches for that. feel free to make appropriate updates to your projects at any time17:43
toskythanks17:43
*** florianf has quit IRC17:45
rfolcothx clarkb AJaeger fungi, will try to find a solution. For now I am moving these jobs to a separate file and parenting them to run a "deprecation warning" playbook.17:45
corvusfungi: should we send only 1 ds record with a sha256 digest (and omit the sha-1 digest record)?17:47
corvusfungi: it appears that's what gandi did for us with zuul-ci17:47
corvusi'm kind of assuming that we've aged out of using sha-1 for that now and it's not necessary.  but i don't really know that fer shure.17:47
fungithat seems safe these days. the number of resolvers out there lacking sha-256 support should be few in number17:47
corvusbootstrapping is the best time to find out anyway :)17:48
fungiagreed17:48
*** dtantsur is now known as dtantsur|afk17:49
*** derekh has quit IRC17:50
clarkbit appears the rfc still has sha1 as set to mandatory but I agree gandi only has sha256 for my domain17:51
openstackgerritJames E. Blair proposed openstack-infra/system-config master: docs: add info on generating DS records  https://review.openstack.org/61933417:51
corvusclarkb, fungi: ^17:51
clarkbI'm on a bit of a review streak. I think I've reivewed all teh things ianw needs for fedora 29 as well as pabelangers zuul executor zone work. At some point I need to write expense report(s) but let me know if there are other broad topics that could use review17:52
clarkbmaybe I should double check on the docker work17:52
*** jamesmcarthur has quit IRC17:54
*** jamesmcarthur has joined #openstack-infra17:54
corvusjamesmcarthur, clarkb, fungi: i sent the info17:55
*** rpittau has quit IRC17:57
jamesmcarthurthanks corvus. Just sent and cc'd you.17:57
*** boden has quit IRC17:59
clarkbreviewing the docker stuff it is occuring to me that ipv6 is apparnetly not a thing that people using docker seem to really think about. But also we tend to front our services with apache anyway for ssl termination. I'm thinking it would probably be simplest for us if we terminated ipv6 in the webserver too then we don't have to rebuild a bunch of images to ipv6 too? On top of that do we want to run18:02
clarkbthese webservers in a container too? Seems like it makes sense to do that since they are largely stateless (if you ignore connection state)18:02
clarkbMostly throwing this out there so that people can think about it, I don't think there is a rush to have an answer18:02
clarkbalso here hoping the NAT holds up :/18:02
fungicorvus: lgtm, though i left an alternative in a comment based on how i did it for my domains... in short: dnssec-dsfromkey can take a dnskey rr as input18:03
corvusfungi: ah yep.  maybe when we expand that doc out some more to encompass the whole process, we should use that instead.  it's a good check.18:04
*** jcoufal_ has joined #openstack-infra18:06
*** jamesmcarthur has quit IRC18:07
*** jamesmcarthur has joined #openstack-infra18:09
corvuscdent: https://review.openstack.org/583346 is the work from pabelanger i was thinking of18:09
AJaegerany config-core to review some changes, please? https://review.openstack.org/616583 https://review.openstack.org/618633 https://review.openstack.org/618722 https://review.openstack.org/#/c/610888/18:09
*** jcoufal has quit IRC18:10
cdentcorvus: great thanks, I'll have a look. In the meantime I managed to get something basic going in a post job and almost have the guts of the job working18:10
AJaegermnaser: do you want to +2A https://review.openstack.org/617212 - that's the openstack-ansible-os_monasca-ui retirement18:10
corvuscdent: cool.  that's definitely the way to go for now since work on the other idea has stalled for the moment.  hopefully we can push it over the line in the not-too-distant future.18:11
clarkbcdent: thinking about light weight I need a database and wsgi, I realized some projects (like zuul) do that with their tox invocations18:12
clarkbcdent: there is a tools/test_setup.sh (or similar name I forget what it is exactly) path that we'll run for you before running tox that you can use to setup the database18:12
clarkbpabelanger: is https://review.openstack.org/#/c/618633/1 used somewhere?18:12
pabelangercorvus: cdent: indeed, I haven't had much time to pick up and push on it. We are using some of it in ansible-network today, and it works well.  I think we decided to hold off on it until openstack switched to swift for logs18:13
pabelangerclarkb: not yet, hopefull for zuul https://review.openstack.org/618634/18:13
*** jpena is now known as jpena|off18:13
pabelangerhopefully*18:13
corvuspabelanger: oh... is there a dependency there, or was it just to reduce churn?18:13
cdentclarkb: thus far the difference in time consumed is only 3 minutes (down to 8 from 11) so less than I was expecting but still satisfying to not use extra stuff18:13
pabelangercorvus: reduce churn IIRC18:13
*** jamesmcarthur has quit IRC18:14
*** jamesmcarthur has joined #openstack-infra18:15
fungiclarkb: cdent: yes, there's even a role in zuul-jobs you can include: http://git.zuul-ci.org/cgit/zuul-jobs/tree/roles/test-setup/tasks/main.yaml18:15
fungiaccording to that it's tools/test-setup.sh18:16
*** florianf has joined #openstack-infra18:16
openstackgerritMerged openstack-infra/project-config master: Move elastic-recheck zuul jobs definitions inside project repo  https://review.openstack.org/61658318:19
*** jamesmcarthur has quit IRC18:20
clarkbAJaeger: pabelanger double check my comments on https://review.openstack.org/#/c/618633/118:21
AJaegerthanks clarkb for reviewing18:22
*** jamesmcarthur has joined #openstack-infra18:23
*** mriedem has quit IRC18:26
openstackgerritMerged openstack-infra/project-config master: Retire openstack-ansible-os_monasca-ui  https://review.openstack.org/61721218:26
pabelangerclarkb: ah yes, you are right. I'll fix up shortly18:27
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Create publish-zuul-python-branch-tarball job  https://review.openstack.org/61863318:29
pabelangerclarkb: AJaeger: ^should address the comments18:29
openstackgerritTobias Henkel proposed openstack-infra/zuul master: Display ref instead of NA for time triggered items  https://review.openstack.org/61925218:30
*** trown is now known as trown|lunch18:33
*** shardy has quit IRC18:34
*** shardy has joined #openstack-infra18:36
*** mriedem has joined #openstack-infra18:37
*** ralonsoh has quit IRC18:40
*** ahosam has joined #openstack-infra18:53
dmsimardI have a weird issue trying to use the synchronize module with zuul.project.src_dir18:55
dmsimardoh, wait, I think I know18:56
*** shardy has quit IRC18:56
openstackgerritMerged openstack-infra/project-config master: Create publish-zuul-python-branch-tarball job  https://review.openstack.org/61863318:57
prometheanfireglean review please (for more networkd stuff https://review.openstack.org/610064 https://review.openstack.org/61010718:58
prometheanfireianw: clarkb you think glean should use a more generic install.py?18:59
*** xek has quit IRC19:00
clarkbprometheanfire: I need more context than that19:00
*** apetrich has quit IRC19:00
prometheanfireclarkb: k, I commented https://review.openstack.org/618964 but generally, should we have each network backend have a --use-foo option19:01
clarkbprometheanfire: ideally we wouldn't I think this is special beacuse its a transition point for fedora and rhel19:02
*** jamesmcarthur has quit IRC19:02
clarkbprometheanfire: an alternative could be to just assume nm if rhel version >=8 or fedora version >=29 ?19:03
prometheanfireclarkb: should the nm support also support other distros?19:03
prometheanfireor check /etc/os-release19:03
clarkbprometheanfire: the related dib changes seem to imply that it should. But I left a note about maybe not implying that since glean only does it for red hat19:03
prometheanfireya, guess that should be answered first (rhel/fedora only or not)19:04
clarkbprometheanfire: looking at https://review.openstack.org/#/c/610064/1/glean/cmd.py the files it wouldn't check for are the slave interfaces and vlan logical interfaces?19:05
clarkbprobably good enough for now to assume that if the main interface is configured so are the other bits (like vlan)19:05
*** xek has joined #openstack-infra19:08
prometheanfireI think it doesn't care what the name is, iirc, name should be eth4-vlan26 and find the netdev file19:09
prometheanfiresee liberty.networkd.network.out for example19:10
clarkbright but that check you are modifying is checking if glean ahs already written things so that it can noop19:11
clarkbso it won't see that eth4-vlan26 is there but it will see eth4 and noop I think19:11
clarkbsee comment on https://review.openstack.org/#/c/610107/2 that breaks an existing behavior I think19:12
*** jamesmcarthur has joined #openstack-infra19:12
prometheanfireclarkb: it just noops the single interface though19:12
clarkbprometheanfire: ah so it will write the others19:12
clarkbprometheanfire: we probably do wnat to fix that properly then19:12
prometheanfireya, continue in the loop19:12
prometheanfireall that I was doing was removing 'gentoo' refrences19:12
clarkbya its an improvement :)19:13
*** jamesmcarthur has quit IRC19:13
prometheanfirethe only 'proper' fix would be to search ALL the netdev and network files for Name=iname19:14
*** jamesmcarthur has joined #openstack-infra19:14
prometheanfireI guess that's possible, but may also give false positives19:14
*** apetrich has joined #openstack-infra19:15
clarkbprometheanfire: thinking about nm as a general contruct it might be a good idea to get a general sense for what the configuration options are on the various currently supported distros19:17
clarkbprometheanfire: then maybe we can see where the venn diagram has overlap and reduce the total number of tools to support19:17
prometheanfireya19:17
clarkbnetworkd should cover ubuntu and gentoo and I think suse for a future point (not sure where that line is) then it sounds like red hat is all nm for the future19:17
clarkbthen maybe the sysconfig and /etc/network/interfaces stuff can be "legacy" and go away19:18
prometheanfiredoesn't cover gentoo-openrc, but does gentoo-systemd19:18
prometheanfirekinda sad rh didn't go with networkd19:18
clarkbor maybe nm will work everwhere as another alternative19:18
clarkbits a bit heavyweight and definitely not my first choice but learning a single thing is probably less effort than paging in 6 tools19:19
prometheanfireya, nm should work on gentoo (both)19:19
prometheanfireya, nm isn't my first choice either19:19
clarkbwhats weird is the rh nm config seems to be write out sysconfig then set use-nm to true19:20
clarkband I guess it configures nm from that rather than direct configuration?19:20
prometheanfirehell if I know19:20
prometheanfireovercomplicated imo19:20
clarkbprometheanfire: before I go hunt down food, did you see my -1 about the dns thing? does it make sense?19:21
funginm is definitely a working option for ubuntu/debian, fwiw19:21
clarkbfungi: ya as well as suse19:21
prometheanfireya19:21
prometheanfireclarkb: ya, I'll see about adding a check19:21
cdentclarkb: got it down to 4 minutes. how many cores does the vm used by the base job typically have?19:21
clarkbnm works everywhere because laptops19:21
clarkbcdent: 819:21
cdentgreat, thanks19:22
fungiwell, the laptop argument is also valid for anything with hotplugged interfaces19:22
fungiwhich virtual machines tend to be able to make use of more than physical machines19:22
fungii mean, not that ifupdown doesn't also support hotplugging network interfaces19:23
clarkbfungi: I guess? the only time I hot plug interfaces other than on my laptop is plugging in network devices so that I can configure them from desktop19:23
clarkbmy VMs all come up with the interface they will live with until recycled19:23
fungilonger-lived pets in some virtual environments may benefit from users dynamically adding new networks/ports, but yeah it does seem like a bit of a corner case19:24
fungiand i agree, i'd personally rather _not_ have new network ports magically autoconfigured for me without some manual intervention19:24
openstackgerritMerged openstack-infra/nodepool master: move 01-nodepool-setup to a later phase  https://review.openstack.org/61018819:29
clarkbok gonna dig up food now. I'm likely to try and file expense reports after lunch but let me know if there are other stacks of changes that could use review19:29
clarkbianw: ^ particularly happy to talk about your changes I've reivewed as I've left notes here and there19:29
openstackgerritMatthew Thode proposed openstack-infra/glean master: write dns info to networkd  https://review.openstack.org/61010719:30
prometheanfireclarkb: ^19:30
*** e0ne has joined #openstack-infra19:33
*** bobh has joined #openstack-infra19:35
*** dklyle has quit IRC19:38
*** jamesmcarthur has quit IRC19:40
*** trown|lunch is now known as trown19:44
*** jamesmcarthur has joined #openstack-infra19:44
*** dklyle has joined #openstack-infra19:46
*** apetrich has quit IRC20:01
openstackgerritDoug Hellmann proposed openstack-infra/project-config master: add jobs to publish library from governance repo  https://review.openstack.org/61934720:05
*** cdent has quit IRC20:18
ianwclarkb / prometheanfire: i haven't caught up yet, but i was planning on switching centos and fedora to NM since it seems to work20:21
fungi(relative?) simplicity ftw!20:22
ianwand is the supported thing really20:22
*** ahosam has quit IRC20:24
*** jamesmcarthur has quit IRC20:25
clarkbianw: I think the concern is whether we want a bunch oc switches for that --use-nm20:27
clarkbinstead maybe we just use nm always20:27
*** bobh has quit IRC20:29
ianwthis is a possibility; however my main concern was that we put it in the gate and something happens, and with this approach we just flip a flag and rebuild the image, or even create a duplicate image with it turned on for testing20:29
*** jamesmcarthur has joined #openstack-infra20:29
ianwit seems all the testing is working, just because i've added centos+f28+f29 to the same job it's too long and times out20:30
clarkbour testing is pretty good for all plattorms but rax20:31
clarkbmaybe rely on that and make it the behavior after testing on rax?20:31
ianwi meant more that the jobs ontop don't like it, rather than not booting20:31
ianwit's in a odd place because it's using the ifcfg-rh plugin, so essentially the guts of the configuration are the same as the network-scripts path20:35
ianwrather than a completely new path20:36
clarkbya20:37
*** bobh has joined #openstack-infra20:39
ianwdoes it work the same on debuntu?  i think there's also a plugin there to read the existing files?20:41
clarkbplugin ifupdown20:42
clarkblooks like yes20:42
clarkbalso ifcfg-suse20:42
ianwright, so that maybe suggests that (if someone cares to work on those) keeping one implementation per distro but with a flag for NM is a saneish thing20:45
*** jamesmcarthur has quit IRC20:46
clarkbya I dont think it is the worst thing but do want to avoid a use wickd and use networkd and use whatver as a bunch of options20:48
clarkbI'm happy to say nm is default on rh and works everywhere else so is a bit of a special case20:48
*** jamesmcarthur has joined #openstack-infra20:49
clarkbianw: fwiw I tried to fix pep8 in your stacks then lef comments on things that were more opinion20:49
funginote that networkmangler has a number of dependencies which we may be avoiding installing so could bloat images and result in additional running daemons20:54
clarkbre nm maybe add a note to glean about its specialness?20:55
clarkbfungi: ya I'm coming around to the current stack where it is selectable20:55
ianwclarkb: yep, still wip, docs etc to add :)20:57
fungiif we don't need it on most platforms, i'm fine with the idea of continuing to run leaner there, but would be curious to try an `apt install network-manager` and finding out what it actually ends up pulling in20:57
ianwfungi: yes, for sure -- however it's a very realistic environment to run it; if we do have problems with the networkmanager daemons, then it doesn't bode well for actually useful code20:57
fungilike, are we already installing and starting dbus everywhere anyway?20:57
ianwfungi: yep, you can't even log in without dbus these days20:58
clarkbdbus is required for systemd things20:58
fungioh, right, ugh20:58
* fungi saves his disdain for more appropriate times and places20:58
* prometheanfire would rather just do networkd for most stuff (and something else for non-systemd)21:00
ianwprometheanfire: unfortunately it seems others don't agree -- as i mentioned inline rhel8 (and centos8 eventually) won't have networkd and networkmanger is the thing21:01
ianwan eye on centos8 is why i've taken this on, really21:01
fungiunfortunate they weren't willing to choke down their own dogfood on this21:02
prometheanfirefungi: but their dns stuff is so stable and good21:03
* fungi wouldn't know21:03
*** BOKALDO_ has quit IRC21:05
ianwso yeah, i am sensitive that turning glean into cloud-init defies the entire point, but we can't really ignore the networkmanager side, especially that the scripts are split out into a now deprecated package on f2921:05
*** cloudnull has joined #openstack-infra21:06
*** jamesmcarthur has quit IRC21:09
*** jamesmcarthur has joined #openstack-infra21:13
mriedemhmm, so i just posted my first email to openstack-discuss and am being told it's held for review since i'm not a member, although i confirmed membership earlier today21:13
mriedemis that just a cron or something that isn't showing my updated status?21:13
*** betherly has joined #openstack-infra21:13
clarkbmriedem: I think that would indicate an issue with your subscription. Maybe you subscribed with a different email address?21:14
fungimriedem: i'll check the logs, but no it should have just worked21:14
clarkbfungi: ^ has been moderating and may be able to see details21:14
clarkbianw: ya thinking a bit more I think probably the place to start is with --use-nm as you've done it. That gets us future rhel support. Then if anyone is interested in adding debuntu or suse or gentoo nm support that can come later. Separately if we decide that networkd is also a special universal like thing maybe we add a --use-networkd flag and do similar21:15
mriedemhmm, i sent as mriedem.os@gmail.com21:15
fungimriedem: the moderation message definitely said "Reason:  Post by non-member to a members-only list" so i'll check the subscribe log21:15
clarkbianw: I imagine we'll want to further refactor things since a lot of that code is still messy (though hopefully the debuntu paths are a little bit happier after my refactorings)21:16
fungialso, that error makes me think of the members only jackets from the 80s21:16
corvusmriedem: the moderation queue says the message was from mriedemos@gmail.com  (cc fungi)21:16
mriedemi believe members only jackets are back21:16
corvusnote the lack of '.'21:16
mriedemcorvus: hmm, ok21:17
corvusand that's what's in the 'From:' header21:17
fungiyep, i concur21:17
fungithe subscription confirmation includes the . in the user part of the address21:17
clarkbianw: all that to say I think the current stack (ignoring bugs with implementation) is likely the path forward21:17
mriedemok changing to mriedemos@gmail.com21:18
fungii guess those are different aliases for the same account?21:18
clarkbprometheanfire: I think your test cases needed to be updated on https://review.openstack.org/#/c/610107/2..321:19
clarkbprometheanfire: the new cases always set the dns settings21:19
fungimriedem: but yes, mailman has no way to know they're the same account since they're distinctly different addresses21:19
mriedemok changed,21:19
fungii'm happy to approve that one through the moderation queue for you anyway so you don't need to resend21:19
mriedemand i can confirm i was using mriedemos@gmail.com with the other lists...21:19
mriedembrowser must have auto-filled21:19
fungifun times!21:19
mriedemfungi: thanks21:20
fungidone!21:20
prometheanfireya21:21
prometheanfireclarkb: I noticed that21:21
prometheanfireclarkb: not actually sure what to do there21:23
clarkbprometheanfire: you just need to update your new test cases to remove the dns stuff21:23
clarkbsince we shouldn't set dns on those because skip dns is set21:23
*** betherly has quit IRC21:23
prometheanfireclarkb: shouldn't there be a test with dns enabled though (along with disabled)?21:24
prometheanfireclarkb: if I do that I fail the 'test_glean' test21:24
clarkbprometheanfire: maybe? I don't know if glean tests both paths for all platforms21:24
prometheanfiresince that sets skip_dns to false21:24
prometheanfireso, since it's testing both true and false, and I only have one template file one or the other will always fail21:26
clarkbprometheanfire: you may need different fixtures in that case21:26
clarkbhave a .dns fixture and check if that exists when skip_dns == True?21:26
prometheanfireianw did the test_glean_systemd one21:26
clarkbthe thing that makes this weird is the dns stuff going into the per interface files and not just resolv.conf21:26
prometheanfireya, because dns is per interface (an interface property)21:27
prometheanfirethat's how it's delivered via the cloud_init info21:27
prometheanfireand supported in networkd21:27
*** bobh has quit IRC21:28
clarkbits actually really weird to think of dns that way (beacuse dns is a layer 7 service for name lookups and interfaces are layer 3 (ish) constructs)21:28
clarkbbut ya21:29
prometheanfireit is odd, but any service could be the same (just listening or sending requests on that interface)21:30
prometheanfirenot normal for things like dns though, but it is how networkd expects it21:30
prometheanfireI think I have to make a 'if distro == networkd: set skip_dns to false'21:32
*** e0ne has quit IRC21:32
prometheanfireeven on the skip dns test21:32
clarkbprometheanfire: no you can load a different fixture if skip_dns is set to false21:32
prometheanfireclarkb: as part of output_filename?21:33
clarkbprometheanfire: at line 195ish https://review.openstack.org/#/c/610107/3/glean/tests/test_glean.py check if there is a output_path + '.dns' file on disk and use that as output_path instead if it exists and skip_dns is false21:34
clarkbthen you have the two versions of that file, with and without the dns data, in the fixtures dir21:34
prometheanfireclarkb: yep, that's where I was going with it (as you led me there) :D21:34
prometheanfireclarkb: means copying a ton of tests though, but sure21:38
clarkbjust those for networkd/gentoo right?21:39
prometheanfireclarkb: with more if statements it could be limited to that (networkd is the 'distro' here, and rax-iad is the only scenario that it's hit in)21:39
clarkbprometheanfire: I think if the .dns file doesn't exist use the other file21:40
clarkbthen you only have to copy them for gentoo/networkd as those are the only cases that will differe21:40
clarkb*differ21:40
ianwcan you do something like i did with networkd and have a %DNS% entry in the file and patch it in or out?21:40
prometheanfireok21:40
*** xek has quit IRC21:40
prometheanfireianw: where is that?21:40
ianwprometheanfire: https://review.openstack.org/#/c/618964/7/glean/tests/test_glean.py line 20421:41
ianwsorry i have to run out for a little, will come back and look properly in a bit21:42
*** ahosam has joined #openstack-infra21:42
cmurphyianw: fungi if you have time and interest i have a few easy noop patches sitting in my queue that i'd like to clear out https://review.openstack.org/601464 https://review.openstack.org/581389 https://review.openstack.org/581697 https://review.openstack.org/582153 https://review.openstack.org/585692 https://review.openstack.org/577521 https://review.openstack.org/58133521:43
prometheanfirewill look, but I'd rather have a fixture for dns than have the test to special things (test the code, not the test)21:44
*** agopi is now known as agopi|brb21:46
*** trown is now known as trown|outtypewww21:48
*** kjackal has quit IRC21:48
*** jcoufal_ has quit IRC21:50
openstackgerritMatthew Thode proposed openstack-infra/glean master: write dns info to networkd  https://review.openstack.org/61010721:50
*** yamamoto has joined #openstack-infra21:50
prometheanfireclarkb: let's try this ^21:50
prometheanfirepasses tests locally at least21:51
*** agopi|brb has quit IRC21:51
*** betherly has joined #openstack-infra21:55
openstackgerritColleen Murphy proposed openstack-infra/system-config master: Update puppetlabs-postgresql to 4.8.0  https://review.openstack.org/55899521:57
openstackgerritColleen Murphy proposed openstack-infra/system-config master: Make ask/ask-staging numeric  https://review.openstack.org/55899221:57
openstackgerritColleen Murphy proposed openstack-infra/system-config master: Minor fork puppet-solr  https://review.openstack.org/55917821:57
cmurphyclarkb: fungi there are a few changes between ianw and i that depend on that stack ^21:57
*** eumel8 has quit IRC21:59
mnaserany reported issues at rax?21:59
*** mriedem has quit IRC21:59
mnaserjust had a job timeout there and zuul couldnt even connect to get logs21:59
mnaserand im trying to know if we broke a vm terribly :) or if its just an issue22:00
clarkbmnaser: could be the duplicate IP issue22:00
*** betherly has quit IRC22:00
prometheanfiremnaser: I haven't seen emails go out (incident management)22:00
mnaseri see22:00
mnaserhttp://zuul.openstack.org/stream/36278e5399614b8bb4c3e089da25aabf?logfile=console.log (warning really really big log)22:01
mnaser Timeout exception waiting for the logger. Please check connectivity to [104.130.216.193:19885]22:01
*** betherly has joined #openstack-infra22:02
clarkbprometheanfire: we've seen evidence of duplicate IP addrs in rax22:02
clarkbprometheanfire: cloudnull followed up on some of them week before summit but its possible whatever is causing it has caused more of them22:02
clarkbprometheanfire: if you are able checking if 104.130.216.193 is used in multiple places would be the check aiui22:02
prometheanfireah, I can bug someone, but he's probably off right now22:04
prometheanfiremsg'd him22:06
*** kgiusti has left #openstack-infra22:06
*** betherly has quit IRC22:06
gmanncorvus: clarkb frickler do you know why it still pick bionic for py3 job - https://review.openstack.org/#/c/618169/11/.zuul.yaml@14822:08
gmannit should pick xenial when py3 job running on stable/rocky22:08
clarkbgmann: I can take a look in a few minutes. The zuul info dir with the inventory file should show you the job inheritance path22:08
*** rcernin has joined #openstack-infra22:09
gmannclarkb: thanks, i checked that and job inheritance looks from stable/rocky all the way - http://logs.openstack.org/76/618176/1/check/tempest-full-py3/96b7329/zuul-info/inventory.yaml22:09
*** priteau has quit IRC22:16
*** agopi|brb has joined #openstack-infra22:19
ianwcmurphy: ok, that pre-existing comment in 581335 is now one of my favourites22:22
cmurphyheh22:24
cmurphythanks for the reviews ianw22:33
openstackgerritMerged openstack-infra/puppet-mailman master: Add beaker tests for custom providers  https://review.openstack.org/60146422:35
*** rpioso is now known as rpioso|afk22:36
clarkbgmann: ok this is weird22:38
*** roman_g has quit IRC22:40
*** roman_g has joined #openstack-infra22:40
clarkbgmann: that ran against 610977,2 but 610977,4 is current. Any chance the changes to make the nodeset explict there fix it?22:41
clarkbgmann: I'm wondering if openstack-single-node nodeset is not looked up in a branch specific context22:41
*** imacdonn has quit IRC22:42
clarkbhrm except that openstack-single-node nodeset appaers to have always been xenial not bionic22:42
*** imacdonn has joined #openstack-infra22:42
gmannyes, that is xenial always22:42
gmanni can check for explicit  nodeset. but what difference we have between tempest-full and tempest-full-py3 is these extra branches setting - http://git.openstack.org/cgit/openstack-infra/openstack-zuul-jobs/tree/zuul.d/project-templates.yaml#n140122:44
gmannwhich does not exclude stable/rocky22:45
*** roman_g has quit IRC22:45
clarkbhttps://review.openstack.org/#/c/618169/4/.zuul.yaml is the version of tempest change that was tested there22:45
clarkbwhich is quite a bit older than current patchset. I'm trying to read through it and see if anything there may explain it22:45
clarkbgmann: https://review.openstack.org/#/c/611572/1/.zuul.yaml that is why22:47
*** roman_g has joined #openstack-infra22:47
gmannclarkb: ah i see, and that has not set the branch thing for py3 job22:47
clarkbgmann: the py3 job depends on devstack-tempest and since tempest is branchless there it takes that as is22:48
gmannbut why it did not take latest version 618169/1122:48
clarkbthat is a different change22:48
clarkbhttps://review.openstack.org/#/c/611572/1 is the change22:49
openstackgerritMerged openstack-infra/puppet-log_processor master: Add beaker tests  https://review.openstack.org/58169722:49
openstackgerritMerged openstack-infra/puppet-statusbot master: Add beaker tests  https://review.openstack.org/58215322:49
openstackgerritMerged openstack-infra/puppet-openstackci master: Update Gemfile for Zuulv3  https://review.openstack.org/57752122:49
openstackgerritMerged openstack-infra/puppet-openstackci master: Add beaker tests for nodepool  https://review.openstack.org/57752222:49
openstackgerritMerged openstack-infra/puppet-subunit2sql master: Fix subunit2sql install guard for idempotency  https://review.openstack.org/58133522:49
openstackgerritMerged openstack-infra/puppet-elastic_recheck master: Add beaker tests  https://review.openstack.org/58138922:49
clarkbDoes that change need to exist? there should be the one chagne in tempest (618169) and the one change in devstack (610977) and the nova change that tests them22:49
clarkbgmann: I think order should be nova change depends on tempest change (618169) depends on devstack change (610977). I don't know where 611572 fits in22:50
gmannthat was old one for old PS.22:50
gmannwith current testing that is not in pic22:50
gmannthat came there as it picked 618169/4 not 618169/1122:51
clarkbok then I think you just need to rerun your tests and check again22:51
gmannand in same run, tempest-full did pick the right one 618169/11  - http://logs.openstack.org/76/618176/1/check/tempest-full/9fe23a5/zuul-info/inventory.yaml22:51
*** betherly has joined #openstack-infra22:51
gmannclarkb: ok let me re run22:52
clarkbgmann: the patchset for 618169 in that inventory is 4 not 1122:52
clarkboh for tempest full22:52
gmannyeah22:52
gmannit was strange in same run different job picked different PS22:53
clarkbgmann: the py3 job ran a week ago22:53
clarkbI think the one with 11 ran after a recheck22:53
clarkbgmann: http://logs.openstack.org/76/618176/1/check/tempest-full-py3/45fac7f/ that is py3 that ran recently22:54
clarkband it ran on xenial so I think it is working22:54
gmannclarkb: ahh, got it.22:57
*** betherly has quit IRC22:57
gmannmy bad i picked up wrong run seems22:57
prometheanfirecloudnull: we've been talking about networkd on rhel8 as well23:00
openstackgerritClark Boylan proposed openstack-infra/glean master: write dns info to networkd  https://review.openstack.org/61010723:00
clarkbprometheanfire: ^ fyi I just fixed the pep8 issue23:00
prometheanfireclarkb: yep, thanks23:01
prometheanfirewas just going to fix it, but less for me is better for me :D23:01
gmannclarkb: thanks, it's my early morning side effect seems.23:01
* prometheanfire hates rhel more than normal right now23:02
* prometheanfire blames oracle23:02
*** diablo_rojo has quit IRC23:02
clarkbprometheanfire: at the summit it came up that we are so afraid of changing other peoples patches that I figure fixing formatting issues like this is a good way to start pushing towards not being afraid ofthat :)23:02
prometheanfiremeh, at this point I don't care so much, guess that means I should touch tonyb's patch then23:04
clarkbconsidering round trip time on CI is a cost being able to make CI green trivially so that when ianw or tonyb wake up its all ready to go is probably a good thing23:05
*** betherly has joined #openstack-infra23:08
ianwwe could maybe do like the debian low-threshold nmu page, and somehow have people opt in to explicitly say "i'm happy for you to update my patch"23:11
ianwit doesn't really change anything, but people might feel better if the author has explicitly said "i won't get mad about it" :)23:12
ianwclarkb: you really got me thinking now on != and = behaviour on that skip_when patch ...23:13
clarkbianw: it was just weird to me that the skip would match if the value was unset23:14
clarkbmakes sense for != becuase unset can't equal a value23:14
clarkbbut for = should be false I think23:14
ianwactually i think the opposite, see my comment :)23:15
ianwi think a unit test here would be most helpful to make sure the combinations are coming out as expected23:16
*** rlandy is now known as rlandy|bbl23:17
ianwwhich of course means a yak shaving adventure on unit testing elements, i'm not sure we do that23:18
*** betherly has quit IRC23:18
*** ahosam has quit IRC23:19
clarkbanother option is for it to be an error if no value is set23:19
clarkbthen it isn't ambiguous but slightly more work on element authors23:20
clarkbI think that would work for your existing use case without any changes to the nm stuff23:20
ianwyeah, i could buy that too23:24
*** rm_work has quit IRC23:33
*** rm_work has joined #openstack-infra23:33
*** florianf has quit IRC23:38
*** slaweq has quit IRC23:45
*** lbragstad is now known as lbragstad_turkey23:46
*** betherly has joined #openstack-infra23:48
*** zul has quit IRC23:51
*** betherly has quit IRC23:53
clarkbhttp://logs.openstack.org/85/605585/7/check/system-config-run-docker/62b9e11/job-output.txt.gz#_2018-10-31_20_34_58_500426 is what our iptables ruleset looks like with docker installed23:55
*** lbragstad_turkey has quit IRC23:56
ianwclarkb: oh, good -- yeah that was what i looked at initially and just threw up my hands :)23:57

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!