Tuesday, 2019-05-14

openstackgerritStas Egorov proposed airship/in-a-bottle master: Support for new hyperkube source in promenade  https://review.opendev.org/65795400:22
*** Nishant__ has quit IRC01:06
*** jamesgu_ has quit IRC05:56
*** jamesgu has quit IRC05:56
*** jamesgu has joined #airshipit05:57
*** jamesgu_ has joined #airshipit05:57
*** jamesgu has quit IRC06:19
*** jamesgu_ has quit IRC06:19
*** jesusaur has quit IRC07:13
*** georgk has quit IRC07:26
*** georgk has joined #airshipit07:27
*** pgaxatte has joined #airshipit07:36
*** pkaralis has joined #airshipit07:55
*** happyhemant has joined #airshipit08:31
*** jesusaur has joined #airshipit09:29
*** roman_g has quit IRC09:53
*** mbuil has joined #airshipit09:55
*** nmimi has quit IRC10:00
*** roman_g has joined #airshipit10:05
*** lemko has joined #airshipit10:58
*** hemanth_n has joined #airshipit12:35
mbuilhey guys, I am trying to deploy multinode airship (./gate.sh multinode_deploy). n0 deploys correctly but n1, n2, n3 don't. After a while, I can see in the drydock logs:  "node.py:2417:start Polling node n2 status attempt 225 of 270: Failed deployment". Same error for n1 and n3. I guess it is a MAAS problem but no clue how to debug it further. Any hint about how could I debug MAAS? MAAS containers show no interesting logs12:36
*** howell has joined #airshipit12:38
*** kskels has joined #airshipit12:50
*** howell has quit IRC12:50
*** alexanderhughes has joined #airshipit12:51
*** bh526r has joined #airshipit12:56
*** sgrasley has joined #airshipit13:02
openstackgerritAlexander Hughes proposed airship/pegleg master: Support b64 encoding of passphrase catalog  https://review.opendev.org/65838013:12
*** kranthikirang has joined #airshipit13:21
*** kguttikonda has joined #airshipit13:24
*** kranthikirang has quit IRC13:27
*** ab2434_ has joined #airshipit13:37
evgenylmbuil: I would suggest to try to access MaaS UI and check the status of the nodes. Also if you check drydock-api logs (for both pods) there may be some useful information on the failure.13:52
mbuilevgenyl: do you know how to access the maas ui? Is it running in any of the containers?13:53
evgenylTry to check if there are some Maas region UI available in k8s services, run `kubectl get svc --all-namespaces`.13:57
*** nmimi has joined #airshipit13:59
*** howell has joined #airshipit13:59
openstackgerritAnthony Bellino proposed airship/shipyard master: Update Airflow logrotate logic  https://review.opendev.org/65603314:02
*** lemko has quit IRC14:08
*** jamesgu_ has joined #airshipit14:12
*** jamesgu has joined #airshipit14:12
*** aaronsheffield has joined #airshipit14:18
openstackgerritMerged airship/armada master: [v2 docs] Overhaul wait API  https://review.opendev.org/63644014:19
*** howell has quit IRC14:25
*** Nishant__ has joined #airshipit14:26
*** AvinashBR has joined #airshipit14:27
*** michael-beaver has joined #airshipit14:38
*** howell has joined #airshipit14:59
*** kskels has quit IRC15:00
openstackgerritAlexander Hughes proposed airship/pegleg master: [WIP][DNM] Testing genesis bundle error  https://review.opendev.org/65909715:11
*** AvinashBR has quit IRC15:13
mattmceuenGM/GE everyone - here's the agenda for our meeting in ~45min:  https://etherpad.openstack.org/p/airship-meeting-2019-05-1415:18
mattmceuenPlease add anything to the agenda that you'd like to discuss today15:18
openstackgerritMerged airship/spyglass-plugin-xls master: Add .gitreview and .zuul.yaml  https://review.opendev.org/65866815:20
openstackgerritMatthew Carter proposed airship/shipyard master: Create new Action class for Shipyard  https://review.opendev.org/65638415:27
openstackgerritMerged airship/spyglass master: Implements PBR for packaging  https://review.opendev.org/65348015:29
openstackgerritEvgeniy L proposed airship/treasuremap master: [WIP] Add Airship-in-a-Bottle site manifests  https://review.opendev.org/65690015:34
openstackgerritRick Bartra proposed airship/promenade master: [WIP] templates: separate genesis and join sources  https://review.opendev.org/65842115:39
openstackgerritMatthew Carter proposed airship/shipyard master: Refactor Shipyard to use new Action class  https://review.opendev.org/65657815:45
*** pgaxatte has quit IRC15:47
*** kskels has joined #airshipit15:52
*** openstackgerrit has quit IRC15:54
*** openstackgerrit has joined #airshipit15:56
openstackgerritIan Pittwood proposed airship/spyglass-plugin-xls master: Fix source directory name  https://review.opendev.org/65910915:56
openstackgerritStas Egorov proposed airship/promenade master: Fixed for hyperkube 1.12  https://review.opendev.org/65579815:57
*** ian-pittwood has joined #airshipit15:59
mattmceuen#startmeeting Airship16:00
openstackMeeting started Tue May 14 16:00:38 2019 UTC and is due to finish in 60 minutes.  The chair is mattmceuen. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
*** openstack changes topic to " (Meeting topic: Airship)"16:00
openstackThe meeting name has been set to 'airship'16:00
mattmceuenHowdy everyone16:00
openstackgerritStas Egorov proposed airship/promenade master: New source for hyperkube binary  https://review.opendev.org/65795316:00
mattmceuen#topic Rollcall16:00
*** openstack changes topic to "Rollcall (Meeting topic: Airship)"16:00
*** cfriesen has joined #airshipit16:00
dwalto/16:01
mattmceuenHere's our agenda for today -- let's give folks a few minutes to add things and filter into the meeting room:  https://etherpad.openstack.org/p/airship-meeting-2019-05-1416:01
arunkanto/16:01
evgenylHi!16:01
michael-beavero/16:01
cfrieseno/16:02
mattmceuenOk!  let's get started16:03
mattmceuen#topic Multi-distro support16:03
*** openstack changes topic to "Multi-distro support (Meeting topic: Airship)"16:03
mattmceuenThe effort to add opensuse images is progressing:     https://review.opendev.org/#/q/status:open+branch:master+topic:airship_suse16:03
mattmceuenarunkant:  this one needs response to some review feedback: https://review.opendev.org/#/c/639495/16:03
arunkantmattmceuen: I will address them today . Can we get some more core reviewers attention to review others as well.16:05
mattmceuenthere was also discussion on this OSH thread, essentially proposing a slightly different naming convention: https://review.opendev.org/#/c/636229/16:05
mattmceuenarunkant I am fine just saying "we have an approved spec, let's do it" - but I'm interested in your feedback on it16:06
arunkantmattmceuen: Yes, I noticed that and am not convinced that we need to rename these files as previous naming convention works better as it maps to image tag name suffix (opensuse_15)16:07
mattmceuen++16:07
mattmceuenif you're happy with it I'm happy with it, I will reply to his comment16:08
mattmceuenAgree - let's please get review on these PS today, I think they're ready16:08
arunkantmattmceuen: So I have added my comment in that review as well. Yes, we want to merge this quickly as its blocing many things on our side. We can certainly revisit to be consistent if it changes on openstack-helm side16:09
mattmceuenSounds like a plan.  Anything else on the multi-distro topic, folks?16:09
mattmceuen#topic Open Infrastructure Summit videos are up16:10
*** openstack changes topic to "Open Infrastructure Summit videos are up (Meeting topic: Airship)"16:10
arunkantalso this review is failing because of io issue https://review.opendev.org/#/c/638301/16:10
mattmceuenThis one is self-explanatory :)16:10
mattmceuenhttps://www.openstack.org/videos/summits/denver-201916:10
roman_gthere is -1 from Thomas Bechtold @suse on one of the patches16:10
mattmceuenThere were many good airship talks (and many good non-airship talks) at the summit, I suggest folks catch up on topics they're interested in16:10
mattmceuenroman_g: yep that's the one that I'm going to reply to, since we already have an approved spec, I don't think we need to change our naming convention now16:11
roman_gtnx16:11
mattmceuenty16:11
arunkantraomn_g: yes noticed that and its talking about the proposed change in openstack-helm side and will clarify with Thomas about it16:11
mattmceuenIf you guys have any particularly favorite talks from the summit, let's share the links in the chat room, and maybe aggregate them for next week's meeting16:12
roman_gvideos: finally 1080p16:13
arunkantmattmceuen: All of orecent pen deckhand reviews are failing .. https://review.opendev.org/#/q/project:airship/deckhand+status:open16:13
mattmceuenoh awesome!  1080p - I have only seen one on my phone so far (and really only listened) so I didn't notice that yet roman_g16:13
arunkantmattmceuen: All of recent open deckhand reviews are failing .. https://review.opendev.org/#/q/project:airship/deckhand+status:open16:13
mattmceuenI saw that too this morning arunkant - need to figure that out.  Is anyone aware of what the deckhand build problem is?16:14
mattmceuen#topic Deckhand gates are failing16:14
*** openstack changes topic to "Deckhand gates are failing (Meeting topic: Airship)"16:14
arunkantmattmceuen: I see these io errors at various steps . http://logs.openstack.org/01/638301/22/check/deckhand-integration-docker-py35-opensuse/022edf1/primary/pod-logs/openstack/keystone-bootstrap-5dpkk/init.txt16:15
arunkantSimilar error is for ubuntu jobs as well.16:16
dwaltarunkant: Looks like Dan Crank may be working to resolve this https://review.opendev.org/#/c/655787/16:18
mattmceuenah thanks dwalt- that certainly seems like a smoking gun16:18
dwaltI know he has had some laptop trouble lately, so he may not be able to carry that forward right now16:19
dwaltI'll follow up with him16:19
mattmceuenthanks dwalt16:19
dwaltnp :)16:19
mattmceuen#topic Airship Design Meetings16:19
*** openstack changes topic to "Airship Design Meetings (Meeting topic: Airship)"16:19
mattmceuenJust in case you missed it - Rodolfo sent out new invites for twice-per-week, hour and a half design meetings to help us get through design of the airship 2.0 scope16:20
mattmceuenTuesdays and Thursdays, starting three hours before this Airship IRC team meeting16:20
mattmceuenSo you're welcome to join that next time if you missed it this time :)16:21
roman_gI forgot to ask again. Matt, IPv6 is in scope for 2.x, or not?16:21
mattmceuenIt hasn't been called out as a scope item, but I believe it should just be a matter of configuring k8s, openstack, calico etc correctly16:22
openstackgerritIan Pittwood proposed airship/spyglass master: [WIP] Separate plugins from Spyglass  https://review.opendev.org/65355516:22
openstackgerritStas Egorov proposed airship/promenade master: New source for hyperkube binary  https://review.opendev.org/65795316:23
*** MattCoachCarter has joined #airshipit16:23
roman_gSo far we have Argo + ClusterAPI + Ironic + airshipctl only?16:23
mattmceuenplus reimplementing Pegleg as a pluggable module for airshipctl, and a POC of Armada Operator -- for memory, there might be more16:24
mattmceuenAlright - moving on:16:26
mattmceuen#topic WIP Airship Governance16:26
*** openstack changes topic to "WIP Airship Governance (Meeting topic: Airship)"16:26
openstackgerritIan Pittwood proposed airship/spyglass-plugin-xls master: Fix source directory name  https://review.opendev.org/65910916:26
openstackgerritIan Pittwood proposed airship/spyglass-plugin-xls master: Various bug fixes for plugin separation  https://review.opendev.org/65911616:26
mattmceuenJust a reminder that there is a patchset up for Airship governance - review welcome: https://review.opendev.org/#/c/653865/16:26
mattmceuenthat's all I have on that16:27
mattmceuen#topic Requests for Review16:27
*** openstack changes topic to "Requests for Review (Meeting topic: Airship)"16:27
evgenylThe plans has not been updated for a quite some time.16:27
mattmceuenevgenyl yes I'm hoping a change will be pushed today16:27
mattmceuenSo provide feedback tomorrow ;-)16:28
evgenylOh, I see :)16:28
mattmceuenIn addition to the opensuse patchsets:     https://review.opendev.org/#/q/status:open+branch:master+topic:airship_suse16:28
mattmceuenany additional requests for review today?  Anybody blocked by needed review?16:28
*** Talion has joined #airshipit16:29
mattmceuen#topic Roundtable16:29
*** openstack changes topic to "Roundtable (Meeting topic: Airship)"16:29
mattmceuenAnything else on your mind team16:30
mattmceuenin that case -- I will NOT give you a half hour of your time back16:30
mattmceuenInstead I will ask -- please review now! :) https://review.opendev.org/#/q/status:open+branch:master+topic:airship_suse16:30
mattmceuenThanks everyone for your time - have a great week16:31
dwaltthanks everyone16:31
roman_gthank you16:31
mattmceuen#endmeeting16:31
*** openstack changes topic to "airshipit.org || General Review Dashboard: https://review.opendev.org/#/q/project:%255Eairship.*+status:open,n,z"16:31
openstackMeeting ended Tue May 14 16:31:30 2019 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:31
openstackMinutes:        http://eavesdrop.openstack.org/meetings/airship/2019/airship.2019-05-14-16.00.html16:31
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/airship/2019/airship.2019-05-14-16.00.txt16:31
openstackLog:            http://eavesdrop.openstack.org/meetings/airship/2019/airship.2019-05-14-16.00.log.html16:31
openstackgerritIan Pittwood proposed airship/spyglass master: Upgrade PyYAML from 3.12 to 5.1  https://review.opendev.org/65813316:34
*** cfriesen has left #airshipit16:34
mbuilhey guys, I am trying to deploy airship multinode using today's code and according to drydock, it is deploying the nodes, but through vnc I can see an ugly error in the nodes: https://hastebin.com/xuhipucugu.sql. At some point (I guess after a timeout) Drydock says that the deployment failed. Is anyone experiencing this?16:34
openstackgerritIan Pittwood proposed airship/spyglass master: Upgrade PyYAML from 3.12 to 5.1  https://review.opendev.org/65813316:38
*** hemanth_n has quit IRC16:40
openstackgerritIan Pittwood proposed airship/spyglass master: Removes remaining yapf: disable statements  https://review.opendev.org/65814316:40
openstackgerritArun Kant proposed airship/pegleg master: Support pegleg to run on opensuse leap15 image  https://review.opendev.org/63949516:41
openstackgerritIan Pittwood proposed airship/spyglass master: Adds Safety dependency vulnerability checks  https://review.opendev.org/65885416:43
mattmceuenhmm I don't recognize that one mbuil16:44
mattmceuenFrom the context, do you know what 'datasource' it's talking about16:45
mattmceuen?16:45
openstackgerritIan Pittwood proposed airship/spyglass master: [WIP] Separate plugins from Spyglass  https://review.opendev.org/65355516:46
roman_gMaaS did not configure cloud-init properly?16:46
mbuilmattmceuen: nope, this is the first time I try to deploy airship multinode. I guess something is wrong in the cloudinit package in the ubuntu image that maas installs. I was wondering if you guys have a gate in CI that runs multinode to see if this is something just happening to me16:46
mattmceuenare you using the airship-in-a-bottle multinode script, or a bare metal multinode setup?16:48
roman_ghttps://artifacts.atlantafoundry.com/artifactory/cicd/logs/integration/airship/airship-multinode/205/16:48
roman_glast multinode run: https://artifacts.atlantafoundry.com/artifactory/cicd/logs/integration/airship/airship-multinode/205/16:48
roman_g"=== HUGE SUCCESS ===" - that's what the logs say16:50
openstackgerritAlexander Hughes proposed airship/pegleg master: [WIP]Clean up genesis bundle code  https://review.opendev.org/65909716:50
openstackgerritArun Kant proposed airship/shipyard master: Adding opensuse support in image building of airflow and shipyard  https://review.opendev.org/63622916:52
mbuilmattmceuen: I am using ./gate.sh multinode_deploy in a baremetal node. It creates 5 VMs build, n0, n1, n2 and n3. n0 is successfully deployed16:55
mbuilroman_g: ok, then no idea why my env. is wrong. How often are you running that test?16:55
roman_gthat's run for every change in gerrit16:56
roman_gtry to compare the log and see the difference16:56
roman_gjenkinsfiles are in repo, so you can see sequence of commands there16:56
mbuilroman_g: does it also run "./gate.sh multinode_deploy"?16:57
roman_gsome libraries used in jenkinsfiles are here: github.com/att-comdev/cicd16:57
*** howell has quit IRC16:57
roman_gmbuil: no16:58
roman_g+ sudo ./setup_gate.sh16:58
mattmceuenmbuil: was there any additional cloud-init output, or only what you pasted above?16:58
mbuilmattmceuen: only what I pasted above16:59
*** howell has joined #airshipit16:59
mbuilroman_g, mattmceuen: I see it runs ./gate.sh after ./setup_gate.sh... I'll try that16:59
mbuilI was running ./setup_gate.sh and then ./gate.sh multinode_deploy17:00
roman_ggood17:00
mbuilthanks!17:00
roman_gwelcome17:00
roman_gmattmceuen:17:08
openstackgerritRoman Gorshunov proposed airship/deckhand master: [WIP] Docs build fix  https://review.opendev.org/65912717:08
roman_gneed your help to merge ^^^^^17:08
roman_gwithout merging I can't understand if patch works or not17:09
openstackgerritRoman Gorshunov proposed airship/deckhand master: Docs build fix (#1)  https://review.opendev.org/65912817:09
roman_gmattmceuen: ^^^17:17
openstackgerritStas Egorov proposed airship/promenade master: Fixed for hyperkube 1.12  https://review.opendev.org/65579817:26
openstackgerritStas Egorov proposed airship/promenade master: New source for hyperkube binary  https://review.opendev.org/65795317:26
openstackgerritIan Pittwood proposed airship/spyglass master: Removes remaining yapf: disable statements  https://review.opendev.org/65814317:30
openstackgerritIan Pittwood proposed airship/spyglass master: Adds Safety dependency vulnerability checks  https://review.opendev.org/65885417:38
openstackgerritMerged airship/deckhand master: Docs build fix (#1)  https://review.opendev.org/65912817:40
openstackgerritMerged airship/promenade master: Add removal of temporary folders  https://review.opendev.org/65752117:53
*** bh526r has quit IRC18:05
arunkantroman_g, mattmceuen: Can this be reviewed and possibly merged (deckhand can be done later) https://review.opendev.org/#/q/status:open+branch:master+topic:airship_suse18:27
openstackgerritStas Egorov proposed airship/promenade master: New source for hyperkube binary  https://review.opendev.org/65795318:32
openstackgerritMerged airship/promenade master: Fixed for hyperkube 1.12  https://review.opendev.org/65579818:35
roman_garunkant: would review tomorrow.18:36
arunkantroman_g: okay. Thanks18:37
*** Talion has quit IRC18:38
openstackgerritRoman Gorshunov proposed airship/deckhand master: Docs build fix (#2)  https://review.opendev.org/65913718:42
roman_gmattmceuen: merge #2, please18:43
openstackgerritEvgeniy L proposed airship/treasuremap master: [WIP] Add Airship-in-a-Bottle site manifests  https://review.opendev.org/65690018:47
*** kguttikonda has quit IRC18:55
openstackgerritAlexander Hughes proposed airship/pegleg master: Clean up genesis bundle code  https://review.opendev.org/65909719:25
openstackgerritMerged airship/deckhand master: Docs build fix (#2)  https://review.opendev.org/65913719:31
openstackgerritRoman Gorshunov proposed airship/deckhand master: Docs build fix (#3)  https://review.opendev.org/65914919:53
*** kranthikirang has joined #airshipit19:59
openstackgerritMerged airship/deckhand master: Docs build fix (#3)  https://review.opendev.org/65914920:21
georgkHi all. I need a little support with adding a new networking definition to the seaworthy site. Specifically, I'd like to add an additional simple network for tenant traffic via DPDK NICs20:33
georgkThe network configuration I used is shown here as well as the resulting Drydock error20:35
georgk"Missing variable value" during VLAN refresh in drydock. Has anybody seen this before?20:35
georgkanything missing in my config?20:35
georgkthanks!20:35
georgkforgot to post the logs: https://hastebin.com/omocaxesoh.rb20:53
*** alexanderhughes has quit IRC21:03
*** MattCoachCarter has quit IRC21:11
openstackgerritEvgeniy L proposed airship/treasuremap master: Airskiff: fix bridge name in a neutron config  https://review.opendev.org/65916621:13
howell`/quit21:16
howelloops21:16
*** howell has quit IRC21:16
openstackgerritRick Bartra proposed airship/promenade master: [WIP] templates: separate genesis and join sources  https://review.opendev.org/65842121:25
openstackgerritIan Pittwood proposed airship/spyglass master: [WIP] Schema validation  https://review.opendev.org/65917221:27
*** kranthikirang has quit IRC21:27
openstackgerritAnthony Bellino proposed airship/drydock master: [WIP] OVS-DPDK configuration support  https://review.opendev.org/65810522:30
openstackgerritStas Egorov proposed airship/promenade master: New source for hyperkube binary  https://review.opendev.org/65795322:32
openstackgerritStas Egorov proposed airship/promenade master: New source for hyperkube binary  https://review.opendev.org/65795322:34
*** ab2434_ has quit IRC22:39
*** happyhemant has quit IRC23:01
georgkhi! can anybody please tell me the password of the MAAS GUI?23:03
mattmceuenhe georgk - I was just looking into your other issue, but I'll help you with this one because it's easy :D23:03
mattmceuenone sec23:03
georgkthanks23:04
mattmceuenwhich site is your site based on?23:04
mattmceuen(which treasuremap site, I mean)23:04
georgkthe default seaworthy one23:05
mattmceuenok - I think it should be the value in the ucp_maas_admin_password.yaml file in the airship-seaworthy site, which is password12323:06
* mattmceuen don't tell anyone - it's a secret23:06
mattmceuenlet me know if that doesn't work23:06
georgkthanks!23:07
georgkworks23:07
mattmceuenawesome23:14
georgkregarding the other issue: any pointers at a broken config are highly appreciated23:16
georgkI added an extra network in the AIAB multinode environment without problems, but this is not a baremetal deployment based on the seaworthy site23:17
georgknot == now23:17
mattmceuentracing through the drydock code a bit, it looks like there may be a fabric id that it can't find23:18
mattmceuenbut I don't have complete certainty of that yet, and I'm not sure why that would be23:19
georgkyes, seems like it23:20
georgkadded some debug statements in the code: https://hastebin.com/obihexohuy.cs23:21
mattmceuenI'm really scratching my head on this one georgk23:32
mattmceuendid you do a greenfield deployment including your changes?  Or was this an update to an existing site to add the new Network?23:33
mattmceuenHave to run now - I will see if someone with more drydock experience can take a look tomorrow23:34
openstackgerritAlexander Vlasov proposed airship/promenade master: Apply docker-default AppArmorProfile for calico-etcd pod  https://review.opendev.org/65918823:35
*** irclogbot_3 has quit IRC23:45
*** irclogbot_3 has joined #airshipit23:47

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!