Deploy Openstack on 3 VMs: 1 Controller + 1 Network + 1 Compute
Using kvm_quicklab project.
VM configuration:
100 GB Disk 4096 MB RAM 2 CPU 2 vNICs eth0 was on the "data" network eth1 was on the "system" network
<WRAP center round important 60%> OSA will overwrite system's ansible binary so better use a VM as deployment machine. </WRAP>
We use the controller node (Fedora 28 cloud) as deployment host.
dnf upgrade (important) dnf install git ntp ntpdate openssh-server python-devel python-virtualenv btrfs-progs ansible iptables sudo '@Development Tools' systemctl start systemd-networkd.service systemctl enable systemd-networkd.service
Error: Unit dbus-org.freedesktop.resolve1.service not found
cd /etc/systemd/system sudo ln -s /usr/lib/systemd/system/systemd-resolved.service dbus-org.freedesktop.resolve1.service
Error: python subprocess32 build
dnf upgrade
VMs to be installed with OS components are connected as following:
[dang@gt130 os_dev_lab]$ brctl show bridge name bridge id STP enabled interfaces br-2b718ca6738a 8000.024259ac89af no br_ql_ext 8000.668a75761f2f yes compute-eth2 controller-eth2 network-eth2 br_ql_int 8000.66aca99d4771 yes compute-eth1 controller-eth1 network-eth1 br_ql_mgmt 8000.72e36f4f30da yes compute-eth0 controller-eth0 network-eth0 docker0 8000.0242ebe08971 no virbr0 8000.5254008d5702 yes virbr0-nic
IPs are configured statically
# ping all host in the subnet nmap -sP 10.10.10.0/24 Starting Nmap 7.60 ( https://nmap.org ) at 2018-10-03 20:25 CEST Nmap scan report for 10.10.10.1 Host is up (0.00042s latency). Nmap scan report for os-compute (10.10.10.32) Host is up (0.0017s latency). Nmap scan report for os-network (10.10.10.84) Host is up (0.00027s latency). Nmap scan report for os-controller (10.10.10.190) Host is up (0.00062s latency). Nmap done: 256 IP addresses (4 hosts up) scanned in 3.73 seconds
[clouduser@os-controller ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br-mgmt state UP group default qlen 1000
link/ether de:ad:be:80:7c:53 brd ff:ff:ff:ff:ff:ff
inet6 fe80::dcad:beff:fe80:7c53/64 scope link
valid_lft forever preferred_lft forever
3: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br-vlan state UP group default qlen 1000
link/ether de:ad:be:80:7c:54 brd ff:ff:ff:ff:ff:ff
inet6 fe80::dcad:beff:fe80:7c54/64 scope link
valid_lft forever preferred_lft forever
4: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether de:ad:be:80:7c:55 brd ff:ff:ff:ff:ff:ff
inet 10.30.30.192/24 brd 10.30.30.255 scope global dynamic ens5
valid_lft 86279sec preferred_lft 86279sec
inet6 fe80::dcad:beff:fe80:7c55/64 scope link
valid_lft forever preferred_lft forever
5: bond0: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 2e:f3:56:53:41:50 brd ff:ff:ff:ff:ff:ff
6: br-mgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether de:ad:be:80:7c:53 brd ff:ff:ff:ff:ff:ff
inet 10.10.10.10/24 brd 10.10.10.255 scope global br-mgmt
valid_lft forever preferred_lft forever
inet 10.10.10.9/24 brd 10.10.10.255 scope global secondary br-mgmt:0
valid_lft forever preferred_lft forever
inet6 fe80::dcad:beff:fe80:7c53/64 scope link
valid_lft forever preferred_lft forever
7: br-vlan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether de:ad:be:80:7c:54 brd ff:ff:ff:ff:ff:ff
inet6 fe80::dcad:beff:fe80:7c54/64 scope link
valid_lft forever preferred_lft forever
8: ens4.30@ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-vxlan state UP group default qlen 1000
link/ether de:ad:be:80:7c:54 brd ff:ff:ff:ff:ff:ff
inet6 fe80::dcad:beff:fe80:7c54/64 scope link
valid_lft forever preferred_lft forever
9: br-vxlan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether de:ad:be:80:7c:54 brd ff:ff:ff:ff:ff:ff
inet 10.40.40.10/24 brd 10.40.40.255 scope global br-vxlan
valid_lft forever preferred_lft forever
inet6 fe80::dcad:beff:fe80:7c54/64 scope link
valid_lft forever preferred_lft forever
[root@os-controller clouduser]# cat /etc/sysconfig/network-scripts/ifcfg-ens3 # internal network DEVICE=ens3 TYPE=Ethernet BOOTPROTO=none ONBOOT=yes BRIDGE=br-mgmt [root@os-controller clouduser]# cat /etc/sysconfig/network-scripts/ifcfg-ens4 # internal network DEVICE=ens4 TYPE=Ethernet BOOTPROTO=no ONBOOT=yes BRIDGE=br-vlan [root@os-controller clouduser]# cat /etc/sysconfig/network-scripts/ifcfg-ens4.30 # internal network DEVICE=ens4.30 BOOTPROTO=none ONBOOT=yes #IPADDR=192.168.1.1 #NETMASK=255.255.255.0 #NETWORK=192.168.1.0 USERCTL=no VLAN=yes BRIDGE=br-vxlan [root@os-controller clouduser]# cat /etc/sysconfig/network-scripts/ifcfg-ens5 # external network DEVICE=ens5 TYPE=Ethernet BOOTPROTO=dhcp ONBOOT=yes [root@os-controller clouduser]# cat /etc/sysconfig/network-scripts/ifcfg-br-vlan DEVICE=br-vlan TYPE=Bridge ONBOOT=yes BOOTPROTO=none [root@os-controller clouduser]# cat /etc/sysconfig/network-scripts/ifcfg-br-vxlan DEVICE=br-vxlan TYPE=Bridge IPADDR=10.40.40.10 NETMASK=255.255.255.0 ONBOOT=yes BOOTPROTO=none [root@os-controller clouduser]# cat /etc/sysconfig/network-scripts/ifcfg-br-mgmt DEVICE=br-mgmt TYPE=Bridge IPADDR=10.10.10.10 NETMASK=255.255.255.0 ONBOOT=yes BOOTPROTO=none [root@os-controller clouduser]# cat /etc/sysconfig/network-scripts/ifcfg-br-mgmt:0 DEVICE=br-mgmt:0 TYPE=Bridge IPADDR=10.10.10.9 NETMASK=255.255.255.0 ONBOOT=yes BOOTPROTO=none
these do not have ifcfg-br-mgmt:0
Configuration reflect the target host network setting and OpenStack architecture.
Autogenerate usersecrets.yml. Change keystoneauthadminpassword: admin
cidr_networks:
container: 10.10.10.0/24 tunnel: 10.40.40.0/24 #storage: 172.29.244.0/24
used_ips:
globaloverrides: # The internal and external VIP should be different IPs, however they # do not need to be on separate networks. #externallbvipaddress: 10.10.10.9
external_lb_vip_address: 10.10.10.10
internal_lb_vip_address: 10.10.10.10
management_bridge: "br-mgmt"
provider_networks:
- network:
container_bridge: "br-mgmt"
container_type: "veth"
container_interface: "eth1"
ip_from_q: "container"
type: "raw"
group_binds:
- all_containers
- hosts
is_container_address: true
- network:
container_bridge: "br-vxlan"
container_type: "veth"
# vir eth in lxc container or tap
container_interface: "eth10"
ip_from_q: "tunnel"
type: "vxlan"
range: "1:1000"
net_name: "vxlan"
group_binds:
- neutron_linuxbridge_agent
- network:
container_bridge: "br-vlan"
container_type: "veth"
container_interface: "eth12"
host_bind_override: "eth12"
type: "flat"
net_name: "flat"
group_binds:
- neutron_linuxbridge_agent
- network:
container_bridge: "br-vlan"
container_type: "veth"
container_interface: "eth11"
type: "vlan"
range: "101:200,301:400"
net_name: "vlan"
group_binds:
- neutron_linuxbridge_agent
#- network:
# container_bridge: "br-storage"
# container_type: "veth"
# container_interface: "eth2"
# ip_from_q: "storage"
# type: "raw"
# group_binds:
# - glance_api
# - cinder_api
# - cinder_volume
# - nova_compute
shared-infra_hosts:
infra1: ip: 10.10.10.10
repo-infra_hosts:
infra1: ip: 10.10.10.10
haproxy_hosts:
infra1: ip: 10.10.10.10
identity_hosts:
infra1: ip: 10.10.10.10
storage-infra_hosts:
infra1: ip: 10.10.10.10
image_hosts:
infra1: ip: 10.10.10.10
compute-infra_hosts:
infra1: ip: 10.10.10.10
orchestration_hosts:
infra1: ip: 10.10.10.10
dashboard_hosts:
infra1: ip: 10.10.10.10
network_hosts:
network: ip: 10.10.10.11
compute_hosts:
compute1: ip: 10.10.10.12
</code>
cp env.d/aio_metal.yml.example env.d/aio_metal.yml
rm /etc/openstack_deploy/openstack_inventory.json
manually move haproxy.* in /etc/ssl/private/ to /etc/pki/tls/private and remove the former dir.
TASK [os_keystone : Create system links] **************************************************************************************************************************************************************************
ok: [infra1] => (item={u'dest': u'/etc/ssl/certs', u'src': u'/etc/pki/tls/certs'})
failed: [infra1] (item={u'dest': u'/etc/ssl/private', u'src': u'/etc/pki/tls/private'}) => {"changed": false, "gid": 0, "group": "root", "item": {"dest": "/etc/ssl/private", "src": "/etc/pki/tls/private"}, "mode": "0755", "msg": "refusing to convert between directory and link for /etc/ssl/private", "owner": "root", "path": "/etc/ssl/private", "secontext": "unconfined_u:object_r:cert_t:s0", "size": 6, "state": "directory", "uid": 0}
#####changed: [infra1] => (item={u'dest': u'/etc/ssl/private', u'src': u'/etc/ssl/private'})
ok: [infra1] => (item={u'dest': u'/var/log/apache2', u'src': u'/var/log/httpd'})
grep 1539369704.507:11792 /var/log/audit/audit.log | audit2why
* enalble prot bind: https://www.itadminstrator.com/2017/07/nginx-no-permission-to-bind-port-9080.html
[root@os-controller clouduser]# grep 1539380598.687:22934 /var/log/audit/audit.log | audit2allow -m nginx5001
module nginx5001 1.0;
require {
type httpd_t;
type commplex_link_port_t;
class tcp_socket name_connect;
}
#============= httpd_t ==============
#!!!! This avc can be allowed using the boolean 'httpd_can_network_connect'
allow httpd_t commplex_link_port_t:tcp_socket name_connect;
[root@os-controller clouduser]# setsebool -P httpd_can_network_connect 1
selinux error less /var/log/audit/audit.log # semanage port -l | grep 8181 intermapper_port_t tcp 8181 seinfo -x --type=intermapper_port_t /etc/selinux/targeted/policy/policy.31 # use -M to create module .pp, -m nginx.te to create te file. grep 1539370891.083:12008 /var/log/audit/audit.log | audit2allow -m nginx
module nginx 1.0;
require {
type httpd_t;
type intermapper_port_t;
class tcp_socket name_bind;
}
#============= httpd_t ============== allow httpd_t intermapper_port_t:tcp_socket name_bind; # TE file must be built # audit2allow(1) man page already describes it:
# Building module policy manually
# Compile the module
$ checkmodule -M -m -o local.mod local.te
# Create the package
$ semodule_package -o local.pp -m local.mod
# Load the module into the kernel
$ semodule -i local.pp
######
# add port to type
semanage port -a -t http_port_t -p tcp 8181
ValueError: Port tcp/8181 already defined
# replace add -a with modify -m
semanage port -m -t http_port_t -p tcp 8181
7. Creating Custom SELinux Policy Modules with audit2allow
Sometimes there are occasions when none of the above methods deal with a given situation and we need to extend the SELinux policy by creating a custom policy module to allow for a certain set of conditions. For example, consider the postgrey service add-on for an smtp mail server. Our smtp server needs to communicate with postgrey over a Unix socket and that is something the default SELinux policy for our smtp server does not allow. Consequently the service is blocked by SELinux. This is an issue that can not be fixed by changing or restoring file type security contexts and isn't something that has a boolean value we can toggle to allow. We could disable SELinux protection of the smtp server through a boolean, which would be better than disabling SELinux completely, but that is still far from ideal.
If we switch SELinux into Permissive mode and run our mail server for a set period of time, we can log SELinux issues whilst still permitting access (as mentioned in Gathering Audit Logs in Permissive Mode). Checking our logs, we see the following SELinux AVC messages:
type=AVC msg=audit(1218128130.653:334): avc: denied { connectto } for pid=9111 comm="smtpd" path="/var/spool/postfix/postgrey/socket"
scontext=system_u:system_r:postfix_smtpd_t:s0 tcontext=system_u:system_r:initrc_t:s0 tclass=unix_stream_socket
type=AVC msg=audit(1218128130.653:334): avc: denied { write } for pid=9111 comm="smtpd" name="socket" dev=sda6 ino=39977017
scontext=system_u:system_r:postfix_smtpd_t:s0 tcontext=system_u:object_r:postfix_spool_t:s0 tclass=sock_file
Then we can use 'audit2allow' to generate a set of policy rules that would allow the required actions. We can generate a local postgrey Type Enforcement policy file (postgreylocal.te):
# grep smtpd_t /var/log/audit/audit.log | audit2allow -m postgreylocal > postgreylocal.te
# cat postgreylocal.te
module postgreylocal 1.0;
require {
type postfix_smtpd_t;
type postfix_spool_t;
type initrc_t;
class sock_file write;
class unix_stream_socket connectto;
}
#============= postfix_smtpd_t ==============
allow postfix_smtpd_t initrc_t:unix_stream_socket connectto;
allow postfix_smtpd_t postfix_spool_t:sock_file write;
Above we see that we can grep the audit.log file for issues relating to our smtp server and pipe those issues to audit2allow which generates a set of rules that it thinks would permit the actions currently denied by the SELinux policy. Reviewing these rules we see our smtp server wants to connect and write to a Unix socket which we see from out logs is the Unix socket that the postgrey service is listening on. As this seems perfectly reasonable, we can go ahead and use audit2allow to make a custom policy module to allow these actions:
# grep smtpd_t /var/log/audit/audit.log | audit2allow -M postgreylocal
We then load our postgrey policy module using the 'semodule' command into the current SELinux policy:
semodule -i postgreylocal.pp
which will add our postgrey policy module to /etc/selinux/targeted/modules/active/modules/postgreylocal.pp. We can check the policy module loaded correctly by listing loaded modules with 'semodule -l'.
We can then continue to monitor our SELinux log files to check that our custom policy module works and once we are satisfied we can re-enable SELinux Enforcing mode and again benefit from SELinux protection of our now fully functional smtp server.
7.1. Manually Customizing Policy Modules
Often audit2allow will automatically create a custom policy module that will resolve a particular issue, but there are times when it doesn't get it quite right and we may want to manually edit and compile the policy module. For example, consider the following AVC audit log:
Summary:
SELinux is preventing postdrop (postfix_postdrop_t) "getattr" to
/var/log/httpd/error_log (httpd_log_t).
Detailed Description:
SELinux denied access requested by postdrop. It is not expected that this access
is required by postdrop and this access may signal an intrusion attempt. It is
also possible that the specific version or configuration of the application is
causing it to require additional access.
Allowing Access:
Sometimes labeling problems can cause SELinux denials. You could try to restore
the default system file context for /var/log/httpd/error_log,
restorecon -v '/var/log/httpd/error_log'
If this does not work, there is currently no automatic way to allow this access.
Instead, you can generate a local policy module to allow this access - see FAQ
(http://fedora.redhat.com/docs/selinux-faq-fc5/#id2961385) Or you can disable
SELinux protection altogether. Disabling SELinux protection is not recommended.
Please file a bug report (http://bugzilla.redhat.com/bugzilla/enter_bug.cgi)
against this package.
Additional Information:
Source Context system_u:system_r:postfix_postdrop_t
Target Context root:object_r:httpd_log_t
Target Objects /var/log/httpd/error_log [ file ]
Source postdrop
Source Path /usr/sbin/postdrop
Port <Unknown>
Host sanitized
Source RPM Packages postfix-2.3.3-2
Target RPM Packages
Policy RPM selinux-policy-2.4.6-137.1.el5
Selinux Enabled True
Policy Type targeted
MLS Enabled True
Enforcing Mode Enforcing
Plugin Name catchall_file
Host Name sanitized
Platform Linux sanitized 2.6.18-53.1.21.el5 #1 SMP Tue
May 20 09:35:07 EDT 2008 x86_64 x86_64
Alert Count 599
First Seen Wed Jul 2 08:27:15 2008
Last Seen Sun Aug 10 22:47:52 2008
Local ID c303a4ea-8e7a-4acc-9118-9cc61c6a2ec8
Line Numbers
Raw Audit Messages
host=sanitized type=AVC msg=audit(1218397672.372:352): avc: denied { getattr } for pid=4262 comm="postdrop"
path="/var/log/httpd/error_log" dev=md2 ino=117005 scontext=system_u:system_r:postfix_postdrop_t:s0
tcontext=root:object_r:httpd_log_t:s0 tclass=file
host=sanitized type=SYSCALL msg=audit(1218397672.372:352): arch=c000003e syscall=5 success=no exit=-13 a0=2
a1=7fffd6febca0 a2=7fffd6febca0 a3=0 items=0 ppid=4261 pid=4262 auid=4294967295 uid=48 gid=48 euid=48 suid=48
fsuid=48 egid=90 sgid=90 fsgid=90 tty=(none) comm="postdrop" exe="/usr/sbin/postdrop"
subj=system_u:system_r:postfix_postdrop_t:s0 key=(null)
Running audit2allow on the above error, and reviewing the resultant postfixlocal.te policy file we see:
# grep postdrop /var/log/audit/audit.log | audit2allow -M postfixlocal
# cat postfixlocal.te
module postfixlocal 1.0;
require {
type httpd_log_t;
type postfix_postdrop_t;
class dir getattr;
class file { read getattr };
}
#============= postfix_postdrop_t ==============
allow postfix_postdrop_t httpd_log_t:file getattr;
Hopefully the first thing to strike us here is why does postdrop needs access to /var/log/httpd/error_log? Presumably this isn't something we would expect so we have to assess if this is an action we wish to allow or not. We have a number of options. We could just ignore the error and allow SELinux to continue blocking and logging access attempts or we could allow the action by creating the custom policy module as suggested by audit2allow. Alternatively we can edit the custom policy module .te file to prevent auditing of this particular error whilst still allowing SELinux to continue preventing access. We do this by editing the allow line, changing it to dontaudit:
#============= postfix_postdrop_t ==============
dontaudit postfix_postdrop_t httpd_log_t:file getattr;
Now we can manually compile and load the edited custom policy module:
# checkmodule -M -m -o postfixlocal.mod postfixlocal.te
# semodule_package -o postfixlocal.pp -m postfixlocal.mod
# semodule -i postfixlocal.pp
Access to /var/log/httpd/error_log by postdrop will still be prevented by SELinux but we won't receive constant alerts and error messages filling up our log files each time access is blocked.