Archives des Oracle - dbi Blog (2023)

Oracle appliance is proposing various possibilities for creating databases either on the Bare Metal or using KVM DB System. Each DB System will host a single database in a separate VM. What about having each DB system running a separate network? In this blog I would like to share my tests and findings about how to create additional network on the ODA and how to create DB System on separate VLAN interface. The principle would be of course the same if we would like to create KVM Virtual Machines (compute instance) on separate VLAN.

Configuration description

For my test I have got an ODA X8-2M with a quad-port 10GBase-T network interface. I’m running ODA version 19.13.
On my network card, the 2 first ports p7p1 and p7p2 will be assigned to btbond1 and the 2 next ports p7p3 and p7p4 will be assigned to my second bonding interface btbond2. ODA are configured by default with active-backup mode without LACP for all bonding interfaces. This is automatically designed by the ODA and can not be changed. More over we need to keep in mind that on an appliance we will never manually change the Linux network scripts. All network configuration changes needs to be done with odacli.

btbond1 is used for my main network and we will use btbond2 to add additionnal networks.

The 2 additional networks are :
10.38.0.1/24 VLAN id 38
10.39.0.1/24 VLAN id 39

Checking network interface

With ethtool I can check and see that the 2 p7p3 and p7p4 ports are twisted pair and connected to the network :

[root@dbi-oda-x8 ~]# ethtool p7p3 | grep -iE "(bportb|detected)"Port: Twisted PairLink detected: yes[root@dbi-oda-x8 ~]# ethtool p7p4 | grep -iE "(bportb|detected)"Port: Twisted PairLink detected: yes

I can see that both those interfaces are assigned to btbond2 interface :

[root@dbi-oda-x8 ~]# grep MASTER /etc/sysconfig/network-scripts/ifcfg-p7p3MASTER=btbond2[root@dbi-oda-x8 ~]# grep MASTER /etc/sysconfig/network-scripts/ifcfg-p7p4MASTER=btbond2

Checking existing and default ODA configuration

After reimaging an ODA we would have the below default configuration.

root@dbi-oda-x8 ~]# odacli list-networksID Name NIC Interface Type Subnet Mask Gateway VLAN ID Node Networks-------------------------------------- -------------------- ------------ --------------- ------------------ ------------------ -------- -----------------------bb05b06c-52eb-41ce-8ed8-92ec712db61d Public-network pubnet BRIDGE 255.255.255.0 10.36.0.1 [IP Address on node0: 10.36.0.241]95ec74b7-4f6f-4cda-9b0c-5bcac731666b ASM-network privasm BRIDGE 255.255.255.128 [IP Address on node0: 192.168.17.2]e77a33d0-2b40-4562-b802-c02cadd93b25 Private-network priv0 INTERNAL 255.255.255.240 [IP Address on node0: 192.168.16.24][root@dbi-oda-x8 ~]# odacli list-vnetworksName Type Interface Bridge Uniform Created Updated-------------------- --------------- --------------- -------------------- -------- ----------------------- -----------------------pubnet Bridged btbond1 pubnet NO 2022-01-28 08:54:55 CET 2022-01-28 08:54:55 CET[root@dbi-oda-x8 ~]#

To use a network interface with the VMs, either a DB System or a Compute instance, the network interface needs to be created in the vnetworks.

VLAN Tagged versus Untagged

If a port on a switch is configured as tagged (trunk port in cisco world), the connected equipment (here our ODA) is VLAN aware, and would need to be tagged as well. The port is enabled for VLAN tagging. Purpose is to pass traffic for multiple VLAN’s. The ethernet frame is enclosing an additional VLAN header. The connected equipment will add the VLAN information in the ethernet frame.

If a port is configured as untagged (access port in cisco world), the connected equipment (here our ODA) does not know anything about in which VLAN it is and does not care about it. The switch will manage it on its own. The port does not tag and only accepts a single VLAN. The ethernet frame coming to the connected equipment will not have any VLAN header.

This is explained in the 802.1Q standard.

Most of the time trunk ports will link switches and access ports will link to end devices, albeit we would like to use several VLAN network on the same ODA network interface.

Creating an additional untagged network

Let’s create an additional untagged physical network on the Bare Metal itself. We will create it on btbond2 interface. Untagged means that the ODA will not add VLAN information in the Ethernet frame.
The option -t bond would be the default one and would here not be required.

[root@dbi-oda-x8 ~]# odacli create-network -n btbond2 -p 10.38.0.10 -m untagged1 -s 255.255.255.0 -g 10.38.0.1 -t bond{ "jobId" : "9a4476dd-b955-433b-9463-377f66ab737a", "status" : "Created", "message" : null, "reports" : [ ], "createTimestamp" : "February 04, 2022 13:48:12 PM CET", "resourceList" : [ ], "description" : "Rac Network service creation with name untagged1", "updatedTime" : "February 04, 2022 13:48:12 PM CET"}[root@dbi-oda-x8 ~]# odacli describe-job -i 9a4476dd-b955-433b-9463-377f66ab737aJob details---------------------------------------------------------------- ID: 9a4476dd-b955-433b-9463-377f66ab737a Description: Rac Network service creation with name untagged1 Status: Success Created: February 4, 2022 1:48:12 PM CET Message:Task Name Start Time End Time Status---------------------------------------- ----------------------------------- ----------------------------------- ----------Setting network February 4, 2022 1:48:12 PM CET February 4, 2022 1:48:18 PM CET SuccessSetting up Network February 4, 2022 1:48:12 PM CET February 4, 2022 1:48:12 PM CET Successrestart network interface btbond2 February 4, 2022 1:48:12 PM CET February 4, 2022 1:48:18 PM CET Success[root@dbi-oda-x8 ~]# odacli list-networksID Name NIC Interface Type Subnet Mask Gateway VLAN ID Node Networks-------------------------------------- -------------------- ------------ --------------- ------------------ ------------------ -------- -----------------------bb05b06c-52eb-41ce-8ed8-92ec712db61d Public-network pubnet BRIDGE 255.255.255.0 10.36.0.1 [IP Address on node0: 10.36.0.241]95ec74b7-4f6f-4cda-9b0c-5bcac731666b ASM-network privasm BRIDGE 255.255.255.128 [IP Address on node0: 192.168.17.2]e77a33d0-2b40-4562-b802-c02cadd93b25 Private-network priv0 INTERNAL 255.255.255.240 [IP Address on node0: 192.168.16.24]6171aad7-e247-4b08-a56e-83bdedd74af1 untagged1 btbond2 BOND 255.255.255.0 10.38.0.1 [IP Address on node0: 10.38.0.10][root@dbi-oda-x8 ~]# odacli list-vnetworksName Type Interface Bridge Uniform Created Updated-------------------- --------------- --------------- -------------------- -------- ----------------------- -----------------------pubnet Bridged btbond1 pubnet NO 2022-01-28 08:54:55 CET 2022-01-28 08:54:55 CET

I can see that our new network exists in the network list but not in the virtual network of course.

I can also check and see that our btbond2 interface has been configured by odacli as an untagged bonding interface with the appropriate information.

[root@dbi-oda-x8 ~]# ip addr sh btbond29: btbond2: mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 3c:fd:fe:92:80:1a brd ff:ff:ff:ff:ff:ff inet 10.38.0.10/24 brd 10.38.0.255 scope global btbond2 valid_lft forever preferred_lft forever[root@dbi-oda-x8 ~]# ls -ltrh /etc/sysconfig/network-scripts/ifcfg*btbond2*-rw-r--r--. 1 root root 262 Feb 4 13:48 /etc/sysconfig/network-scripts/ifcfg-btbond2[root@dbi-oda-x8 ~]# cat /etc/sysconfig/network-scripts/ifcfg-btbond2#This file was created by ODA. Do not edit.NETMASK=255.255.255.0GATEWAY=10.38.0.1BOOTPROTO=nonePEERDNS=noDEVICE=btbond2ONBOOT=yesNM_CONTROLLED=noIPADDR=10.38.0.10BONDING_OPTS="mode=active-backup miimon=100 primary=p7p3"IPV6INIT=noUSERCTL=noTYPE=BOND

Of course it is not possible to create another untagged network on btbond2 interface knowing there is already one existing :

[root@dbi-oda-x8 ~]# odacli create-network -n btbond2 -p 10.39.0.10 -m untagged2 -s 255.255.255.0 -g 10.39.0.1 -t bondDCS-10001:Internal error encountered: nicnamebtbond2 already exists in the networks list .. .

Creating an additional tagged network would also not be possible. Note the option -t and -v to create tagged networks on the ODA :

[root@dbi-oda-x8 ~]# odacli create-network -n btbond2 -p 10.39.0.10 -m tagged2 -s 255.255.255.0 -g 10.39.0.1 -t VLAN -v 39DCS-10001:Internal error encountered: Creating vlan in the interface btbond2 is not allowed. Physical network untagged1 already exists in interface btbond2.

Let’s delete our additional untagged network

[root@dbi-oda-x8 ~]# odacli delete-network -m untagged1{ "jobId" : "3d7a033f-fd9c-4632-a6f1-b80f7d3fcf57", "status" : "Running", "message" : null, "reports" : [ { "taskId" : "TaskSequential_10041", "taskName" : "deleting network", "taskResult" : "", "startTime" : "February 04, 2022 14:08:32 PM CET", "endTime" : "February 04, 2022 14:08:32 PM CET", "status" : "Running", "taskDescription" : null, "parentTaskId" : "TaskSequential_10039", "jobId" : "3d7a033f-fd9c-4632-a6f1-b80f7d3fcf57", "tags" : [ ], "reportLevel" : "Info", "updatedTime" : "February 04, 2022 14:08:32 PM CET" }, { "taskId" : "TaskZJsonRpcExt_10045", "taskName" : "Setting up Network", "taskResult" : "Network setup success", "startTime" : "February 04, 2022 14:08:32 PM CET", "endTime" : "February 04, 2022 14:08:32 PM CET", "status" : "Success", "taskDescription" : null, "parentTaskId" : "TaskSequential_10041", "jobId" : "3d7a033f-fd9c-4632-a6f1-b80f7d3fcf57", "tags" : [ ], "reportLevel" : "Info", "updatedTime" : "February 04, 2022 14:08:32 PM CET" }, { "taskId" : "TaskZJsonRpcExt_10048", "taskName" : "restart network interface btbond2", "taskResult" : "", "startTime" : "February 04, 2022 14:08:32 PM CET", "endTime" : "February 04, 2022 14:08:32 PM CET", "status" : "Running", "taskDescription" : null, "parentTaskId" : "TaskSequential_10047", "jobId" : "3d7a033f-fd9c-4632-a6f1-b80f7d3fcf57", "tags" : [ ], "reportLevel" : "Info", "updatedTime" : "February 04, 2022 14:08:32 PM CET" } ], "createTimestamp" : "February 04, 2022 14:08:32 PM CET", "resourceList" : [ { "resourceId" : "6171aad7-e247-4b08-a56e-83bdedd74af1", "resourceType" : null, "resourceNewType" : "Network", "jobId" : "3d7a033f-fd9c-4632-a6f1-b80f7d3fcf57", "updatedTime" : null } ], "description" : "Network service deleteRacNetwork with id 6171aad7-e247-4b08-a56e-83bdedd74af1", "updatedTime" : "February 04, 2022 14:08:32 PM CET"}[root@dbi-oda-x8 ~]# odacli describe-job -i 3d7a033f-fd9c-4632-a6f1-b80f7d3fcf57Job details---------------------------------------------------------------- ID: 3d7a033f-fd9c-4632-a6f1-b80f7d3fcf57 Description: Network service deleteRacNetwork with id 6171aad7-e247-4b08-a56e-83bdedd74af1 Status: Success Created: February 4, 2022 2:08:32 PM CET Message:Task Name Start Time End Time Status---------------------------------------- ----------------------------------- ----------------------------------- ----------deleting network February 4, 2022 2:08:32 PM CET February 4, 2022 2:08:34 PM CET SuccessSetting up Network February 4, 2022 2:08:32 PM CET February 4, 2022 2:08:32 PM CET Successrestart network interface btbond2 February 4, 2022 2:08:32 PM CET February 4, 2022 2:08:34 PM CET Success

Network has been deleted :

[root@dbi-oda-x8 ~]# odacli list-networksID Name NIC Interface Type Subnet Mask Gateway VLAN ID Node Networks-------------------------------------- -------------------- ------------ --------------- ------------------ ------------------ -------- -----------------------bb05b06c-52eb-41ce-8ed8-92ec712db61d Public-network pubnet BRIDGE 255.255.255.0 10.36.0.1 [IP Address on node0: 10.36.0.241]95ec74b7-4f6f-4cda-9b0c-5bcac731666b ASM-network privasm BRIDGE 255.255.255.128 [IP Address on node0: 192.168.17.2]e77a33d0-2b40-4562-b802-c02cadd93b25 Private-network priv0 INTERNAL 255.255.255.240 [IP Address on node0: 192.168.16.24]

Creating additional tagged network

Let’s create the first tagged network on btbond2 interface :

[root@dbi-oda-x8 ~]# odacli create-network -n btbond2 -p 10.38.0.10 -m tagged38 -s 255.255.255.0 -g 10.38.0.1 -t VLAN -v 38{ "jobId" : "b3dd6d7b-7ee7-4418-afc6-fd71af0a01bc", "status" : "Created", "message" : null, "reports" : [ ], "createTimestamp" : "February 04, 2022 14:40:28 PM CET", "resourceList" : [ ], "description" : "Rac Network service creation with name tagged38", "updatedTime" : "February 04, 2022 14:40:28 PM CET"}[root@dbi-oda-x8 ~]# odacli describe-job -i b3dd6d7b-7ee7-4418-afc6-fd71af0a01bcJob details---------------------------------------------------------------- ID: b3dd6d7b-7ee7-4418-afc6-fd71af0a01bc Description: Rac Network service creation with name tagged38 Status: Success Created: February 4, 2022 2:40:28 PM CET Message:Task Name Start Time End Time Status---------------------------------------- ----------------------------------- ----------------------------------- ----------Setting network February 4, 2022 2:40:28 PM CET February 4, 2022 2:40:33 PM CET SuccessSetting up Vlan February 4, 2022 2:40:28 PM CET February 4, 2022 2:40:33 PM CET Success

The tagged network has been created :

[root@dbi-oda-x8 ~]# odacli list-networksID Name NIC Interface Type Subnet Mask Gateway VLAN ID Node Networks-------------------------------------- -------------------- ------------ --------------- ------------------ ------------------ -------- -----------------------bb05b06c-52eb-41ce-8ed8-92ec712db61d Public-network pubnet BRIDGE 255.255.255.0 10.36.0.1 [IP Address on node0: 10.36.0.241]95ec74b7-4f6f-4cda-9b0c-5bcac731666b ASM-network privasm BRIDGE 255.255.255.128 [IP Address on node0: 192.168.17.2]e77a33d0-2b40-4562-b802-c02cadd93b25 Private-network priv0 INTERNAL 255.255.255.240 [IP Address on node0: 192.168.16.24]fbe05bfa-636e-4f9c-a348-c59ba23e2296 tagged38 btbond2.38 VLAN 255.255.255.0 10.38.0.1 38 [IP Address on node0: 10.38.0.10]

And we can see that respective tagged network interface has now been created on the Linux operating system side (note the <btbond>.<vlan_id>) :

[root@dbi-oda-x8 ~]# ls -ltrh /etc/sysconfig/network-scripts/ifcfg*btbond2*-rw-r--r--. 1 root root 239 Feb 4 14:38 /etc/sysconfig/network-scripts/ifcfg-btbond2-rw-r--r-- 1 root root 520 Feb 4 14:40 /etc/sysconfig/network-scripts/ifcfg-btbond2.38[root@dbi-oda-x8 ~]# cat /etc/sysconfig/network-scripts/ifcfg-btbond2.38#ODA_VLAN_CONFIG ===#ODA_VLAN_CONFIG Name=tagged38#ODA_VLAN_CONFIG VlanId=38#ODA_VLAN_CONFIG VlanInterface=btbond2#ODA_VLAN_CONFIG Type=VlanType#ODA_VLAN_CONFIG VlanSetupType=Other#ODA_VLAN_CONFIG VlanIpAddr=10.38.0.10#ODA_VLAN_CONFIG VlanNetmask=255.255.255.0#ODA_VLAN_CONFIG VlanGateway=10.38.0.1#ODA_VLAN_CONFIG NodeNum=0#=== DO NOT EDIT ANYTHING ABOVE THIS LINE ===DEVICE=btbond2.38BOOTPROTO=noneONBOOT=yesVLAN=yesNM_CONTROLLED=noDEFROUTE=noIPADDR=10.38.0.10NETMASK=255.255.255.0GATEWAY=10.38.0.1[root@dbi-oda-x8 ~]#

And I can reach the ODA on the new network :

C:Users>ping 10.38.0.10Pinging 10.38.0.10 with 32 bytes of data:Reply from 10.38.0.10: bytes=32 time=1ms TTL=63Reply from 10.38.0.10: bytes=32 time<1ms TTL=63Reply from 10.38.0.10: bytes=32 time<1ms TTL=63Reply from 10.38.0.10: bytes=32 time<1ms TTL=63Ping statistics for 10.38.0.10: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 1ms, Average = 0ms

The ODA will not permit to create any untagged additional network on the btbond2 interface knowing one tagged network already exists :

[root@dbi-oda-x8 ~]# odacli create-network -n btbond2 -p 10.39.0.10 -m untagged2 -s 255.255.255.0 -g 10.39.0.1 -t bondDCS-10001:Internal error encountered: Creating non-VLAN typed network on the interface btbond2 is not allowed. VLAN tagged38 already exists in interface btbond2.

Let’s create the second tagged network on the same btbond2 interface :

[root@dbi-oda-x8 ~]# odacli create-network -n btbond2 -p 10.39.0.10 -m tagged39 -s 255.255.255.0 -g 10.39.0.1 -t VLAN -v 39{ "jobId" : "e0452663-544c-47c4-8b9b-5fbe6e0a5cd9", "status" : "Created", "message" : null, "reports" : [ ], "createTimestamp" : "February 04, 2022 14:45:18 PM CET", "resourceList" : [ ], "description" : "Rac Network service creation with name tagged39", "updatedTime" : "February 04, 2022 14:45:18 PM CET"}[root@dbi-oda-x8 ~]# odacli describe-job -i e0452663-544c-47c4-8b9b-5fbe6e0a5cd9Job details---------------------------------------------------------------- ID: e0452663-544c-47c4-8b9b-5fbe6e0a5cd9 Description: Rac Network service creation with name tagged39 Status: Success Created: February 4, 2022 2:45:18 PM CET Message:Task Name Start Time End Time Status---------------------------------------- ----------------------------------- ----------------------------------- ----------Setting network February 4, 2022 2:45:18 PM CET February 4, 2022 2:45:23 PM CET SuccessSetting up Vlan February 4, 2022 2:45:18 PM CET February 4, 2022 2:45:23 PM CET Success

New tagged network has been created and we now have 2 tagged network running on the same btbond2 interface :

[root@dbi-oda-x8 ~]# odacli list-networksID Name NIC Interface Type Subnet Mask Gateway VLAN ID Node Networks-------------------------------------- -------------------- ------------ --------------- ------------------ ------------------ -------- -----------------------bb05b06c-52eb-41ce-8ed8-92ec712db61d Public-network pubnet BRIDGE 255.255.255.0 10.36.0.1 [IP Address on node0: 10.36.0.241]95ec74b7-4f6f-4cda-9b0c-5bcac731666b ASM-network privasm BRIDGE 255.255.255.128 [IP Address on node0: 192.168.17.2]e77a33d0-2b40-4562-b802-c02cadd93b25 Private-network priv0 INTERNAL 255.255.255.240 [IP Address on node0: 192.168.16.24]fbe05bfa-636e-4f9c-a348-c59ba23e2296 tagged38 btbond2.38 VLAN 255.255.255.0 10.38.0.1 38 [IP Address on node0: 10.38.0.10]dc11ecbe-c4be-4e18-9c37-9f6360b37ee1 tagged39 btbond2.39 VLAN 255.255.255.0 10.39.0.1 39 [IP Address on node0: 10.39.0.10]

The respective new tagged interface has been configured by odacli on the linux operating system :

[root@dbi-oda-x8 ~]# ls -ltrh /etc/sysconfig/network-scripts/ifcfg*btbond2*-rw-r--r--. 1 root root 239 Feb 4 14:38 /etc/sysconfig/network-scripts/ifcfg-btbond2-rw-r--r-- 1 root root 520 Feb 4 14:40 /etc/sysconfig/network-scripts/ifcfg-btbond2.38-rw-r--r-- 1 root root 520 Feb 4 14:45 /etc/sysconfig/network-scripts/ifcfg-btbond2.39[root@dbi-oda-x8 ~]# cat /etc/sysconfig/network-scripts/ifcfg-btbond2.39#ODA_VLAN_CONFIG ===#ODA_VLAN_CONFIG Name=tagged39#ODA_VLAN_CONFIG VlanId=39#ODA_VLAN_CONFIG VlanInterface=btbond2#ODA_VLAN_CONFIG Type=VlanType#ODA_VLAN_CONFIG VlanSetupType=Other#ODA_VLAN_CONFIG VlanIpAddr=10.39.0.10#ODA_VLAN_CONFIG VlanNetmask=255.255.255.0#ODA_VLAN_CONFIG VlanGateway=10.39.0.1#ODA_VLAN_CONFIG NodeNum=0#=== DO NOT EDIT ANYTHING ABOVE THIS LINE ===DEVICE=btbond2.39BOOTPROTO=noneONBOOT=yesVLAN=yesNM_CONTROLLED=noDEFROUTE=noIPADDR=10.39.0.10NETMASK=255.255.255.0GATEWAY=10.39.0.1

And I can ping my new network as well :

C:Users>ping 10.39.0.10Pinging 10.39.0.10 with 32 bytes of data:Reply from 10.39.0.10: bytes=32 time=2ms TTL=63Reply from 10.39.0.10: bytes=32 time<1ms TTL=63Reply from 10.39.0.10: bytes=32 time<1ms TTL=63Reply from 10.39.0.10: bytes=32 time<1ms TTL=63Ping statistics for 10.39.0.10: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 2ms, Average = 0ms

I can now as well check my global btbond2 configuration on the linux side :

[root@dbi-oda-x8 ~]# ip addr sh | grep -iE "(btbond2|btbond2.38|btbond2.39)"4: p7p3: mtu 1500 qdisc mq master btbond2 state UP group default qlen 10005: p7p4: mtu 1500 qdisc mq master btbond2 state UP group default qlen 10009: btbond2: mtu 1500 qdisc noqueue state UP group default qlen 100057: btbond2.38@btbond2: mtu 1500 qdisc noqueue state UP group default qlen 1000 inet 10.38.0.10/24 brd 10.38.0.255 scope global btbond2.3858: btbond2.39@btbond2: mtu 1500 qdisc noqueue state UP group default qlen 1000 inet 10.39.0.10/24 brd 10.39.0.255 scope global btbond2.39[root@dbi-oda-x8 ~]#

Creating virtual networks

Knowing I need to use the networks for the next DB Systems I have deleted the created physical networks and I’m going to create the 2 same tagged networks as virtual networks on the btbond2 interface :

[root@dbi-oda-x8 ~]# odacli create-vnetwork -n tagged38 -if btbond2 -t bridged-vlan -ip 10.38.0.10 -nm 255.255.255.0 -vlan 38 -gw 10.38.0.1Job details---------------------------------------------------------------- ID: eb57ff9b-c9b0-4e1d-bc51-162a42ea4fb1 Description: vNetwork tagged38 creation Status: Created Created: February 4, 2022 3:19:02 PM CET Message:Task Name Start Time End Time Status---------------------------------------- ----------------------------------- ----------------------------------- ----------[root@dbi-oda-x8 ~]# odacli describe-job -i eb57ff9b-c9b0-4e1d-bc51-162a42ea4fb1Job details---------------------------------------------------------------- ID: eb57ff9b-c9b0-4e1d-bc51-162a42ea4fb1 Description: vNetwork tagged38 creation Status: Success Created: February 4, 2022 3:19:02 PM CET Message:Task Name Start Time End Time Status---------------------------------------- ----------------------------------- ----------------------------------- ----------Validate Virtual Network doesn‘t exist February 4, 2022 3:19:02 PM CET February 4, 2022 3:19:02 PM CET SuccessValidate interface to use exists February 4, 2022 3:19:02 PM CET February 4, 2022 3:19:02 PM CET SuccessValidate interfaces to create not exist February 4, 2022 3:19:02 PM CET February 4, 2022 3:19:02 PM CET SuccessCreate bridge February 4, 2022 3:19:02 PM CET February 4, 2022 3:19:02 PM CET SuccessCreate VLAN February 4, 2022 3:19:02 PM CET February 4, 2022 3:19:02 PM CET SuccessBring up VLAN February 4, 2022 3:19:02 PM CET February 4, 2022 3:19:07 PM CET SuccessCreate metadata February 4, 2022 3:19:07 PM CET February 4, 2022 3:19:07 PM CET SuccessPersist metadata February 4, 2022 3:19:07 PM CET February 4, 2022 3:19:07 PM CET Success[root@dbi-oda-x8 ~]# odacli create-vnetwork -n tagged39 -if btbond2 -t bridged-vlan -ip 10.39.0.10 -nm 255.255.255.0 -vlan 39 -gw 10.39.0.1Job details---------------------------------------------------------------- ID: 15cf889f-9c6e-4e0c-a676-b2203e40cfd2 Description: vNetwork tagged39 creation Status: Created Created: February 4, 2022 3:19:40 PM CET Message:Task Name Start Time End Time Status---------------------------------------- ----------------------------------- ----------------------------------- ----------[root@dbi-oda-x8 ~]# odacli describe-job -i 15cf889f-9c6e-4e0c-a676-b2203e40cfd2Job details---------------------------------------------------------------- ID: 15cf889f-9c6e-4e0c-a676-b2203e40cfd2 Description: vNetwork tagged39 creation Status: Success Created: February 4, 2022 3:19:40 PM CET Message:Task Name Start Time End Time Status---------------------------------------- ----------------------------------- ----------------------------------- ----------Validate Virtual Network doesn‘t exist February 4, 2022 3:19:40 PM CET February 4, 2022 3:19:40 PM CET SuccessValidate interface to use exists February 4, 2022 3:19:40 PM CET February 4, 2022 3:19:40 PM CET SuccessValidate interfaces to create not exist February 4, 2022 3:19:40 PM CET February 4, 2022 3:19:40 PM CET SuccessCreate bridge February 4, 2022 3:19:40 PM CET February 4, 2022 3:19:40 PM CET SuccessCreate VLAN February 4, 2022 3:19:40 PM CET February 4, 2022 3:19:40 PM CET SuccessBring up VLAN February 4, 2022 3:19:40 PM CET February 4, 2022 3:19:45 PM CET SuccessCreate metadata February 4, 2022 3:19:45 PM CET February 4, 2022 3:19:45 PM CET SuccessPersist metadata February 4, 2022 3:19:45 PM CET February 4, 2022 3:19:45 PM CET Success

My 2 new tagged networks exists as virtual network and not physical networks :

[root@dbi-oda-x8 ~]# odacli list-networksID Name NIC Interface Type Subnet Mask Gateway VLAN ID Node Networks-------------------------------------- -------------------- ------------ --------------- ------------------ ------------------ -------- -----------------------bb05b06c-52eb-41ce-8ed8-92ec712db61d Public-network pubnet BRIDGE 255.255.255.0 10.36.0.1 [IP Address on node0: 10.36.0.241]95ec74b7-4f6f-4cda-9b0c-5bcac731666b ASM-network privasm BRIDGE 255.255.255.128 [IP Address on node0: 192.168.17.2]e77a33d0-2b40-4562-b802-c02cadd93b25 Private-network priv0 INTERNAL 255.255.255.240 [IP Address on node0: 192.168.16.24][root@dbi-oda-x8 ~]# odacli list-vnetworksName Type Interface Bridge Uniform Created Updated-------------------- --------------- --------------- -------------------- -------- ----------------------- -----------------------tagged39 BridgedVlan btbond2 brtagged39 NO 2022-02-04 15:19:45 CET 2022-02-04 15:19:45 CETtagged38 BridgedVlan btbond2 brtagged38 NO 2022-02-04 15:19:07 CET 2022-02-04 15:19:07 CETpubnet Bridged btbond1 pubnet NO 2022-01-28 08:54:55 CET 2022-01-28 08:54:55 CET

Tagged network interfaces have been created by the dcs-agent on the linux operating side :

[root@dbi-oda-x8 ~]# ls -ltrh /etc/sysconfig/network-scripts/ifcfg*btbond2*-rw-r--r--. 1 root root 239 Feb 4 14:38 /etc/sysconfig/network-scripts/ifcfg-btbond2-rw-r--r-- 1 root root 145 Feb 4 15:19 /etc/sysconfig/network-scripts/ifcfg-btbond2.38-rw-r--r-- 1 root root 145 Feb 4 15:19 /etc/sysconfig/network-scripts/ifcfg-btbond2.39[root@dbi-oda-x8 ~]# cat /etc/sysconfig/network-scripts/ifcfg-btbond2.38#This file was created by ODA. Do not edit.DEVICE=btbond2.38ONBOOT=yesNM_CONTROLLED=noBOOTPROTO=noneVLAN=yesONPARENT=yesBRIDGE=brtagged38[root@dbi-oda-x8 ~]# cat /etc/sysconfig/network-scripts/ifcfg-btbond2.39#This file was created by ODA. Do not edit.DEVICE=btbond2.39ONBOOT=yesNM_CONTROLLED=noBOOTPROTO=noneVLAN=yesONPARENT=yesBRIDGE=brtagged39

On the linux side, I can see that the respective IP addresses have not been assigned to the physical interfaces :

[root@dbi-oda-x8 ~]# ip addr sh | grep -iE "(btbond2|btbond2.38|btbond2.39)"4: p7p3: mtu 1500 qdisc mq master btbond2 state UP group default qlen 10005: p7p4: mtu 1500 qdisc mq master btbond2 state UP group default qlen 10009: btbond2: mtu 1500 qdisc noqueue state UP group default qlen 100060: btbond2.38@btbond2: mtu 1500 qdisc noqueue master brtagged38 state UP group default qlen 100062: btbond2.39@btbond2: mtu 1500 qdisc noqueue master brtagged39 state UP group default qlen 1000

But on new virtual interfaces linked to the tagged btbond2 interfaces :

[root@dbi-oda-x8 ~]# ip addr sh | grep -iE "(tagged38|tagged39)"59: brtagged38: mtu 1500 qdisc noqueue state UP group default qlen 1000 inet 10.38.0.10/24 brd 10.38.0.255 scope global brtagged3860: btbond2.38@btbond2: mtu 1500 qdisc noqueue master brtagged38 state UP group default qlen 100061: brtagged39: mtu 1500 qdisc noqueue state UP group default qlen 1000 inet 10.39.0.10/24 brd 10.39.0.255 scope global brtagged3962: btbond2.39@btbond2: mtu 1500 qdisc noqueue master brtagged39 state UP group default qlen 1000

And I can ping them :

C:Users>ping 10.38.0.10Pinging 10.38.0.10 with 32 bytes of data:Reply from 10.38.0.10: bytes=32 time=2ms TTL=63Reply from 10.38.0.10: bytes=32 time<1ms TTL=63Reply from 10.38.0.10: bytes=32 time<1ms TTL=63C:Users>ping 10.39.0.10Pinging 10.39.0.10 with 32 bytes of data:Reply from 10.39.0.10: bytes=32 time=2ms TTL=63Reply from 10.39.0.10: bytes=32 time<1ms TTL=63Reply from 10.39.0.10: bytes=32 time=1ms TTL=63Reply from 10.39.0.10: bytes=32 time<1ms TTL=63Ping statistics for 10.39.0.10: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),Approximate round trip times in milli-seconds: Minimum = 0ms, Maximum = 2ms, Average = 0ms

I can see the network bridge information from the operating system as well :

[root@dbi-oda-x8 ~]# brctl showbridge name bridge id STP enabled interfacesbrtagged388000.5254003bdc76 no btbond2.38 vnet2brtagged398000.52540060eb57 no btbond2.39 vnet4privasm 8000.5a8241cdb508 no priv0.100 vnet1 vnet3pubnet 8000.3cfdfe928018 no btbond1 vnet0virbr0 8000.525400dc8c09 yes virbr0-nic

Creating 2 DB Systems using respectively tagged38 and tagged39 virtual networks

My repository has already been updated with the KVM DB System Image :

[root@dbi-oda-x8 ~]# odacli describe-dbsystem-imageDB System Image details--------------------------------------------------------------------------------Component Name Supported Versions Available Versions-------------------- -------------------- --------------------DBVM 19.13.0.0.0 19.13.0.0.0GI 19.13.0.0.211019 19.13.0.0.211019 19.12.0.0.210720 not-available 19.11.0.0.210420 not-available 21.4.0.0.211019 not-available 21.3.0.0.210720 not-availableDB 19.13.0.0.211019 19.13.0.0.211019 19.12.0.0.210720 not-available 19.11.0.0.210420 not-available 21.4.0.0.211019 not-available 21.3.0.0.210720 not-available

I have created the first DB System json file assigning a new IP address from the VLAN38 network using tagged38 virtual network. The IP connection will use bridging through the virtual VLAN38 IP address we created on the Bare Metal itself :

[root@dbi-oda-x8 ~]# cat /opt/dbi/create_dbsystem_srvdb38.json........."network": { "domainName": "dbi-lab.ch", "ntpServers": ["216.239.35.0"], "dnsServers": [ "8.8.8.8","8.8.4.4" ], "nodes": [ { "name": "srvdb38", "ipAddress": "10.38.0.20", "netmask": "255.255.255.0", "gateway": "10.38.0.1", "number": 0 } ],"publicVNetwork": "tagged38"},"grid": { "language": "en"}}

Creation of the first DB System on network VLAN 38 :

[root@dbi-oda-x8 ~]# odacli create-dbsystem -p /opt/dbi/create_dbsystem_srvdb38.jsonEnter password for system "srvdb38":Retype password for system "srvdb38":Enter administrator password for DB "DB38":Retype administrator password for DB "DB38":Job details---------------------------------------------------------------- ID: ed88ef81-5cb3-4214-ac5c-bc255b67577f Description: DB System srvdb38 creation Status: Created Created: February 4, 2022 4:12:33 PM CET Message:Task Name Start Time End Time Status---------------------------------------- ----------------------------------- ----------------------------------- ----------[root@dbi-oda-x8 ~]# odacli describe-job -i ed88ef81-5cb3-4214-ac5c-bc255b67577fJob details---------------------------------------------------------------- ID: ed88ef81-5cb3-4214-ac5c-bc255b67577f Description: DB System srvdb38 creation Status: Success Created: February 4, 2022 4:12:33 PM CET Message:Task Name Start Time End Time Status---------------------------------------- ----------------------------------- ----------------------------------- ----------Create DB System metadata February 4, 2022 4:12:33 PM CET February 4, 2022 4:12:33 PM CET SuccessPersist new DB System February 4, 2022 4:12:33 PM CET February 4, 2022 4:12:33 PM CET SuccessValidate DB System prerequisites February 4, 2022 4:12:33 PM CET February 4, 2022 4:12:37 PM CET SuccessSetup DB System environment February 4, 2022 4:12:37 PM CET February 4, 2022 4:12:39 PM CET SuccessCreate DB System ASM volume February 4, 2022 4:12:39 PM CET February 4, 2022 4:12:45 PM CET SuccessCreate DB System ACFS filesystem February 4, 2022 4:12:45 PM CET February 4, 2022 4:12:54 PM CET SuccessCreate DB System VM ACFS snapshots February 4, 2022 4:12:54 PM CET February 4, 2022 4:13:27 PM CET SuccessCreate temporary SSH key pair February 4, 2022 4:13:27 PM CET February 4, 2022 4:13:27 PM CET SuccessCreate DB System cloud-init config February 4, 2022 4:13:27 PM CET February 4, 2022 4:13:27 PM CET SuccessProvision DB System VM(s) February 4, 2022 4:13:27 PM CET February 4, 2022 4:13:28 PM CET SuccessAttach disks to DB System February 4, 2022 4:13:28 PM CET February 4, 2022 4:13:29 PM CET SuccessAdd DB System to Clusterware February 4, 2022 4:13:29 PM CET February 4, 2022 4:13:29 PM CET SuccessStart DB System February 4, 2022 4:13:29 PM CET February 4, 2022 4:13:30 PM CET SuccessWait DB System VM first boot February 4, 2022 4:13:30 PM CET February 4, 2022 4:17:06 PM CET SuccessSetup Mutual TLS (mTLS) February 4, 2022 4:17:06 PM CET February 4, 2022 4:21:36 PM CET SuccessExport clones repository February 4, 2022 4:21:36 PM CET February 4, 2022 4:21:37 PM CET SuccessSetup ASM client cluster config February 4, 2022 4:21:37 PM CET February 4, 2022 4:22:00 PM CET SuccessInstall DB System February 4, 2022 4:22:00 PM CET February 4, 2022 5:07:11 PM CET SuccessCleanup temporary SSH key pair February 4, 2022 5:07:11 PM CET February 4, 2022 5:07:52 PM CET SuccessSet DB System as configured February 4, 2022 5:07:52 PM CET February 4, 2022 5:07:52 PM CET Success

I have then created the second DB System json file assigning a new IP address from the VLAN39 network using tagged39 virtual network. That time the IP connection will use bridging through the virtual VLAN39 IP address we created on the Bare Metal itself :

[root@dbi-oda-x8 ~]# cat /opt/dbi/create_dbsystem_srvdb39.json.........},"network": { "domainName": "dbi-lab.ch", "ntpServers": ["216.239.35.0"], "dnsServers": [ "8.8.8.8","8.8.4.4" ], "nodes": [ { "name": "srvdb39", "ipAddress": "10.39.0.20", "netmask": "255.255.255.0", "gateway": "10.39.0.1", "number": 0 } ],"publicVNetwork": "tagged39"},"grid": { "language": "en"}}

Creation on the second DB System on network VLAN 39 :

[root@dbi-oda-x8 ~]# odacli create-dbsystem -p /opt/dbi/create_dbsystem_srvdb39.jsonEnter password for system "srvdb39":Retype password for system "srvdb39":Enter administrator password for DB "DB39":Retype administrator password for DB "DB39":Job details---------------------------------------------------------------- ID: 38303453-8524-4c65-b1d2-3717dfc79a1f Description: DB System srvdb39 creation Status: Created Created: February 4, 2022 5:14:01 PM CET Message:Task Name Start Time End Time Status---------------------------------------- ----------------------------------- ----------------------------------- ----------[root@dbi-oda-x8 ~]# odacli describe-job -i 38303453-8524-4c65-b1d2-3717dfc79a1fJob details---------------------------------------------------------------- ID: 38303453-8524-4c65-b1d2-3717dfc79a1f Description: DB System srvdb39 creation Status: Success Created: February 4, 2022 5:14:01 PM CET Message:Task Name Start Time End Time Status---------------------------------------- ----------------------------------- ----------------------------------- ----------Create DB System metadata February 4, 2022 5:14:01 PM CET February 4, 2022 5:14:01 PM CET SuccessPersist new DB System February 4, 2022 5:14:01 PM CET February 4, 2022 5:14:01 PM CET SuccessValidate DB System prerequisites February 4, 2022 5:14:01 PM CET February 4, 2022 5:14:05 PM CET SuccessSetup DB System environment February 4, 2022 5:14:05 PM CET February 4, 2022 5:14:07 PM CET SuccessCreate DB System ASM volume February 4, 2022 5:14:07 PM CET February 4, 2022 5:14:13 PM CET SuccessCreate DB System ACFS filesystem February 4, 2022 5:14:13 PM CET February 4, 2022 5:14:22 PM CET SuccessCreate DB System VM ACFS snapshots February 4, 2022 5:14:22 PM CET February 4, 2022 5:14:52 PM CET SuccessCreate temporary SSH key pair February 4, 2022 5:14:52 PM CET February 4, 2022 5:14:52 PM CET SuccessCreate DB System cloud-init config February 4, 2022 5:14:52 PM CET February 4, 2022 5:14:53 PM CET SuccessProvision DB System VM(s) February 4, 2022 5:14:53 PM CET February 4, 2022 5:14:54 PM CET SuccessAttach disks to DB System February 4, 2022 5:14:54 PM CET February 4, 2022 5:14:54 PM CET SuccessAdd DB System to Clusterware February 4, 2022 5:14:54 PM CET February 4, 2022 5:14:54 PM CET SuccessStart DB System February 4, 2022 5:14:54 PM CET February 4, 2022 5:14:56 PM CET SuccessWait DB System VM first boot February 4, 2022 5:14:56 PM CET February 4, 2022 5:18:32 PM CET SuccessSetup Mutual TLS (mTLS) February 4, 2022 5:18:32 PM CET February 4, 2022 5:23:07 PM CET SuccessExport clones repository February 4, 2022 5:23:07 PM CET February 4, 2022 5:23:07 PM CET SuccessSetup ASM client cluster config February 4, 2022 5:23:07 PM CET February 4, 2022 5:23:30 PM CET SuccessInstall DB System February 4, 2022 5:23:30 PM CET February 4, 2022 6:09:43 PM CET SuccessCleanup temporary SSH key pair February 4, 2022 6:09:43 PM CET February 4, 2022 6:10:23 PM CET SuccessSet DB System as configured February 4, 2022 6:10:23 PM CET February 4, 2022 6:10:23 PM CET Success

Virtual network configuration checks

So I have got my both virtual networks configured on respective IP address (10.38.0.10 for VLAN38 network and 10.39.0.10 for VLAN39 network) :

[root@dbi-oda-x8 ~]# odacli list-vnetworksName Type Interface Bridge Uniform Created Updated-------------------- --------------- --------------- -------------------- -------- ----------------------- -----------------------tagged39 BridgedVlan btbond2 brtagged39 NO 2022-02-04 15:19:45 CET 2022-02-04 15:19:45 CETtagged38 BridgedVlan btbond2 brtagged38 NO 2022-02-04 15:19:07 CET 2022-02-04 15:19:07 CETpubnet Bridged btbond1 pubnet NO 2022-01-28 08:54:55 CET 2022-01-28 08:54:55 CET[root@dbi-oda-x8 ~]# odacli describe-vnetwork -n tagged38VNetwork details-------------------------------------------------------------------------------- ID: 0d8099ba-ff42-4d73-82b4-f7497fb68ca5 Name: tagged38 Created: 2022-02-04 15:19:07 CET Updated: 2022-02-04 15:19:07 CET Type: BridgedVlan Interface name: btbond2 Bridge name: brtagged38 VLAN ID: 38 IP: 10.38.0.10 Netmask: 255.255.255.0 Gateway: 10.38.0.1 Attached in VMs (config): xa287c764d Attached in VMs (live): xa287c764d[root@dbi-oda-x8 ~]# odacli describe-vnetwork -n tagged39VNetwork details-------------------------------------------------------------------------------- ID: 00df04d4-9a01-4547-9a3c-5a27dab10494 Name: tagged39 Created: 2022-02-04 15:19:45 CET Updated: 2022-02-04 15:19:45 CET Type: BridgedVlan Interface name: btbond2 Bridge name: brtagged39 VLAN ID: 39 IP: 10.39.0.10 Netmask: 255.255.255.0 Gateway: 10.39.0.1 Attached in VMs (config): x793d6c5ce Attached in VMs (live): x793d6c5ce

DB System information checks

I have my 2 DB Systems.
srvdb38 configured on VLAN38 network with 10.38.0.20 IP Address.
srvdb39 configured on VLAN39 network with 10.39.0.20 IP Address.

[root@dbi-oda-x8 ~]# odacli list-dbsystemsName Shape Cores Memory GI version DB version Status Created Updated-------------------- ---------- ----- ---------- ------------------ ------------------ --------------- ----------------------- -----------------------srvdb39 odb2 4 16.00 GB 19.13.0.0.211019 19.13.0.0.211019 CONFIGURED 2022-02-04 17:14:01 CET 2022-02-04 18:10:23 CETsrvdb38 odb2 4 16.00 GB 19.13.0.0.211019 19.13.0.0.211019 CONFIGURED 2022-02-04 16:12:33 CET 2022-02-04 17:07:52 CET[root@dbi-oda-x8 ~]# odacli describe-dbsystem -n srvdb38DB System details-------------------------------------------------------------------------------- ID: 859466fe-a4a5-40ae-aa00-8e02cf74dedc Name: srvdb38 Image: 19.13.0.0.0 Shape: odb2 Cluster name: dbsa287c764d Grid version: 19.13.0.0.211019 Memory: 16.00 GB NUMA enabled: YES Status: CONFIGURED Created: 2022-02-04 16:12:33 CET Updated: 2022-02-04 17:07:52 CET CPU Pool-------------------------- Name: cpupool4dbsystems Number of cores: 4 Host: dbi-oda-x8 Effective CPU set: 5-6,21-22,37-38,53-54 Online CPUs: 5, 6, 21, 22, 37, 38, 53, 54 Offline CPUs: NONE VM Storage-------------------------- Disk group: DATA Volume name: SA287C764D Volume device: /dev/asm/sa287c764d-390 Size: 200.00 GB Mount Point: /u05/app/sharedrepo/srvdb38 VMs-------------------------- Host: dbi-oda-x8 VM Name: xa287c764d VM Host Name: srvdb38.dbi-lab.ch VM image path: /u05/app/sharedrepo/srvdb38/.ACFS/snaps/vm_xa287c764d/xa287c764d Target State: ONLINE Current State: ONLINE VNetworks-------------------------- Host: dbi-oda-x8 VM Name: xa287c764d Public: 10.38.0.20 / 255.255.255.0 / ens3 / BRIDGE(brtagged38) ASM: 192.168.17.4 / 255.255.255.128 / ens4 / BRIDGE(privasm) VLAN(priv0.100) Extra VNetworks-------------------------- Host: dbi-oda-x8 VM Name: xa287c764d tagged38: 10.38.0.20 / 255.255.255.0 / PUBLIC Databases-------------------------- Name: DB38 Resource ID: 5ff22777-f24b-446a-8c58-3fa6dd2ad383 Unique name: DB38_SITE1 Database ID: 715007309 Domain name: dbi-lab.ch DB Home ID: 8b66a7d6-9e5e-43c0-a69c-00b80d2dba81 Shape: odb2 Version: 19.13.0.0.211019 Edition: EE Type: SI Role: PRIMARY Class: OLTP Storage: ASM Redundancy: Target node name: Character set: AL32UTF8 NLS character set: Language: ENGLISH Territory: AMERICA Console enabled: false SEHA enabled: false Associated networks: Public-network Backup config ID: Level 0 Backup Day: sunday Autobackup enabled: true TDE enabled: false CDB type: false PDB name: PDB admin user:[root@dbi-oda-x8 ~]# odacli describe-dbsystem -n srvdb39DB System details-------------------------------------------------------------------------------- ID: 5238f074-6126-4051-8b9e-e392b55a2328 Name: srvdb39 Image: 19.13.0.0.0 Shape: odb2 Cluster name: dbs793d6c5ce Grid version: 19.13.0.0.211019 Memory: 16.00 GB NUMA enabled: YES Status: CONFIGURED Created: 2022-02-04 17:14:01 CET Updated: 2022-02-04 18:10:23 CET CPU Pool-------------------------- Name: cpupool4dbsystems Number of cores: 4 Host: dbi-oda-x8 Effective CPU set: 5-6,21-22,37-38,53-54 Online CPUs: 5, 6, 21, 22, 37, 38, 53, 54 Offline CPUs: NONE VM Storage-------------------------- Disk group: DATA Volume name: S793D6C5CE Volume device: /dev/asm/s793d6c5ce-390 Size: 200.00 GB Mount Point: /u05/app/sharedrepo/srvdb39 VMs-------------------------- Host: dbi-oda-x8 VM Name: x793d6c5ce VM Host Name: srvdb39.dbi-lab.ch VM image path: /u05/app/sharedrepo/srvdb39/.ACFS/snaps/vm_x793d6c5ce/x793d6c5ce Target State: ONLINE Current State: ONLINE VNetworks-------------------------- Host: dbi-oda-x8 VM Name: x793d6c5ce Public: 10.39.0.20 / 255.255.255.0 / ens3 / BRIDGE(brtagged39) ASM: 192.168.17.5 / 255.255.255.128 / ens4 / BRIDGE(privasm) VLAN(priv0.100) Extra VNetworks-------------------------- Host: dbi-oda-x8 VM Name: x793d6c5ce tagged39: 10.39.0.20 / 255.255.255.0 / PUBLIC Databases-------------------------- Name: DB39 Resource ID: e796e7e1-c8c8-4a63-809c-57976ce2163d Unique name: DB39_SITE1 Database ID: 2043690133 Domain name: dbi-lab.ch DB Home ID: 67f6df00-83d8-477b-8884-201664a3701b Shape: odb2 Version: 19.13.0.0.211019 Edition: EE Type: SI Role: PRIMARY Class: OLTP Storage: ASM Redundancy: Target node name: Character set: AL32UTF8 NLS character set: Language: ENGLISH Territory: AMERICA Console enabled: false SEHA enabled: false Associated networks: Public-network Backup config ID: Level 0 Backup Day: sunday Autobackup enabled: true TDE enabled: false CDB type: false PDB name: PDB admin user:[root@dbi-oda-x8 ~]#

Direct connection on the DB Systems

Finally I can access directly the DB Systems through SSH from my laptop. Here as example, connection to the srvdb38 DB System.

maw@DBI-LT-MAW2 ~ % ssh root@10.38.0.20root@10.38.0.20's password:Last login: Thu Feb 17 16:07:31 2022[root@srvdb38 ~]# ip addr sh ens32: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 52:54:00:6e:3f:9f brd ff:ff:ff:ff:ff:ff inet 10.38.0.20/24 brd 10.38.0.255 scope global noprefixroute ens3 valid_lft forever preferred_lft forever [root@srvdb38 ~]# ps -ef | grep [p]monoracle 21506 1 0 14:05 ? 00:00:00 ora_pmon_DB38[root@srvdb38 ~]#

And directly from my laptop I can connect and use the Oracle database DB38 :

Creating a virtual network without assigning an IP address to the bridge interface

Of course it is possible to create a virtual bridge interface without assigning any IP address.

I deleted tagged39 virtual network in order to create it again without IP address :

[root@dbi-oda-x8 ~]# odacli list-vnetworksName Type Interface Bridge Uniform Created Updated-------------------- --------------- --------------- -------------------- -------- ----------------------- -----------------------tagged38 BridgedVlan btbond2 brtagged38 NO 2022-02-04 15:19:07 CET 2022-02-04 15:19:07 CETpubnet Bridged btbond1 pubnet NO 2022-01-28 08:54:55 CET 2022-01-28 08:54:55 CETtagged39 BridgedVlan btbond2 brtagged39 NO 2022-02-04 22:13:18 CET 2022-02-17 16:38:02 CET[root@dbi-oda-x8 ~]# odacli delete-vnetwork -n tagged39Job details---------------------------------------------------------------- ID: 2cae44c4-39e7-4102-8453-597f707ed10b Description: vNetwork tagged39 deletion Status: Created Created: February 17, 2022 4:46:45 PM CET Message:Task Name Start Time End Time Status---------------------------------------- ----------------------------------- ----------------------------------- ----------[root@dbi-oda-x8 ~]# odacli describe-job -i 2cae44c4-39e7-4102-8453-597f707ed10bJob details---------------------------------------------------------------- ID: 2cae44c4-39e7-4102-8453-597f707ed10b Description: vNetwork tagged39 deletion Status: Success Created: February 17, 2022 4:46:45 PM CET Message:Task Name Start Time End Time Status---------------------------------------- ----------------------------------- ----------------------------------- ----------Validate Virtual Network exists February 17, 2022 4:46:45 PM CET February 17, 2022 4:46:45 PM CET SuccessValidate VM does not have the VNetwork attached February 17, 2022 4:46:45 PM CET February 17, 2022 4:46:45 PM CET SuccessDelete bridge February 17, 2022 4:46:45 PM CET February 17, 2022 4:46:45 PM CET SuccessDelete VLAN February 17, 2022 4:46:45 PM CET February 17, 2022 4:46:46 PM CET SuccessDelete metadata February 17, 2022 4:46:46 PM CET February 17, 2022 4:46:46 PM CET Success[root@dbi-oda-x8 ~]# odacli list-vnetworksName Type Interface Bridge Uniform Created Updated-------------------- --------------- --------------- -------------------- -------- ----------------------- -----------------------tagged38 BridgedVlan btbond2 brtagged38 NO 2022-02-04 15:19:07 CET 2022-02-04 15:19:07 CETpubnet Bridged btbond1 pubnet NO 2022-01-28 08:54:55 CET 2022-01-28 08:54:55 CET[root@dbi-oda-x8 ~]# ip addr sh | grep btbond24: p7p3: mtu 1500 qdisc mq master btbond2 state UP group default qlen 10005: p7p4: mtu 1500 qdisc mq master btbond2 state UP group default qlen 100010: btbond2: mtu 1500 qdisc noqueue state UP group default qlen 100011: btbond2.38@btbond2: mtu 1500 qdisc noqueue master brtagged38 state UP group default qlen 1000[root@dbi-oda-x8 ~]#

Of course I could have updated the vnetwork interface using odacli modify-vnetwork and assigning 0.0.0.0 as IP address.

I created again the tagged39 virtual network without assigning an IP address to the bridge interface.

[root@dbi-oda-x8 ~]# odacli list-vnetworksName Type Interface Bridge Uniform Created Updated-------------------- --------------- --------------- -------------------- -------- ----------------------- -----------------------tagged38 BridgedVlan btbond2 brtagged38 NO 2022-02-04 15:19:07 CET 2022-02-04 15:19:07 CETpubnet Bridged btbond1 pubnet NO 2022-01-28 08:54:55 CET 2022-01-28 08:54:55 CET[root@dbi-oda-x8 ~]# odacli create-vnetwork -n tagged39 -if btbond2 -t bridged-vlan -vlan 39Job details---------------------------------------------------------------- ID: a70a1313-f2cd-48ab-ac5f-c17b2f2fae08 Description: vNetwork tagged39 creation Status: Created Created: February 17, 2022 4:49:44 PM CET Message:Task Name Start Time End Time Status---------------------------------------- ----------------------------------- ----------------------------------- ----------[root@dbi-oda-x8 ~]# odacli describe-job -i a70a1313-f2cd-48ab-ac5f-c17b2f2fae08Job details---------------------------------------------------------------- ID: a70a1313-f2cd-48ab-ac5f-c17b2f2fae08 Description: vNetwork tagged39 creation Status: Success Created: February 17, 2022 4:49:44 PM CET Message:Task Name Start Time End Time Status---------------------------------------- ----------------------------------- ----------------------------------- ----------Validate Virtual Network doesn't exist February 17, 2022 4:49:44 PM CET February 17, 2022 4:49:44 PM CET SuccessValidate interface to use exists February 17, 2022 4:49:44 PM CET February 17, 2022 4:49:44 PM CET SuccessValidate interfaces to create not exist February 17, 2022 4:49:44 PM CET February 17, 2022 4:49:44 PM CET SuccessCreate bridge February 17, 2022 4:49:44 PM CET February 17, 2022 4:49:44 PM CET SuccessCreate VLAN February 17, 2022 4:49:44 PM CET February 17, 2022 4:49:44 PM CET SuccessBring up VLAN February 17, 2022 4:49:44 PM CET February 17, 2022 4:49:45 PM CET SuccessCreate metadata February 17, 2022 4:49:45 PM CET February 17, 2022 4:49:45 PM CET SuccessPersist metadata February 17, 2022 4:49:45 PM CET February 17, 2022 4:49:45 PM CET Success[root@dbi-oda-x8 ~]# odacli list-vnetworksName Type Interface Bridge Uniform Created Updated-------------------- --------------- --------------- -------------------- -------- ----------------------- -----------------------tagged38 BridgedVlan btbond2 brtagged38 NO 2022-02-04 15:19:07 CET 2022-02-04 15:19:07 CETpubnet Bridged btbond1 pubnet NO 2022-01-28 08:54:55 CET 2022-01-28 08:54:55 CETtagged39 BridgedVlan btbond2 brtagged39 NO 2022-02-17 16:49:45 CET 2022-02-17 16:49:45 CET[root@dbi-oda-x8 ~]# odacli describe-vnetwork -n tagged39VNetwork details-------------------------------------------------------------------------------- ID: 6e40ca21-1d4a-4bcc-8436-f29ffb2eb8bc Name: tagged39 Created: 2022-02-17 16:49:45 CET Updated: 2022-02-17 16:49:45 CET Type: BridgedVlan Interface name: btbond2 Bridge name: brtagged39 VLAN ID: 39 IP: Netmask: Gateway: Attached in VMs (config): NONE Attached in VMs (live): NONE[root@dbi-oda-x8 ~]# ip addr sh | grep btbond24: p7p3: mtu 1500 qdisc mq master btbond2 state UP group default qlen 10005: p7p4: mtu 1500 qdisc mq master btbond2 state UP group default qlen 100010: btbond2: mtu 1500 qdisc noqueue state UP group default qlen 100011: btbond2.38@btbond2: mtu 1500 qdisc noqueue master brtagged38 state UP group default qlen 100031: btbond2.39@btbond2: mtu 1500 qdisc noqueue master brtagged39 state UP group default qlen 1000[root@dbi-oda-x8 ~]# ip addr sh | grep brtagged11: btbond2.38@btbond2: mtu 1500 qdisc noqueue master brtagged38 state UP group default qlen 100012: brtagged38: mtu 1500 qdisc noqueue state UP group default qlen 1000 inet 10.38.0.10/24 brd 10.38.0.255 scope global brtagged3825: vnet6: mtu 1500 qdisc pfifo_fast master brtagged38 state UNKNOWN group default qlen 100030: brtagged39: mtu 1500 qdisc noqueue state UP group default qlen 100031: btbond2.39@btbond2: mtu 1500 qdisc noqueue master brtagged39 state UP group default qlen 1000[root@dbi-oda-x8 ~]#

And let’s deploy the srvdb39 DB Systems again after having removed it with odacli delete-dbsystem command :

[root@dbi-oda-x8 ~]# odacli list-dbsystemsName Shape Cores Memory GI version DB version Status Created Updated-------------------- ---------- ----- ---------- ------------------ ------------------ --------------- ----------------------- -----------------------srvdb38 odb2 4 16.00 GB 19.13.0.0.211019 19.13.0.0.211019 CONFIGURED 2022-02-04 16:12:33 CET 2022-02-04 17:07:52 CET[root@dbi-oda-x8 ~]# odacli create-dbsystem -p /opt/dbi/create_dbsystem_srvdb39.jsonEnter password for system "srvdb39":Retype password for system "srvdb39":Enter administrator password for DB "DB39":Retype administrator password for DB "DB39":Job details---------------------------------------------------------------- ID: 934c752b-7eb6-4b5b-8856-7a90102a1e65 Description: DB System srvdb39 creation Status: Created Created: February 17, 2022 5:06:37 PM CET Message:Task Name Start Time End Time Status---------------------------------------- ----------------------------------- ----------------------------------- ----------[root@dbi-oda-x8 ~]# odacli describe-job -i 934c752b-7eb6-4b5b-8856-7a90102a1e65Job details---------------------------------------------------------------- ID: 934c752b-7eb6-4b5b-8856-7a90102a1e65 Description: DB System srvdb39 creation Status: Success Created: February 17, 2022 5:06:37 PM CET Message:Task Name Start Time End Time Status---------------------------------------- ----------------------------------- ----------------------------------- ----------Create DB System metadata February 17, 2022 5:06:37 PM CET February 17, 2022 5:06:37 PM CET SuccessPersist new DB System February 17, 2022 5:06:37 PM CET February 17, 2022 5:06:37 PM CET SuccessValidate DB System prerequisites February 17, 2022 5:06:37 PM CET February 17, 2022 5:06:41 PM CET SuccessSetup DB System environment February 17, 2022 5:06:41 PM CET February 17, 2022 5:06:43 PM CET SuccessCreate DB System ASM volume February 17, 2022 5:06:43 PM CET February 17, 2022 5:06:49 PM CET SuccessCreate DB System ACFS filesystem February 17, 2022 5:06:49 PM CET February 17, 2022 5:06:58 PM CET SuccessCreate DB System VM ACFS snapshots February 17, 2022 5:06:58 PM CET February 17, 2022 5:07:31 PM CET SuccessCreate temporary SSH key pair February 17, 2022 5:07:31 PM CET February 17, 2022 5:07:32 PM CET SuccessCreate DB System cloud-init config February 17, 2022 5:07:32 PM CET February 17, 2022 5:07:32 PM CET SuccessProvision DB System VM(s) February 17, 2022 5:07:32 PM CET February 17, 2022 5:07:33 PM CET SuccessAttach disks to DB System February 17, 2022 5:07:33 PM CET February 17, 2022 5:07:34 PM CET SuccessAdd DB System to Clusterware February 17, 2022 5:07:34 PM CET February 17, 2022 5:07:34 PM CET SuccessStart DB System February 17, 2022 5:07:34 PM CET February 17, 2022 5:07:36 PM CET SuccessWait DB System VM first boot February 17, 2022 5:07:36 PM CET February 17, 2022 5:08:48 PM CET SuccessSetup Mutual TLS (mTLS) February 17, 2022 5:08:48 PM CET February 17, 2022 5:09:08 PM CET SuccessExport clones repository February 17, 2022 5:09:08 PM CET February 17, 2022 5:09:08 PM CET SuccessSetup ASM client cluster config February 17, 2022 5:09:08 PM CET February 17, 2022 5:09:11 PM CET SuccessInstall DB System February 17, 2022 5:09:11 PM CET February 17, 2022 5:36:57 PM CET SuccessCleanup temporary SSH key pair February 17, 2022 5:36:57 PM CET February 17, 2022 5:36:57 PM CET SuccessSet DB System as configured February 17, 2022 5:36:57 PM CET February 17, 2022 5:36:57 PM CET Success[root@dbi-oda-x8 ~]# odacli list-dbsystemsName Shape Cores Memory GI version DB version Status Created Updated-------------------- ---------- ----- ---------- ------------------ ------------------ --------------- ----------------------- -----------------------srvdb38 odb2 4 16.00 GB 19.13.0.0.211019 19.13.0.0.211019 CONFIGURED 2022-02-04 16:12:33 CET 2022-02-04 17:07:52 CETsrvdb39 odb2 4 16.00 GB 19.13.0.0.211019 19.13.0.0.211019 CONFIGURED 2022-02-17 17:06:37 CET 2022-02-17 17:36:57 CET[root@dbi-oda-x8 ~]# odacli describe-dbsystem -n srvdb39DB System details-------------------------------------------------------------------------------- ID: ed13bacb-be28-4e31-a5d7-39fd713d72a0 Name: srvdb39 Image: 19.13.0.0.0 Shape: odb2 Cluster name: dbs793d6c5ce Grid version: 19.13.0.0.211019 Memory: 16.00 GB NUMA enabled: YES Status: CONFIGURED Created: 2022-02-17 17:06:37 CET Updated: 2022-02-17 17:36:57 CET CPU Pool-------------------------- Name: cpupool4dbsystems Number of cores: 4 Host: dbi-oda-x8 Effective CPU set: 5-6,21-22,37-38,53-54 Online CPUs: 5, 6, 21, 22, 37, 38, 53, 54 Offline CPUs: NONE VM Storage-------------------------- Disk group: DATA Volume name: S793D6C5CE Volume device: /dev/asm/s793d6c5ce-390 Size: 200.00 GB Mount Point: /u05/app/sharedrepo/srvdb39 VMs-------------------------- Host: dbi-oda-x8 VM Name: x793d6c5ce VM Host Name: srvdb39.dbi-lab.ch VM image path: /u05/app/sharedrepo/srvdb39/.ACFS/snaps/vm_x793d6c5ce/x793d6c5ce Target State: ONLINE Current State: ONLINE VNetworks-------------------------- Host: dbi-oda-x8 VM Name: x793d6c5ce Public: 10.39.0.20 / 255.255.255.0 / ens3 / BRIDGE(brtagged39) ASM: 192.168.17.5 / 255.255.255.128 / ens4 / BRIDGE(privasm) VLAN(priv0.100) Extra VNetworks-------------------------- Host: dbi-oda-x8 VM Name: x793d6c5ce tagged39: 10.39.0.20 / 255.255.255.0 / PUBLIC Databases-------------------------- Name: DB39 Resource ID: 6ecee323-cd18-4971-acb4-b476d4d76d2c Unique name: DB39_SITE1 Database ID: 2044811647 Domain name: dbi-lab.ch DB Home ID: 3a6f8134-407a-4272-8512-7c52bdca68ea Shape: odb2 Version: 19.13.0.0.211019 Edition: EE Type: SI Role: PRIMARY Class: OLTP Storage: ASM Redundancy: Target node name: Character set: AL32UTF8 NLS character set: Language: ENGLISH Territory: AMERICA Console enabled: false SEHA enabled: false Associated networks: Public-network Backup config ID: Level 0 Backup Day: sunday Autobackup enabled: true TDE enabled: false CDB type: false PDB name: PDB admin user:

This DB System is also now accessible through his IP address 10.39.0.20 and is using the bridge interface without any IP address. A bridge interface does not need any IP address.

I can ping the new DB System from my laptop :

maw@DBI-LT-MAW2 ~ % ping 10.39.0.20PING 10.39.0.20 (10.39.0.20): 56 data bytes64 bytes from 10.39.0.20: icmp_seq=0 ttl=62 time=56.761 ms64 bytes from 10.39.0.20: icmp_seq=1 ttl=62 time=56.734 ms64 bytes from 10.39.0.20: icmp_seq=2 ttl=62 time=64.759 ms^C--- 10.39.0.20 ping statistics ---3 packets transmitted, 3 packets received, 0.0% packet lossround-trip min/avg/max/stddev = 56.734/59.418/64.759/3.777 ms

I can ssh the new DB System from my laptop :

maw@DBI-LT-MAW2 ~ % ssh root@10.39.0.20root@10.39.0.20's password:Last login: Thu Feb 17 18:10:10 2022 from gateway[root@srvdb39 ~]# ip addr sh ens32: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 52:54:00:86:87:88 brd ff:ff:ff:ff:ff:ff inet 10.39.0.20/24 brd 10.39.0.255 scope global noprefixroute ens3 valid_lft forever preferred_lft forever [root@srvdb39 ~]# ps -ef | grep [p]monoracle 93401 1 0 17:35 ? 00:00:00 ora_pmon_DB39[root@srvdb39 ~]#

And I can join the DB39 database from my laptop directly :

Conclusion

KVM DB System main purpose is hard partitioning for oracle licensing. All ODA’s cpu cores can remain active. DB System with appropriate pinned CPU core related to the number of licenses will be created for the oracle database. The others ODA’s CPU core can be used for further KVM or application needs.
As we could see KVM DB System will then have an additional pros. With DB System we can separate Database traffic through separated VLANs.

On the other hand KVM DB System will have some other cons :

  • More complex installation.
  • More resources will be consumed : each DB Systems will run its own oracle restart (grid) infrastructure.
  • Each DB Systems will have to be patched separately as well during ODA patching, making patching process more heavy.

Also please note that current ODA release 19.13 supports only creation of database in version 19.11, 19.12, 19.13, 21.3 and 21.4 for the DB Systems.

L’article Creating KVM Database System on separate VLAN network on ODA est apparu en premier sur dbi Blog.

Top Articles
Latest Posts
Article information

Author: Nathanael Baumbach

Last Updated: 03/03/2023

Views: 5730

Rating: 4.4 / 5 (55 voted)

Reviews: 94% of readers found this page helpful

Author information

Name: Nathanael Baumbach

Birthday: 1998-12-02

Address: Apt. 829 751 Glover View, West Orlando, IN 22436

Phone: +901025288581

Job: Internal IT Coordinator

Hobby: Gunsmithing, Motor sports, Flying, Skiing, Hooping, Lego building, Ice skating

Introduction: My name is Nathanael Baumbach, I am a fantastic, nice, victorious, brave, healthy, cute, glorious person who loves writing and wants to share my knowledge and understanding with you.