VMware Cloud Expert

Lab 5 - Log Management

Updated on

Introduction

VMware Aria Operations for Logs is a part of the VMware Cloud suite of services. Use this service to develop sophisticated analytics that aid in rapid troubleshooting of your SDDC or VMware Cloud on AWS environment..

As part of the VMware Cloud suite of services, VMware Aria Operations for Logs (formerly known as vRealize Log Insight Cloud) provides a fully managed and integrated log analytics and troubleshooting service.

VMware Aria Operations for Logs includes VMware-authored SDDC (ESXi, VC, NSX, and VSAN) insight for troubleshooting, a flexible and comprehensive query facility that supports troubleshooting for novice and experienced administrators, built-in SDDC and custom alerting capability, flexible notification mechanisms, and centralized support for local or federated authentication.

Setting up Log Insight Cloud for non- Cloud services organization

Setting up VMware Aria Operations for Logs for VMware Cloud Subscribers

TASKS

Task 1 - Enabling Firewall logging and generating log entries
  1. From your VDI Desktop, open the browser and log into your VMC on AWS SDDC
    https://vmc.vmware.com/console/sddcs
  2. On your SDDC Tile Click View Details
  3. Select Open NSX Manager (next to Open vCenter)
    • Select the blue box Access Via the Internet
  4. Click the Security Tab
  5. Modify the Distributed Firewall
  6. Click Add Policy
  7. Name the Policy Class Log Test
    • Check the box next to the policy you created, Add Rule is now available
  8. Select Add Rule
  9. Name the Rule Allow All HTML
  1. Under Sources click the pencil, select RFC1918, click Apply
  2. Under Destination click the pencil, select RFC1918, click Apply

 

  1. Under Services click the pencil, filter for http, select HTTP and HTTPS, click Apply
  2. Under Actions leave Allow selected, the rule should be enabled by default
  3. Publish the DFW rule by clicking the blue Publish button
  1. Select the Gear in the far right side of the Allow All HTML rule
  2. Move the slider next to logging to enable logging 
  3. Set the Log Label to vmcexpert#-##_Test (using your Student ID)
  4. Click Apply 
  5. Click Publish
  1. Click Open vCenter from the SDDC Console
  2. Click Show Credentials
  3. Copy the Password and Click Open vCenter
  4. Log into vCenter as:
  5. In vCenter, Click the frontend VM and record its IP address
  1. Select the Ubuntu-DT VM and click Open Console
  2. Enter the Password of VMware1! if prompted
  3. Launch the Firefox browser in the Ubunt-DT VM and type in {the address of your Frontend VM} for the Cats & Dogs Application
  4. Click the Gato & Cachorro buttons multiple times until the image that appears is a hedgehog.
  5. When this image (hedgehog) appears an error is generated and logged with log-insight and captured by vRealize Operations.  You are also performing this step to generate some firewall logs
  6. In a new Browser tab, go to VMware.com and one or more public websites
Task 2 - VMware Aria Operations for Logs Overview
  1. Click the stacked squares in the upper right-hand corner
  2. Right-Click VMware Aria Operations for Logs
  3. Click Open link in new tab
  1. If Collapsed, Click the double arrows to expand the left pane
    • Expand the Configuration Section
    • Click Subscriptions
  2. Notice we are using the PERPETUAL Subscription. This subscription comes with VMC on AWS and only allows you to view Audit logs and Firewall Logs.
    To view other types of logs (Application logs, non-SDDC logs, etc..) the subscription must be upgraded.
  1. Click Dashboards to view the available dashboards
  2. In the Search Bar, type Gateway Firewall
  3. Select Gateway Firewall - Traffic (latest version)
  4. Review the Pie Graphs for Top Sources and Top Destinations
    You'll see an aggregated and processed view of all network traffic leaving the SDDC in the past 5 mins.
  5. In the Upper right corner note that you can increase the time scale to review date beyond the last 5 mins

All NSX related log events must first be enabled before those log messages will be sent to Log Insight. E.G. If you want to see DHCP, NAT or VPN related log messages then you must enable logging for those services as we did in task 1 for our firewall rule.

  1. Select Dashboards in the left column
  2. Click on All Dashboards:
    • Select NSX-T events for VMware Cloud SDDC v6.0
    • Select Distributed Firewall - Traffic
  3. Observe the traffic, you may have to change the time window by selecting 30M or 1H to see relevant data
  1. From the left hand navigation pane, select Home
  2. In the search bar, type vmcexpert* to see traffic that has PASSed the DFW rule.  You may need to change the time period to 30M or 1H.  
    • HINT:  Use your full student account name (vmcexpert#-##) to see events for your SDDC only.    
  1. Return to the VMware Cloud on AWS SDDC console tab in your browser, ensuring that you are in your SDDC
    • Select Networking & Security
    • Select Distributed Firewall under Security
  2. Open the Class Log Test policy to see the Allow All HTML rule
  3. Change Allow to Reject under Action
  4. Publish the rule change
  1. Return to the Ubuntu Desktop Console window, relaunch from vCenter if it has timed out.
  2. Open Firefox and connect to your FrontEnd VM IP address recorded earlier making sure to use http.
    • This should result in an immediate unable to connect message
    • If the DFW rule had been set to DROP, you would have to wait for the http timeout to see the failed message
    • From the Firefox web browser access vmware.com, try google.com.  That access isn't blocked by the DFW.
  3. Return to your open Aria Operations for Logs browser tab, or start a new Aria Operations for Logs session if closed
  4. From the left hand navigation pane, select Home
  5. In the search bar, type vmcexpert* to see traffic that has been REJECTed by the DFW rule.  You may need to change the time period to 30M or 1H.  
    • HINT:  Use your full student account name (vmcexpert#-##) to see events for your SDDC only.
  1. From the Aria Operations for Logs navigation pane, select Dashboards
  2. Click the All Dashboards drop down
    • Select NSX-T events for VMware Cloud SDDC V6.0
    • Select Distributed Firewall - Traffic
  3. Observe Application Ports Denied,  there should be entries from the REJECT rule.  You might need to adjust the time band to 30M or 1H
  4. When finished, return to the SDDC console, updating the DFW setting back to ALLOW from REJECT and PUBLISH the rule.
    • You can Verify that the DFW is set correctly by accessing the Cats & Dogs app from your Ubuntu Desktop
  1. Click Alerts in the left hand navigation pane
  2. Click Alert --> Alert Definitions to review the built-in alerts
  3. In the search bar type  vcenter and hit return to see the Alert definitions
  4. Select the Audit Events for VMware Cloud SDDC | User Session Login Alert
  5. Review the settings and notice there is currently no notification set when this alert is triggered, the rule is also disabled by default.
  6. In the upper right hand, click the edit Icon, close any warning for partitions if present.  
  1. Under Trigger Condition 1, click Choose Notification and  Input Your Email address in the notify field, then click the +
  2. Move the slider to enable the alert
  3. Click Save
  4. If logged into vCenter, logout.  Login to your vCenter server.  You should receive both user and application login notifications.
  5. After verifying incoming email notifications, Disable the alert by moving the slider, edit the alert and remove your email address by clicking the 'x' and then Save.

Note: You can also create custom alerts.

  1. In the left pane, click Content Packs to review the available content packs for Log Insight Cloud
  2. Notice that not all content packs are enabled. Enabling a content pack allows Log Insight to begin processing log messages for the system
  3. If Disabled, Enable the content packs for:
    • Audit Events for VMware Cloud SDDC (v2)
    • General
  1. Select Public, then Applications
    • NOTE:  You may need to clear the search bar in order to see the desired results
  2. Enable Apache - HTTP Server and Nginx
  3. Select Others
  4. Enable the Linux and Linux - Systemd Content Packs, if not already enabled
  1. Click on Dashboards.  In the search bar type the name of one of the content packs you enabled (Linux, Nginx etc).  You'll now notice additional dashboards (Activity, Alerts, etc...)
Task 3 - Application Logs
  1. In the log Insight cloud interface expand Configuration
  2. Click Cloud Proxy
    If there are any existing inactive Proxies, click Delete and confirm the deletion to remove them
  3. Click ADD Proxy
  4. Click Existing
  5. Select the Aria Automation Proxy you deployed earlier
  6. Click Add
  1. In the left pane under the Configuration section, click Operations for Log Agents
  2. Under Agent Configuration clink New next to File Logs, add configuration settings for the following.  Note: hit Save as you complete each section.
    • MongoDB
      • Directory: /var/log/mongodb
      • include files: *.log
    • syslog
      • Directory: /var/log
      • Include files: *.log
    • docker
      • Directory: /var/lib/docker/containers
      • Include files: *.log
  1. At the top of the page, in the Agents search field, click the drop-down and select Create New Group
  2. Name the Group Linux_XX (Linux_01) matching your student number
  3. Click OK
  1. Configure the Group as follows:
    • Filter
      • OS
      • Starts with
      • Ubuntu*
  1. Under the Agent Configuration section add the following configuration settings:
    • Under General Click New, and add the general section logging
      • Set the logging level to verbose
    • Click Save
      • Under Common Click New, and add the section global
    • Click Save
  1. In the Agent Configuration section, Under Parsers, Let's create and define 4 parsers:
    1. Next to Parsers Click New, name the section syslog_appname_parser 
      • Set the Parser to use/extend to CLF (default Common Log Format)
      • Set Format to:
%{appname}i[%{thread_id}i]
Click to copy

Click Save

  1. Next to Parsers Click New, name the section syslog_parser
    • Set the Parser to use/extend to CLF (default Common Log Format)
    • Decode Field click Add
      • set the field to appname
      • set the value to syslog_appname_parser
    • Set Format to:
%t %i %{appname}i: %M
Click to copy

Click Save

  1. Next to Parsers Click New, name the section auth_Parser_sles
    • Set the Parser to use/extend to CLF (default Common Log Format)
      • Set Next Parser to syslog_parser
      • Set Format to:
%t %i %{appname}i[%{thread_id}i]: password changed - account=%{linux_user}i, uid=%{uid}i, %i
Click to copy

Click Save

  1. Net to Parsers Click New, name the section auth_Parser
    • Set the Parser to use/extend to CLF (default Common Log Format)
      • Set Next Parser to syslog_parser
      • Set Format to:
%t %i %{appname}i[%{thread_id}i]: password changed - account=%{linux_user}i, uid=%{uid}i, %i
Click to copy

Click Save

  1. In the Agent Configuration section, Under File Logs, Let's create and define 4 log configurations:
    1. Next to File Log Click New, name the section auth
      • Set the Directory to: /var/log
      • Set Include files to: auth.log;auth.log.?
      • Tag Field click New Tag
        • set the field to vmc_cp
        • set the value to linux
      • Set Parse fields by to: auth_parser

Click Save

  1. Next to File Log Click New, name the section messages
    • Set the Directory to: /var/log
    • Set Include files to: messages;messages.?
    • Tag Field click New Tag
      • set the field to vmc_cp
      • set the value to linux
    • Set Parse fields by to: syslog_parser

Click Save

  1. Next to File Log Click New, name the section syslog
    • Set the Directory to: /var/log
    • Set Include files to: syslog;syslog.?
    • Tag Field click New Tag
      • set the field to vmc_cp
      • set the value to linux
    • Set Parse fields by to: syslog_parser

Click Save

  1. Next to File Log Click New, name the section docker
    • Set the Directory to: /var/log
    • Set Include files to: docker
    • Tag Field click New Tag
      • set the field to vmc_cp
      • set the value to linux
    • Set Parse fields by to: syslog_parser

Click Save

  1. At the top of the VMware Aria Operations for Logs Agents page choose All Agents in the search field

You should now see the agents from your Cats & Dog application, there would also be more than 10 events recorded.  Click on the Status field to sort by Active if you see multiple Disconnected agents.

Return to the Aria Operations for Logs Explore Logs and observe a change in the data stream.  You can adjust the time band as necessary, 5M - 30M - 1H etc.

Conclusion

0 Comments

Add your comment

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.