Symptoms for missing Multi Factor Authentication (MFA) app password in O365:
You enabled Multi-factor Authentication (MFA) on Office 365 (O365) portal or Microsoft 365
MFA works, and your user(s) are able to log on to the office.com website using it.
You don’t see the “app password” listed anywhere, and the Account Settings view seems to be missing that section.
You set up a conditional access policy for Multi Factor, per the Microsoft how-to article.
What are app passwords in office 365 or Azure?
App passwords are unique passwords used to authenticate legacy software to Office 365.
The primary use case is Outlook 2010 , or Outlook 2016. For example, these could have been standalone deployments before Office 365 was adopted. These older versions of Office may not know how to handle a multi-factor prompt.
If your organization uses Office 365, I recommend installing the latest version of Office 365 from office.com. The latest versions know how to handle multi-factor and don’t need app passwords.
Most users don’t use app passwords.
What does missing app password look like?
You should be able to view or create app passwords from the user’s account settings in Office 365.
If you are having this issue, the area for app passwords is blank. The picture above shows a missing app password area. It should be right under Contact Preferences.
Root cause for missing multi-factor app password:
The Office 365 admin portal has two separate ways to enable MFA for users. One way is to use the Admin > Users > Multi-factor Authentication menu for individual users. The second way is to set up a Conditional Access Policy. Microsoft articles say to use the Conditional Access Policy, but there is a problem with that.
There appears to be a bug where app passwords aren’t enabled properly when you use Conditional Access Policies.
To fix the problem, you need to enable multi factor using the Users menu, not conditional access policies.
The good news is that you don’t need to abandon your conditional access policy entirely. Just use this fix for individuals who need an app password, not everyone.
How to fix missing multi-factor authentication app password in Office 365
To enable application passwords in O365, you need to use the Multi-factor authentication page in admin center.
Navigate to Office 365, log on with your admin account
Click Admin from the menu
Click Users > Active Users from the menu
In the center, you will see a link to “Multi-factor authentication”. Click this link.
Your users list will display. You can filter for individual users here.
Pick the user that needs an app password and click “Enable”.
Enable the user here even if you have already set up a conditional access policy for MFA that includes the user.
For most organizations, you should be fixed now. When I tested, this took effect immediately and I create my app password as the user within 30 seconds.
Recently (May 2020 and later), we are seeing more complex issues that are not resolved by enabling MFA in the users area. Read on for information about fixing those issues.
Modern Authentication fix for missing O365 app password
Thanks to Rob Ryan for sending me these fix steps. Cheers!
Symptoms:
The App Password is missing no matter if we enable or disable MFA from Admin Center > Users.
Disabling MFA did not remove the requirement for multi-factor when we log on with an incognito browser.
We could not find any conditional access policies, and no one had set up any.
Tested email connectivity using testconnectivity.microsoft.com (Outlook test). This failed with an error saying that a conditional access policy denied access.
His account definitely had conditional access policies applied but they were hidden, possibly because his O365 license didn’t have Active Directory Premium 2 rights.
We eventually put in a ticket to Microsoft support, and they pointed us in the correct direction.
The issue was a setting under Office 365 Admin Panel. There is a setting for “Modern Authentication”, that needs to be turned off for some apps that have a different or older Exchange Authentication process.
(Admin center > Settings > Org Settings > Services Tab > Modern Authentication)
This setting can be found in the Admin Panel by clicking “Settings” in the left hand menu panel and then clicking “Org Settings” under the drop down menu. Then click on “Modern Authentication” in the list of settings. Toggle to “off” wait 2 – 3 hours.
Then to set “App “Password, go under the user My Account, “Security & privacy”, “Additional security verification”, “Create and manage app passwords”. In older versions of Office 365 the App Password option was in the title banner of the page, in larger font next to “Security Info”. It is now found by clicking on the “+ Add Method Button”, when available.
Links for O365 Modern Authentication articles by Microsoft:
Thanks to Jim Hill for sending these alternate fix steps which worked for him. You can see his original message in the comments. Cheers!
I figured out a solution! I am not sure if these are the exact steps I did. But the root cause was previously having a conditional access policy previously applied to that user.
-Make sure the CA policy for MFA is not enabled. Mine was the Legacy rule, “Baseline policy: Require MFA for admins (Preview)” -Disable the MFA for that user in the Office 365 Admin. https://account.activedirectory.windowsazure.com/UserManagement/MultifactorVerification.aspx?BrandContextID=O365 -Create a new CA policy in Azure AD. Have it grant access and require MFA, and only apply just to that user. -Go back to the Office 365 MFA admin and enable then disable the MFA requirement for that user a few times. I did it four times. Leave with it disabled. -Return to Azure and remove that user from the CA policy requiring MFA. Leave it enabled but just not applied to any users. -Return to Office 365 admin and enable MFA for that user. -Initiate a screen sharing session with that user. Have them log into their MFA set up screen. https://aka.ms/MFASetup They will then see and be able to create a new app password.
This took me a month to figure out, and my exact steps may not have been exactly as I said above. Hopefully the whole community can figure out the best way.
Where can I manage Office 365 app passwords for multi factor?
Log on to office.com with the user account that needs an app password
Click on your account icon at the top right corner (normally shows your initials)
I recently went through the steps to set up conference call bridges on Microsoft Teams for my own organization. Specifically, this means dial-in phone numbers that I or people outside my organization can use for a conference line. Here is how I did it, with tips and information to save you time.
This article is for you if:
You are an Office 365 Admin for your organization
You have at least one user with Office 365 E3 or E5, Education, or Government plans. Note: Looks like Office 365 “Business” does not include Teams. Office 365 E5 includes the “Audio Conferencing” license for each user, which will save you some money.
You want your user to be able to host conference calls with a phone number that non-teams people can dial.
I’m excited about using Teams for conference calling. I’ve used many other services such as Webex, Zoom, GoToMeeting over the years but it is annoying that they don’t automatically synchronize with Outlook. Microsoft Teams has a major advantage in that you can see availability, invite people, and reserve your calendar and room all in one place. It is also a great price at $4 per month per conference leader. And it is really easy to set up and use!
The Microsoft documentation for Teams is terrible (sorry, Microsoft, I love you, but seriously!), which is why I wrote this article. I read through all of it and still ended up contacting support several times. It doesn’t have to be that hard. Read through my article to easily set up Microsoft Teams conference calling. Note: It will take about 24 hours for the feature to enable itself.
How much does a conference line for O365 Teams cost?
At the time of this writing (Oct 2019), there are two license options.
“Audio Conferencing” costs $4 per user per month. This “Enables users to dial-in a number to join meetings, or dial-out to bring participants into the meeting”. Per Microsoft support, this is the correct license for our purposes here. This should enable basic conference call ability, very similar to WebEx or Zoom. This license is only needed for conference hosts (people who organize the meeting). Participants do not need an audio conferencing license. Per Microsoft, a conference number can support up to 250 callers.
“Phone system” costs $8 per user per month. This is “for organizations that need call management capabilities (make, receive and transfer calls in the cloud…” I haven’t tested this license fully yet. I think it is for users who need their own phone number for make calls (similar to having a phone at their desk). Update: You do not need this license to host a conference bridge with phone number. But if you get it, it will also handle the conference calling.
If you want to test this out without spending any money, get a trial subscription to “Office 365 E5” – this includes the Office prerequisite and the Audio Conferencing license.
Choose “Add-ons” – this is a link at the bottom of the page.
I selected “Audio Conferencing” and for my purposes, a monthly commitment. Unchecked “automatically assign to all of your users with no licenses” since I only need a conference line for one person right now.
I followed the purchase license wizard. Success.
I went back to the Admin center > Users
Select the user I want to host conference calls.
In the user management window that appears, I selected the Licenses and Apps menu item. This brings up a list of available licenses.
I add a check to the “Audio Conferencing” license and save changes.
You user needs an Office 365 license (Enterprise, Government, or School since these include Teams) and the Audio Conferencing license.
2. Wait for 24 hours for the audio conferencing license to synchronize
If this is the first time you’ve activated an audio conferencing license, it can take several (up to 24) hours for Microsoft to enable the bridge ID / conference line functionality. When I didn’t wait, I encountered lots of errors on the Teams Admin site. I don’t know if the setup time applies to future audio conference licenses as well (please comment if you know).
Symptoms of pending Microsoft Teams synchronization:
Your O365 user has NOT gotten an email about their audio conference yet.
From Teams Admin, go to Legacy Portal (Skype for Business admin center) and select “audio conferencing” , then select “users” from the top menu… when I select my user, it shows the message “Microsoft audio conferencing license status Pending: 1”
From Teams Admin, go to Legacy Portal and select Audio > Phone numbers. Select a phone number and edit it. The Name field is blank. It will give an error if you try to assign it.
From Teams Admin, go to Users. Select a user that is licensed for audio conferencing and O365. At the bottom of the page, you will see “audio conferencing”. When you try to edit this option, you cannot enable it.
3. Your user will get an email with conferencing dial-in for Teams or Skype
Within 24 hours (it took 20 hours for me), your user will get an email from Microsoft
This email will tell the user that they have Audio Conferencing for Microsoft Teams
It will give them their conference phone number
It will provide the conference leader PIN (used if you are leading the meeting without the Teams app)
If you open O365 Admin > Teams Admin > Voice > Phone Numbers, you will see a new phone number appear here. It will be “assigned”.
Your Microsoft Conference Calling should work now!
If you use Outlook to schedule a Meeting, you will see an icon in the top menu that says “Teams Meeting” or “New Teams Meeting”. Use the Teams Meeting” option to automatically include dial-in and online meeting information to your Outlook invite.
If you don’t see this option, make sure that your user is logged into their Office 365 account with their office products. This should be already be working if they are using the email address that is registered in Office 365.
Creating your first audio conference (with phone #) meeting in Teams App
Once your user has gotten the Audio Conferencing welcome email, you can try it out!
Have your user close and reopen Microsoft Teams (this may be an extra step, but just in case…).
In Teams app, go to Calendar
Click the + New Meeting button
Fill out the meeting details (name, start date and time). Invite some people – in my case, I invited someone who doesn’t use Teams. Click Schedule.
Don’t worry that the audio conference information isn’t displayed. It will appear on the next step.
You will see the meeting confirmation and options. Right now, your invitees are getting emails and your O365 outlook calendar will update with the meeting.
The dial-in phone number and conference ID for teams will display. Note that the Conference ID should change for each meeting.
On this screen, you can change the Meeting options. I recommend modifying the “Who can bypass the lobby” option to “everyone“, especially if you will be hosting the call from your phone.
Outlook audio conference (with phone #) Teams meeting
You can also create a conference-call enabled Teams meeting directly from Outlook’s Calendar.
Schedule a normal meeting in Outlook, but before you send it out, click the “Teams Meeting” button. This adds the conference to it (you will see the conference information in the meeting body).
You can also open existing Outlook meetings and click the Teams Meeting button to add conference calling to them.
Teams automatically synchronizes with your Outlook Calendar. This is the meeting information as viewed by the host. From here you can forward to additional invitees (the dial-in information is included) or change other settings.
Again, automatic synchronization with Outlook. It will remind you to start your Teams meeting.
This is what a Teams conference invite looks like to people outside your organization (who don’t have teams). The audio dial-in number is listed in the invite.
When your meeting starts, remember that by default, dial-in participants will be stuck in the lobby. You can allow them to join the meeting from the Teams app.
If you will be dialing in to your own meeting, I recommend setting the meeting options to allow Everyone to bypass the lobby.
About assigning phone numbers and meeting IDs in Teams
When you set up your first conference phone number in Teams, it looks like all licensed users will use that phone number by default. This may change if you are a large organization with more users. If you know how this works with more users, please leave a comment below!
Different meetings on that phone line will be split out using the Meeting ID, which is a different (dynamically assigned) number for each meeting. So potentially your meeting host could run two different meetings simultaneously.
This is the same way that conferences work on WebEx or Zoom. Many meetings can use a single dial-in phone number at the same time. Each meeting uses a different Conference ID.
If you read through the Microsoft KB articles on this process, the steps to create a new phone number show up everywhere. From what I can tell, reserving a phone number doesn’t cost anything. However, there doesn’t seem to be any advantage to having more than one phone number (for small/med business audio conferences). You can’t use the phone numbers without licensing your user(s). For basic conference setup, you only need one phone number which should get created automatically and assigned by Microsoft when you assign the “audio conferencing” license to the user.
On the left menu, click … Show All to see the other admin centers. Pick the Teams admin center.
On the left menu, click Users.
Select the user you want to host dial-in meetings.
At the bottom of the user view, you will see an area for Audio Conferencing. You can click the Edit button to change your settings.
You may need to enable Audio Conferencing for the user. In my case, this area updated automatically 20 hours after I assigned the Audio Conferencing license to my user.
2. Reset Teams User PIN and Conference ID, re-send welcome email
If your user loses their audio conference information, particularly the leader code / PIN, you can reset it from the Teams admin center.
On the left menu, click … Show All to see the other admin centers. Pick the Teams admin center.
On the left menu, click Voice then Phone numbers
Click the + Add button to request phone numbers
The wizard will start.
Country or region: (I picked United States)
Number type: I picked Dedicated Conference Bridge (Toll). Toll normally means that the caller has to pay long-distance charges (not an issue for my clients). Toll-free means that my organization would pay long-distance charges for the caller, which could result in an unexpected expense, so I avoid it.
Location: I needed to Add a location – an option which appeared when I selected the drop down. In my case, I typed my office address in and it auto-resolved the rest. You can drag and drop the map locator to identify your precise coordinates. This is important for the “Emergency Location” identity, which is automatically provided to emergency services if someone uses that number to call for help (9-1-1 in the USA).
Area code: I picked my preferred area code (from a list of codes that match my location)
Quantity: I picked 1 for this exercise.
Order Name: I typed “001” for the name and “testing teams conference call phone order” for the description at the top of the page.
Next…
A progress circle displays “Thanks for your patience, but we are making sure we reserve the right numbers for you.”
Get numbers displays: You get 10 minutes to confirm your order. It displays the phone number you will get if you place the order. I clicked Next…
Success message. The website says that the phone numbers will appear in the list as soon as they are available. Finish.
The phone number displays in the list.
Error requesting Teams phone number
When I first tried this Teams wizard, I got an error: “We can’t get the list of available countries or regions.”
The second time I tried the Teams wizard, I got further, but then failed again at the last step (to reserve the phone number).
The support link sent me to PTN@microsoft.com . I emailed them with my woes and they responded immediately. I eventually completed the phone number registration process via email. However, I’m including the normal steps below in case you don’t see an error. The wizard has worked fine for me since then.
4. Not needed – use Teams / Skype Legacy Portal to assign a phone number
According to the Microsoft KB articles, you need to assign the phone number using Legacy portal to enable the conference bridge. In my experience, this was NOT necessary. Once my user got the welcome email, everything worked. Microsoft automatically generated a new phone number and assigned it to the user.
This setup may be needed once you have dozens of licensed audio conference users. Or if you don’t want to use the automatically assigned phone number for your organization.
On the left menu, click … Show All to see the other admin centers. Pick the Teams admin center.
On the left menu, click Legacy Portal (this takes you to the Skype for Business admin center)
On the left menu, select Voice > Phone Numbers
Select a phone number. You will see an option appear to “assign” the number. Click Assign.
This brings up an Assign dialogue. If synchronization is complete, you will see “Name: Conference bridge” listed.
If you assign here, your users will have that phone number assigned to them for their conference line. The meetings will be created individually using the Conference ID (just like WebEx and Zoom, you can have many different meetings with one phone number)
5. Not needed – Assign phone number to user in Teams Admin
This step appears to be used for the “phone system” license only (phone numbers assigned to users to replace their desk phones). Currently, the conference bridge assign functionality works better from the Teams Admin > Legacy Portal
On the left menu, click … Show All to see the other admin centers. Pick the Teams admin center.
On the left menu, click Voice > Phone Numbers.
Select a phone number. Click the Edit button.
Error: “You need to buy a Phone System license and make sure it’s been assigned to the user”. when I don’t have a phone system license….
If you have a phone system license… You will see a drop-down option for “Assigned to” –
You will have several options to pick from. The first option is a user line, which I think is appropriate for the Phone System license.
6. Not needed – Set up communications credits for international calls
Several Microsoft KB articles say to set up communications credits, and that they a “free”. I have not needed to do this yet. I believe this is used for toll-free calling, which is not necessary for most users. Please leave a comment if you know more!
I hope this article has been helpful to you. I am very excited about using Teams and Office 365 for official conference calls. It is the best value that I’ve seen out of the various competitors, and automatic synchronization with Outlook calendar is a time saving benefit.
Please comment if this helped you or if you can share your lessons learned with setting up Microsoft Teams for conference calls!
I am a consultant in the Maryland/DC area in the USA. My specialties are Windows enterprises (upgrades to 2016 and to Office 365 / Azure), VMware migrations, Netapp and SAN, and high availability / disaster recovery planning. If you would like help with your complex project, training, or would like a architectural review to improve your availability, please reach out! More information and contact can be found on the About page. – Amira Armond
This article is for you if your Exchange servers are filling up their C: drive even though you configured different drives for the mailbox databases and transaction logs.
Symptoms for Exchange 2016 C: drive use
C: drive used space increases 20-100 GB per month
Exchange 2016 Server or 2019 or 2013
Exchange mail flow services stops around 10-15 GB free
Root cause of C: drive full (Exchange 2013 , 2016, 2019)
Out of the box, Exchange 2016 has a lot of logging enabled, including performance logs. The performance logs can grow 2 GB per day, the inetpub logs grow about 300 MB per day, and various Exchange logs grow about 1 GB per day.
How to manually clean up the C: drive in Exchange 2016 or 2013 or 2019
Note: The logs discussed in this article are disposable. The file extensions should always be .log or .blg. Do not delete folders or other file types!
Remember to “permanently delete” the files by holding down SHIFT while selecting Delete. If you don’t they will just move to the recycle bin and not fix your disk space issue.
Navigate to these locations and delete older .log files
How to reduce C: drive use on Exchange 2013 2016 2019
The following steps show you how to disable some logs at the source. Remember, you can reverse these steps to begin generating logs in case you need to diagnose an issue.
1. Modify the registry to reduce the amount of some logs kept by default.
This article is about an infrequent issue I see with Windows Exchange servers that causes a complete outage. It is really hard to figure out what is wrong. Normally, you will see this when you restore an Exchange server to new hardware or a new virtual platform. But I’ve also seen it occur spontaneously with existing servers.
Symptoms for Exchange server very slow:
Windows 2008 R2 Server (normally). Also applies to 2012 and 2016.
Exchange 2007 or Exchange 2010 or Exchange 2013
Exchange Information Store service is “starting”, will not start
Services does not display and/or blank for a long time
Server manager information does not display or very slow
Network properties , NIC settings, does not come up
Control Panel does not display
Additional symptoms:
Rebooting does not work (and the server normally will not shutdown gracefully)
You can ping the server and ping from it, but remote desktop does not work.
(Often) The server has recently been restored from backup
(Often) The server is running in a virtual environment
(Often) Task manager shows low CPU activity
NETLOGON: If your services menu is *not* blank, and you can open your network settings, you have a different problem!
How to troubleshoot other causes of Exchange Info Store “starting”:
Check your “netlogon” service on the exchange server. Make sure it is started (and restart it).
Can’t start netlogon? Check that your network is functional and DNS settings are pointing at a domain controller. Yes, still check it even if the Domain Controller is the same server as Exchange.
Check that your Domain Controller has Active Directory and DNS running properly. From an admin command prompt, or admin powershell, run dcdiag
Check that your Domain Controller doesn’t have a firewall blocking it, or other network problems.
Check the event log for clues on both your Exchange server and your Domain controller.
Root cause for information store “starting” and services blank:
Exchange is trying to use a network adapter (NIC) that doesn’t exist anymore.
This normally happens when you restore to new hardware. For example, your server may have had one network card: “Network Connection 1”.
When you turn on Windows with new hardware, “Network Connection 1” is hidden and you will see “Network Connection 2” (or potentially a new “Network Connection 1”).
Most Windows servers handle this just fine, but with Exchange, the Information Store service is still trying to use the hidden network card. This locks up the Network Connections window, Control Panel, Services, and other administrative panels. The Information store churns at “starting” during this process.
How to fix NIC that doesn’t exist anymore:
You need to remove the hidden network adapter from your server’s Device Manager.
Click Start, click Run, type cmd.exe, and then press ENTER.
Type set devmgr_show_nonpresent_devices=1 and then press ENTER.
Open device manager (run devmgmt.msc or Right-click My Computer > Manage)
Type Start DEVMGMT.MSC, and then press ENTER.
From the top menu, click View, and then click Show Hidden Devices.
Expand Network Adapters.
Right-click the hidden network adapter, and then click Uninstall. If in doubt, I’d remove everything that looks like a network adapter. The server will re-install the live ones during reboot.
Give it a reboot.
You may need to re-configure your IP address on the existing NIC.
At this point your information store should start properly and your server will begin responding normally.
What to do if Device Manager won’t open?
If you can’t manage your server because none of the admin windows will open, you need to get the Exchange Information Store service stopped and disabled. Once you get the Information Store out of status “starting”, your network properties, device manager, control panel, etc, will open. This will let you fix the problem.
Here are some ideas to try.
If you are using a virtual server (VMWare, KVM, or HyperV), remove all virtual network adapters from the server. Reboot the Exchange server afterward. You should be able to get to Services now.
Try waiting for services to display. Give it 10 minutes.
Try using Computer Management from another computer to access the server (I don’t remember if this works).
Try using command prompt to stop and disable the Information Store service (I don’t remember if this works).
sc config "store" start= disabled sc stop "store"
Try booting into safe mode to stop and disable the Information Store Service.
If you are using a physical server, and nothing else is working, try removing the physical network adapter.
Helpful reference material (for this and other related errors):
I hope that works for you! If it does, or does not, please leave a comment to help others!
If you need a senior escalation point for Exchange, VMware, or Netapp technologies, consider Kieri Solutions. You can find more about me and my company in the menu above.
This blog describes my lessons learned with Exchange 2016 and 2019 Database Availability Groups. Particularly the information that most of us will need for the medium business market – less than 5 servers, multiple sites, etc.
This blog is for you if:
You are designing an Exchange 2016 or 2019 deployment
Trying to decide whether to use DAG or if it even supports your situation
You have multiple sites, but not a huge amount of email servers
You’d like to know common administrative steps and preventative maintenance
You are worried about user impact and unintended consequences of the DAG setup
If you are designing a DAG, trying to decide whether to use DAG, not sure how the servers will react, or how to administer them, read on!
Important disclaimer: I’m not from Microsoft, I’ve just done the work. These are my opinions and personal lessons-learned and may not be right for your organization.
How does Exchange 2016 and 2019 DAG work?
This is a simple DAG with 2 member Exchange Servers and one witness. When everything is going well, both Exchange servers host their active databases. The inactive database copies are kept in sync with constant communication between the DAG members.
Failover scenario: If one Exchange server goes down (reboots, hardware failure, etc), the other exchange server will check with the witness. The communicating DAG members (the 2 remaining servers) talk and decide that MBX2 should take over. All databases activate on MBX2. Active Directory and the Exchange organization redirects users to MBX2.
DAG across multiple sites or datacenters
Exchange 2016 DAG across multiple sites. . A copy of the mailbox database is held at both sites. The WAN link is used to synchronize the databases from active to passive.
Problem with multiple sites. This is what happens when DAG servers are across a WAN link that goes down. The single server will disable itself because it cannot reach the other servers. If your SITE-B clients can’t reach SITE-A (because the WAN link is down), they will lose email. This is a significant problem with DAG across multiple sites. Some organizations handle this problem by letting their users connect to Exchange servers from external (across the Internet) rather than only using internal links.
As far as I can tell, this DAG scheme (2 DAGs across 4 servers) is not supported. In my cursory checks of EAC, I don’t see an option to add my Exchange servers to a second DAG. It would be a great solution for the WAN link issue. If you have successfully configured this or a similar solution, please drop a comment and let me know!
How should I size my Exchange DAG hard drives and CPU / RAM?
Each of your exchange servers needs to be sized to hold all the mailbox databases that are on it.
In most small/medium organizations, all mailbox databases are synchronized over the DAG. In this case, ALL of your exchange servers in the DAG need to be able to hold and run ALL the mailbox databases.
Each database is only active on one server, but a full copy exists on EVERY server.
Each DAG server should be sized as though the other DAG partners don’t exist.
Example… under normal circumstances, your usage looks like this:
MBX1
1 active database (600 GB)
2 passive databases (1200 GB)
80 users
MBX2
2 active databases (1200 GB)
1 passive database (600 GB)
160 users
If MBX2 goes offline, the usage will look like this:
MBX1
3 active database (1800 GB)
0 passive databases
240 users
See why you need to build each server as though it is the only server?
For hard drive: You need disk space available to hold full copies of all databases, logs, etc.
For CPU and RAM: You need processing ability to respond to all client connections.
Lesson learned about C: drive space for Exchange
Even if you use a different log drive and database drive, your C: drive space will rapidly grow. A plain vanilla Exchange 2016 server will create logs at a rate of 30-50 GB / month on the C: drive. Once the C: drive reaches about 10-15 GB free, Exchange will disable itself. This is unfortunately less than the critical amount of space for most monitoring programs, so admins don’t get a warning about it. (hooray).
To prevent running out of space on C:, I recommend a 1-2x monthly deletion of logs on the C: drive. You can also configure your server to reduce Exchange logging significantly.
Just adding a server to a DAG will not sync its databases. That is a second manual step. If you don’t synchronize all databases across your DAG, you don’t need to size for them.
Describing resource requirements is tricky because you don’t have to copy all databases across a DAG. Some databases can be held on a single host. If a database isn’t shared, you don’t need to worry about the other servers hosting it.
Configure DNS, Autodiscover for Exchange 2016
Setting up Autodiscover correctly is probably the trickiest part of an Exchange 2016 migration. This is not specific to DAG.
If you want users to be able to reach your Exchange servers from external, you will need to open firewall ports on 443 to at least one of your Exchange servers. For failover purposes, I recommend opening at least two of your Exchange servers to port 443.
Then add a round-robin DNS records for each Exchange server. Or at least two of them. Example:
Firewall allow 443 67.50.50.4
Firewall allow 443 67.50.50.5
DNS A 67.50.50.4 MBX1.contoso.com
DNS A 67.50.50.5 MBX2.contoso.com
DNS CNAME MBX1.contoso.com EMAIL.contoso.com
DNS CNAME MBX2.contoso.com EMAIL.contoso.com
Don’t forget to modify the web URLs in EAC to point to your round-robin DNS.
Don’t forget to use the Microsoft Remote Connectivity Analyzer toolto verify your DNS, firewall, autodiscover, and and Web URL configs. This really is an Exchange admin’s best friend.
How hard is it to set up a Witness server?
Not hard. Pretty much any Windows server can do it (Server 2008+).
Most administrators pick an existing file server that is already performing the file sharing role. Pick a server that won’t be rebuilt anytime soon.
Before you set up the DAG, make sure your witness server will allow management from Exchange.
Ensure Windows Firewall on the witness server allows Windows Management Instrumentation (WMI). Normally if file sharing works, WMI is allowed. I wouldn’t worry about this until you get an error.
“Exchange Trusted Subsystem” is a Local Administrator on the witness server. You will need to do this. Just go to Computer Management > Local Users and Groups > Groups. Edit Administrators and add Exchange Trusted Subsystem from your domain.
When you are creating the Database Availability Group using the Exchange Admin Center, the first step of the wizard asks for the DAG Name (pick any name), the witness server ( FILESERVER1.company.com ), the Witness directory ( c:\DAGshare ), and the DAG IP addresses (leave blank for Exchange 2016).
Once the DAG creates successfully, then you can Manage Database Availability Group Membership and add your exchange servers to it. This will not affect clients and does not migrate any mailbox databases yet.
How do I remove my witness server from an existing DAG?
If you have to rebuild or decommission your witness server, no worries.
Common sense: Don’t change your witness server when it is being actively used for quorum. For example, if you have a DAG Exchange server offline, don’t change your witness server until it is working again.
Make sure that the new witness server has firewall rules and permissions set properly.
In EAC (Exchange Admin Center), go to Servers > Database Availability Groups. Manage your DAG and change the witness server to a new host. When you save, Exchange should create the new file share and migrate everything over.
Once the DAG is created, sync the mailbox database
Note: Once you add a database copy to another DAG partner, it is in production!
What I mean is that the copy could activate automatically on the new Exchange server. If it activates (because the original server reboots, has network latency, etc), then all your clients are going to automatically fail over to the new server. If the new server doesn’t work, they will have a bad time.
How do I test my DAG servers without impacting clients?
The way I test a new DAG server is to create a new (empty) mailbox database called TEST. I create copies of TEST across all DAG members, and migrate my test account to that mailbox database.
Now I can activate, suspend, failover, etc the TEST database without impacting my regular users.
This is important for testing functionality across multiple servers and sites. For example, clients at SITE-A might not know how to route to SITE-B. It is good to find that out with a test account.
When you are sure that all your clients will communicate correctly with each of the DAG servers, then add the copies of your production databases.
Adding database copies to other servers
This will start a wizard which lets you select the server to add a copy to. Depending on your active directory synchronization speed, it may error at this point. Check the troubleshooting article below. Generally, I complete the wizard (even with an error), give it about 20 minutes to sync with active directory, then re-seed the database copy manually.
If your database health is not good on the the Passive copies, you will see a link to fix them. For example, “Update” will attempt to re-seed a Failed and Suspended database copy.
Note: Make sure you get your SOURCE server correct for these commands. Source = the server that has the active/mounted database copy.
Before you reboot a DAG server – failover and health checks
Even if you are in a maintenance window, I recommend failing over the databases any time you reboot a DAG member.
If you don’t do a manual failover, you will often see sync and index issues after the server is back up.
What happens with clients? Well, assuming your network is good and you’ve tested the client experience on each server already, they shouldn’t even notice that the database failed over. Newer versions of Outlook (Desktop and Phone) will automatically re-point to the active copy.
Exchange Admin Center EAC, how to click Activate to fail over the active database.
Symptom: Your ONTAP 9.x system is giving errors about third party CA certificates expiring.
Source = mgwd
“mgmtgwd.certificate.expiring: A digital certificate with Fully Qualified Domain Name (FQDN) Class2PrimaryCA, Serial Number 85BD4BF3D8DAE369F694D75FC3A54423, Certificate Authority ‘Class 2 Primary CA’ and type server-ca for Vserver Netapp1 will expire in the next 4 day(s).”
“mgmtgwd.certificate.expiring: A digital certificate with Fully Qualified Domain Name (FQDN) DeutscheTelekomRootCA2, Serial Number 26, Certificate Authority ‘Deutsche Telekom Root CA 2’ and type server-ca for Vserver Netapp1 will expire in the next 7 day(s)”
“mgmtgwd.certificate.expiring: A digital certificate with Fully Qualified Domain Name (FQDN) UTN-USERFirst-Hardware, Serial Number 44BE0C8B500024B411D3362AFE650AFD, Certificate Authority ‘UTN-USERFirst-Hardware’ and type server-ca for Vserver Netapp1 will expire in the next 7 day(s).”
Symptom: The certificates are starting to expire in July 2019.
Symptom: The expiring certificates are type “server-ca”
Symptom: When you connect to your Netapp command prompt and type “security certificate show”, you see about 100 certificates, not 2-6.
Symptom: Can’t upgrade from ONTAP 9.1 to later versions
Symptom: You can’t create a new cluster on a brand new Netapp.
“Vserver Management .Error: Failed to add the Cserver record in RDB . The certificate has expired.”
Different issue: If you see expiration warnings about certificates that are type = server and are named the same as your Netapp, see this article instead.
Disclaimer: I’m just someone who admins Netapp SAN for my job. You should go to the Netapp website and contact their support for official guidance. This bug has the potential to cause a major outage. I’ve included links to relevant bug reports below.
Root cause of expiring server-ca certificates:
This is a bug in Netapp ONTAP 9.2 – 9.3 and later versions. The bug causing failures to upgrade or install new Netapp clusters is: BUG 1250500. (You will need a Netapp account to view it)
The general bug that addresses certificates expiring but you aren’t trying to upgrade or install a new Netapp is BUG 1245418 (You will need a Netapp account to view it).
The simple version is that Netapp included a bunch of third party certificates in their ONTAP 9.2+ releases for compatibility purposes. This brought the certificate count up from 2-6 to 100+. Each of those certificates has a different expiration date, with a few expiring in July 2019, several expiring in 2020, etc.
If you aren’t trying to upgrade your system or install a new Netapp, there is normally no impact from this bug. You just get a bunch of scary errors in the event log that repeat daily.
IF YOUR NETAPP IS AFFECTED, WAIT UNTIL THE NEW PATCH IS OUT (ETA JULY 15, 2019). I have been told that Netapp is working on an ONTAP update which includes a fix for the certificates and BUG 1250500. I would monitor this Netapp KB article for release timelines and for the exact versions of patches that have the fix.
How to fix the expiring server-ca certificates in Netapp
Updated July 11, 2019: Netapp is releasing ONTAP 9.3 , 9.5, and 9.6 updates which fix the expiring certificates.
Netapp Support does not want customers to try to fix this problem on their own.
Put in a case with Netapp for a custom workaround or wait for the patch, due around July 15, 2019.
How to show the full list of certificates in Netapp:
These steps can be used to show the list of certificates installed on your Netapp.
Looking at your event log, make a list of which certificates are expiring or expired. If you look at my examples above, you will see that I bolded the FQDN or “Common name” in the error messages. This is the information you need.
Type security certificate show -common-name Class2PrimaryCA Class2PrimaryCA is an example certificate common name. Put the common name of an expiring certificate here.
You should see information about the expiring certificate display.
Selfish plug time!
Thanks for reading this article! I hope it helps you! If you have tips or feedback, please comment or send me an email so that others can benefit.
I am a consultant in the Maryland/DC area in the USA. My specialties are Windows migrations (to 2016 and to Office 365 / Azure), VMware migrations, Netapp and SAN, and high availability / disaster recovery planning. If you would like help with your complex project, or would like a architectural review to improve your availability, please reach out! More information and contact can be found on the About page. – Amira Armond
Symptom: Your vCenter appliance was deployed using “Tiny” resource sizing of 10 GB RAM and 2 vCPU.
Symptom: Your vCenter is version 6.7
Symptom: When you open your vCenter Appliance Management website at :5480, you see an alert in the Memory area.
Symptom: You may see occasional alarms stating “Memory Exhaustion on vCenter”
Symptom: You may see this in your event log: “event.vsphere.online.health.alarm.event.fullFormat (vsphere.online.health.alarm.event)”
Symptom: You see constant alarms in vSphere Web Client stating “vSphere Health detected new issues in your environment” referencing your vCenter server.
Root cause for memory error with Tiny deployment:
The memory issue may be a bug specific to the 6.7.0 Update 2a release of vCenter 6.7 (6.7.0-13643870) April 2019, which is the first place I’ve seen it.
In general, the memory warning can be ignored as long as your vCenter is working properly. However, it is very annoying.
Root cause for constant vSphere health alarm (not memory)
VMware added new functionality in vCenter 6.7 update 2 to perform preventative health checks using the Customer Experience Improvement Program.
Almost everyone who upgrades to 6.7 will get constant warnings and alerts saying “vSphere health detected new issues in your environment” because of these new health checks.
Honestly, I recommend just ignoring this alert. I think the ultimate fix will be a patch from VMware.
If you assign 12 GB of RAM to your vCenter, the alert should go away.
Once you make the change, your vCenter appliance dashboard will go green for memory.
The steps to assign more RAM are as follows.
Easy mode: If you have purchased the right VMware licenses, you can simply edit settings on your vCenter appliance VM and assign it more RAM. If you don’t know if you have the licenses, try it anyways. Worst that happens is you will see an error in the task area.
How to assign more RAM to vCenter appliance if you don’t have the hot-pluggable virtual hardware license.
Connect to the vSphere website for the host that is running vCenter
Connect to the management website for your vCenter appliance ( https://vcenter:5480)
In the management website for vCenter, click Actions > Shutdown.
Monitor the vSphere website on your host until you see vCenter powered off.
Right-click vCenter and select Edit Settings.
Modify the Memory to 12288 MB (or 12 GB) and click OK.
Thanks for reading this article! I hope it helps you! If you have tips or feedback, please comment or send me an email so that others can benefit.
I am a consultant in the Maryland/DC area in the USA. My specialties are Windows migrations (to 2016 and to Office 365 / Azure), VMware migrations, Netapp and SAN, and high availability / disaster recovery planning. If you would like help with your complex project, training, or would like a architectural review to improve your availability, please reach out! More information and contact can be found on the About page. – Amira Armond
These screenshots show the entire process to upgrade vCenter from 6.0 to 6.7 using a Windows 10 desktop as my administrative workstation. The same steps work for vCenter 6.5 to 6.7.
Is your existing vCenter server running on Windows? This article is primarily about upgrading vCenter appliances (linux-based). If you have a Windows vCenter server, check the comments for instructions from Greg Curry. Thanks Greg!
Preparing for the vCenter appliance upgrade
First, create a snapshot for your existing vCenter server. If you have disk space available, make sure to Quiesce memory.Log on to vmware.com and navigate to the Downloads section. Pick vCenter Server from the list. (It does not matter if you choose Essentials, Standard, etc, they are all the same).Verify the version you want to use (it will default to the latest one), and click Download for VMware vCenter Server Appliance, file type = iso. Now wait for a bit for it to download.Navigate to your downloads folder when the download completes. In Windows 10, you can double-click the file to mount it to a virtual DVD drive. For non Windows workstations, use an .iso viewer or burn the image to a CD then open it on the CD.Navigate to the contents of your .iso file. On Windows 10, use Windows Explorer and look for the files inside your DVD drive. Then expand vcsa-ui-installer to open that folder.Expand the folder that matches your computer (Windows 7, 8, and 10, pick win32)(Windows 7, 8, 10) Run the installer.exe file to start the upgrade wizard.
Appliance Upgrade Stage 1
Options display. For this scenario, pick Upgrade (to upgrade an existing vCenter appliance from 6.0 or 6.5 to 6.7)The VCSA introduction screen displays.Agreement displays. You need to accept it to proceed.Enter the DNS name for your existing vCenter server. You can look this up in the address bar when you open vCenter to manage your VMware environment. Port 443 is appropriate for almost everyone, unless the IT department has customized this. When you Connect to Source, the rest of the window will display.Most environments will use administrator@vsphere.local for the SSO User name. SSO password is the password for the user name you entered above (administrator@vsphere.local). Appliance (OS) root password means the root password for your vCenter appliance website (https://vcenter01.company.com:5480). See next step to identify which ESXi host is managing your appliance.
——————
Additional info from Todd who provided this quote in the comments (Thanks Todd!)
I ran into one minor issue during the “Upgrade Stage 1: Deploy Appliance” phase. Operation would halt with “Failed to authenticate with the guest operating system using the supplied credentials.” I know they’re good as I can login to VCSA, VAMI, MOB, ESXi using them.
I changed all of the passwords to numbers and letters only with a single known good special character tacked on at the end. Et Voila! Problem solved. The rest of the upgrade proceeds without errors.
Passwords like these will not work with this utility:
Tv5$8FG#m*Djn 7xjg8sP%C#usXH8H X8xk$vR^q9ccw@WxR
———————–
In the VMware hosts & clusters view, if you select your existing vCenter server and go to the Summary tab, you can see the address of the ESXi host.You will get prompted to review the certificate. Just click Yes.Now enter the ESXi server that will host your upgraded vCenter server. Yes, we are making a new vCenter server with this procedure. The old vCenter will be powered off if the upgrade succeeds. Make sure you pick a host that has plenty of RAM and disk (at least 12 GB RAM, at least 500 GB disk) available.This is where you set the name and password for your new vCenter. The name will be displayed in VMware Hosts & Clusters. This is not the DNS name or IP address for your new vCenter. Your new vCenter will inherit the existing DNS name and IP address since we are doing an upgrade.Almost all medium and small businesses use Tiny for deployment size and Large for storage size. If you have 10 or less ESXi servers, use Tiny.Pick the datastore that will host your new vCenter server. Remember that by default it will use 850 GB thick provisioned for a Tiny deployment.
Optional: Check the “Enable Thin Disk Mode” so that your vCenter server uses less disk space (about 80 GB). This is mildly dangerous because the usage can grow over time and cause a datastore to fill up (crashing all VMs on the datastore), but for most small environments it doesn’t grow more than a few GB per year and 850 GB is WAY more than you need.
These are the network settings that your new vCenter server will use. Make sure to pick a Network that both your existing vCenter and the new vCenter can communicate on, with their IP addresses. The new vCenter will use the Temporary IP address for a few minutes during the upgrade process, then it will switch to the existing IP address of your vCenter server. Make sure no other devices are using the Temporary IP address.Ready to complete displays. When you click Finish, the deployment will start.Even across WAN links, this process takes less than an hour. Make sure that your workstation doesn’t disconnect from the work network during this process. (If it does disconnect, no problem, just re-start from the beginning.)
Errors that can occur during stage 1
If you used the wrong password for the old vCenter, the source host, or the destination host, you may get an error at this point.
If you are on an unstable VPN link, you may get an error. Try running the upgrade from a workstation or server on the LAN.
If you used DNS instead of IP addresses, try changing to IP addresses.
Temporary IP error: The upgrade will attempt to ping the IP address you chose for the temporary vCenter IP address. If it pings, the upgrade will fail. Make sure nothing is using that IP.
For information about cleaning up failed upgrade steps, see troubleshooting at the bottom of the article.
Stage 2 Upgrade VCSA
Phase 1 shows complete. The new vCenter virtual machine is deployed, but not configured. Beginning Phase 2 of the upgrade.Introduction for Stage 2 displays. Click Next.It is normal to get several warnings. Errors are bad, and will cause your upgrade to fail. Warnings are not bad. Read them and take action if appropriate.Most small and medium businesses don’t have that much data in their vCenter. In this example, there is only 3.18 GB of data to transfer. Since there is not a big difference between the sizes of each choice, I choose to move over everything. It is good to retain historical data when possible.Most businesses are fine to join the CEIP. If you are working on a secure network, then don’t check it.Ready to complete displays. You will need to check that you have backed up the source vCenter server (you did, right?). Ways to perform backups: 1) Veeam. 2) Create that snapshot at the beginning of this post. 3) Use the appliance website (https://vcenter.company.com:5480) and click the Backup button (you will need a web server or SCP host to do this). The reality is that this upgrade process doesn’t delete your original server. At the worst, it powers it off. There is very little risk of harming your original vCenter.Phase 2 proceeds. It should finish within about 30 minutes.
Possible errors during phase 2
NTP error: Make sure your source VCSA has good NTP settings which are the same as the host you are using for source and destination. See this VMware article about configuring NTP. https://kb.vmware.com/s/article/57146 If you don’t have an internal time server, then the recommended option is to point to internet NTP servers: 0.vmware.pool.ntp.org ; 1.vmware.pool.ntp.org ; 2.vmware.pool.ntp.org
Success!!
At this point, your upgrade should be successful. When the upgrade wizard completes, your old vCenter will be powered off and your new vCenter (running 6.7) will be powered on.
You may want to rename your old vCenter to something like “old_vCenter01”.
Test your ability to log onto the vCenter appliance (https://vcenter01.company.com:5480)
Test your ability to manage your virtual machines (https://vcenter01.company.com/vsphere-client)
Make sure to check your backup jobs. They should switch to using the new vCenter automatically, but double-check.
Once you are comfortable with the new vCenter and you have at least one good backup of it, you can delete your old one.
Remember that if the new vCenter doesn’t work, you can revert your changes simply by powering it off and powering the new vCenter on. You may need to log into individual hosts (https://host_ip_address) to do this if your vCenter isn’t working.
Troubleshooting:
Manually verify every password for each device (your existing vCenter administrator@vsphere.local, your existing vCenter root, the source ESXi host root, the destination ESXi host root). It is common for the vCenter root to be expired. See my other blog for easy steps to fix an expired vCenter root password.
Orphaned vCenter attempts: If your deployment fails, the new vCenter may display as orphaned. There is no impact at this time, because you will continue using your existing vCenter. I’ve heard that using VMware Workstation to connect to your vCenter will allow removal. There is a blog that describes various methods for dealing with orphaned VMs: https://www.altaro.com/vmware/how-to-deal-with-orphaned-virtual-machines/ For a more authoritative source, see VMware KB article for removal steps: https://kb.vmware.com/s/article/1003742#vmdeleted
Thanks for reading this article! I hope it helps you! If you have tips or feedback, please comment or send me an email so that others can benefit.
I am a consultant in the Maryland/DC area in the USA. My specialties are Windows migrations (to 2016 and to Office 365 / Azure), VMware migrations, Netapp and SAN, and high availability / disaster recovery planning. If you would like help with your complex project, or would like a architectural review to improve your availability, please reach out! More information and contact can be found on the About page. – Amira Armond
Ms. Katie Arrington (Special Assistant to the Assistant Secretary of Defense for Acquisition for Cyber) gave a presentation to small DoD contractors on May 23, 2019 to announce a new program which will require cyber-security audits and certification for all DoD contractors.
The proposed program is called the Cybersecurity Capability Model Certification (CMMC).
Here are my notes from the presentation.
Background
Per Federal Acquisition Regulations (FAR), federal contractors must comply with a list of cyber-security best practices. Currently, contractors are required to self-certify that their computer systems follow cyber-security best practices if they deal withh Controlled Unclassified Information (CUI).
What is CUI?
Controlled Unclassified Information is data that needs to be protected against release, but has not been classified by the United States Government. For example, military vehicle design schematics could be considered Controlled Unclassified Information. This information may be shared with federal contractors in order to manufacture replacement parts, or for reference while performing maintenance contracts. CUI also includes well-known sensitive categories such as personally identifiable information and health records for service members.
The proposed CMMC will now require every DoD contractor to have their computer systems certified in order to bid on RFPs. The contractor does not need to be handling CUI.
The problem identified by Ms. Katie Arrington is that the self-certification requirement for cybersecurity is not working. Theft of sensitive information from federal contractor systems has increased over time, not decreased. As a solution, her office (DoD Cyber, Acquisitions) is leading an initiative called the Cybersecurity Maturity Model Certification (CMMC).
The CMMC initiative will require DoD contractor information systems to be certified compliant by an outside auditor. This solves an issue where some businesses self-certify compliance without fully implementing (or understanding) needed security controls.
CMMC predecessor – NIST SP 800-171
In 2016, the National Institute of Standards and Technology (NIST) released a document named “Special Publication 800-171”. This lists 110 cyber-security best practices that contractors with access to CUI must comply with. An example of an individual best practice is requiring strong passwords for any account that can access CUI.
While compliance with NIST 800-171 is a requirement in order to win new contracts, there is no auditing or certification program in place yet. Government contractors are allowed to self-certify that they meet the security requirements.
Highlights of the CMMC
A single standard used across all DoD contracts starting in 2020-2021
Considered a “go/no-go” requirement
Based on the NIST 800-171 controls
Identifies five levels of data security so that contractors can implement reasonable security for the data they deal with. Encourages government contract officers to pick an appropriate tier (not everything requires level 5)
Provide automated tool which automatically gathers data to simplify reporting efforts
Required CMMC level will be contained in RFP sections L & M
Authorizes a non-profit organization to oversee the program and accredit private-sector auditors
Makes cybersecurity an “allowable cost” in DoD contracts
DoD contractors will need to be certified at a specific security level before they qualify to bid on contracts.
Timeline for CMMC enforcement
Ms. Katie Arrington described the following timeline for CMMC:
Mid 2019 – Working groups and creation of automated assessment tools.
Early 2020 – Begin developing oversight and certifier accreditation program, processes.
Mid 2020 – Test the certification program and revise it.
Mid/late 2020 – Accredit third-party certifiers.
Future – Begin adding CMMC requirement to all new DoD RFPs
Updated: How to become a CMMC auditor
Check this article on cmmcaudit.org which gives the latest info on the CMMC Accreditation Body and what is still needed to accredit auditors and certifiers.
Next are my thoughts about what DoD contractors should be doing to prepare. These steps are my recommendations as a cyber-security consultant and are not from Kate Arrington’s presentation.
There are three factors for estimating the cost and work involved with compliance.
How complex is the network you are evaluating?
Does your network already have secure configurations and security programs installed?
What CMMC level are you trying to meet?
This Reddit thread has a frank discussion of the effort involved with a NIST 800-171 or 800-53 compliance project. The original poster thought that 60 hours to do a “gap analysis” (no fixes, just finding out what is wrong) was insane. Almost all responses agreed that that estimate was low. Many people gave estimates for the full effort involved (which includes fixes) – this was between 1,000 and 2,000 hours by a knowledgeable consultant.
CMMC level 1 and 2 should take less effort than the above estimates, but it will still be significant.
DoD Contractors – How to Prepare for the CMMC
My recommendations below are going to be a mixture of NIST SP 800-171 and the newer CMMC draft requirements. Remember that if your company deals with CUI, you should already be compliant with 800-171…start there. If not, then skip straight to the draft CMMC requirements and begin reviewing your network.
1. Ask these questions first
What security level applies to my company’s network? If you manage Controlled Unclassified Information (CUI) in any way, you have to meet at least security level 3.
Not all government contractors deal with CUI. If you aren’t sure, ask your contracting officer or read the RFP. Examples of CUI are personally identifiable information, schematics of military equipment, sensitive information about schedules and personnel, and configuration documentation for government networks.
It looks like most subcontractors won’t need the same security level as primes. But the latest news is that every DoD contractor will need to be at least CMMC level 1 in order to bid on RFPs.
As time progresses toward 2021, RFPs will specify the CMMC level requirement for bid. For existing contracts, you may be able to ask the contract officer to identify which CMMC level will apply to the renewal.
Is it possible to isolate your information to fewer systems, fewer networks, or fewer users, while still fulfilling the terms of your contract? You don’t need to secure ALL computer systems for the entire company. You just need to secure the systems that store data about the contract. Make the job easier by reducing your footprint.
Companies dealing with CUI
Is the CUI stored or accessed on your contractor information systems right now? If you have CUI on your systems now, you need to protect it according to NIST SP 800-171 requirements. It will take time before CMMC comes into effect. 800-171 is in effect now.
If you are a subcontractor – ask your prime to identify what information is CUI, and whether you NEED to store or access it using your information systems (as opposed to government systems or your prime’s systems). If you can avoid storing CUI on your own systems, do so! It reduces risk to your company (and probably will protect the data better) if you can store it elsewhere.
If you are a prime – ask your contract officer to identify what information is CUI, and whether you NEED to store or access it using your information systems (as opposed to government systems). Government systems have to adhere to very stringent requirements for cyber security. This is the best place to store CUI if you have an option.
2. Perform a Risk Assessment
Work with a cybersecurity professional who currently specializes in NIST 800-171 and have them perform a risk assessment. This assessment will review your progress toward compliance with the NIST 800-171 controls and list the ones that are deficient. Some form of vulnerability scanning and penetration testing will normally be included, with a report of findings.
For example, a reseller company with no CUI and very little proprietary data will probably need to attain CMMC level 1. If you review the draft requirements, they aren’t very hard to implement, but you still need to create policies, write plans, and gather evidence for your audit.
Using the NIST SP 800-171 document templates is still a valid move. If you prepare for this standard, you should have about 95% of the work done toward CMMC levels 1-3.
3. Write a Systems Security Plan
NIST provides a template for this plan here. You should describe how your information systems are secured and what policies are in place that relate to cybersecurity. This plan should give a POA&M (Plan of Action & Milestones) to resolve each deficient control. Note that you do not need to be 100% compliant with all security controls. You do need to have the most critical 17 addressed (as defined in DFARS 252.204-7012), a plan to fix the rest or explain why they don’t apply, and show progress over time.
4. Prepare for Incident Management
Make sure that you have a high quality Incident Management plan and practice it regularly. Besides implementing security controls, you are also expected to report security incidents to the DoD within 72 hours.
Make sure to register with the DoD reporting website ahead of time. The DoD will want to issue you a certificate to verify your identity, which can take a few days. The reporting website is https://dibnet.dod.mil
5. Follow Up and Continual Improvements
Ensure that your policies are realistic. Many organizations write policies that state that they will keep all systems fully patched at all times. If the organization then fails to patch systems for two months, and has an incident as a result, their failure to follow their own policy will count doubly against them. To the converse, if you have a really good reason to only patch every three months, and have written that into your policy, it might protect you against liability if there is an incident at the two month mark.
Your vCenter 6.5 or 6.7 server automatically expired your root password
You left the default setting to expire the root password in your vCenter Appliance
When you try to log on to the management website :5480 with root, it says the password is expired.
You know what the root password is.
How to fix:
Open console to your VCSA by logging on to the vSphere ESXi server that is hosting it
Press F2 to configure your vCenter appliance
When prompted, type in your root password (it should work, even if expired)
Note: Even though there is an option here to change your root password, it doesn’t seem to work when root is expired.
Go to troubleshooting options and enable BASH and enable SSH
Connect to your VCSA using SSH – logging on with root and your expired password will work.
Select BASH shell from the menu
type “passwd” and press Enter
If you want to keep your current root password, just enter it twice here.
If you want to use a new root password, enter it twice here.
Verify it works by opening the management website :5480 to your vCenter server and logging on.
Prevent the root password from expiring again
To prevent the root password from expiring again, log onto vCenter management website ( :5480), go to Administration menu, and change the password settings here.
If you use a long, complex password, there is no reason to automatically expire it. Long and complex means 14+ characters!
Selfish plug time (sorry!)
Thanks for reading this article! I hope it helps you! If you have tips or feedback, please comment or send me an email so that others can benefit.
I am a consultant in the Maryland/DC area in the USA. My specialties are Windows migrations (to 2016 and to Office 365 / Azure), VMware migrations, Netapp and SAN, and high availability / disaster recovery planning. If you would like help with your complex project, or would like a architectural review to improve your availability, please reach out! More information and contact can be found on the About page. – Amira Armond
