Happy New Year everyone! This is just a quick post to announce the release of my second Pluralsight course, VMware vSphere Security, which was just published today. If you have a current subscription (or want to sign up for a new one), please check it out:
Hope the course is helpful and any comments/feedback/suggestions are welcomed.
What was supposed to be a afternoon of host memory upgrades, cluster re-balancing and DRS changes, as well as an upgrade from vSphere 5.1 to 5.1U1, turned into quite the troubleshooting exercise. A few people asked me on Twitter to document the experience so hopefully this post saves a few of you some time if this issue comes up. Here we go!
Before I started, the environment looked something like this:
- vCenter Server 5.1 installed on a Windows Server 2008 R2 standard VM. This same machine also had vSphere SSO, vSphere Web Client, vCenter Inventory Service, and vCenter Update Manager on it. All running 5.1 unpatched.
- vCenter Server database is stored on an external SQL 2008 server. vCenter SSO database is stored locally on the virtual machine within a SQL 2008 Express instance.
- 9 ESXi 5.1 hosts all in a single cluster with HA enabled and DRS set to manual
- Active Directory authentication was enabled on both the ESXi hosts and vCenter Server.
- AD/DNS/DHCP are hosted on separate physical servers
Given that I needed to power off the hosts to upgrade the memory and they all needed a reboot to patch to ESXi 5.1U1, this meant downtime. Additionally, the current cluster did not have EVC enabled and there are a mix of different processor families here, so we scheduled downtime for the entire environment to shutdown each VM and move the hosts into two, EVC-enabled clusters.
Now that we have some background, the first thing on my list was to upgrade vCenter Server to 5.1u1. Now if you’re not familiar with the Windows installer, when you run the autorun program that comes with the vCenter Server iso, you install the components in the following order:
- vCenter Single Sign On
- vCenter Inventory Service
- vCenter Server
- vSphere Client
- vSphere Web Client
- vCenter Update Manager
There is a simple install option which automates a lot of this, but that is not available for upgrades. It only works if you are doing a new install. So I proceeded with the install order and completed the SSO, Inventory Service, and vCenter Server pieces. Everything installed just fine. Next I updated the vSphere Client and after that installed I attempted to login to vCenter Server using active directory credentials. This is where things went downhill…
Active Directory authentication did not work. I verified AD was actually working properly so this was not the issue. Not being able to login with AD, I tried the default administrator@System-Domain account which let me in. I also updated the vSphere Web Client, in hopes that it would let me in, but the installer wouldn’t let me past the part where you connect it to vCenter SSO. Even though I was typing the correct lookup service URL and username/password, it would come back with “password incorrect or blank”. Now I’m not going to list out all my troubleshooting steps here as it was lengthy, but suffice it to say that I had a corrupt SSO installation/database. Database repairs failed, so my only option was to re-install SSO. And this is where the fun begins!
So after uninstalling vCenter SSO and attempting to reinstall, it came back with an error saying unable to re-create database users. Now this gave me a clue that uninstalling SSO doesn’t actually wipe out the current database configuration. So what you’ll want to do here is use the SQL Server 2008 Management Studio which should be installed on the VM to browse to the local database instance. The default name of the instance is VIM_SQLXP, so the full server name looks like: localhost\VIM_SQLXP or .\VIM_SQLXP
The next thing you need to do is delete both the database used for SSO and the database users. In my case I backed up the database before deleting if, for some reason, I needed something inside of it. The default database name is RSA. After deleting that, I deleted the two DB users: RSA_User and RSA_DBA. Once that was completed, vCenter SSO installed properly.
Now after re-installing SSO, you are left with an environment that is no longer linked to vCenter SSO. In my case, this meant that the vCenter Inventory Service, vCenter Server, and vSphere Web Client all needed to be repointed.
You will find the following VMware KB very helpful if you ever run into this issue: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2033620
The first step in my case was to repoint vCenter Server to the new SSO instance. You do this by performing the following steps (I’m assuming in all these steps that vCenter Server is installed in the default location and ports):
- Use Windows explorer and navigate to: C:\Program Files\VMware\Infrastructure\VirtualCenter Server\ssoregtool
- Locate the sso_svccfg.zip file and extract it to a folder here
- Open a command prompt and CD to that folder you just unzipped the files to
- Run the following command, updating your vCenter SSO URL, and user/pass as appropriate:
repoint.cmd configure-vc –lookup-server https://vc5.corp.com:7444/lookupservice/sdk –user “admin@System-Domain” –password “SSO_pw1!” –openssl-path “C:\Program Files\VMware\Infrastructure\Inventory Service\bin/”
If you try to start vCenter Server at this point, it will try to, but fail. You need to re-populate the certificate file names within the VPXD.conf after re-pointing to the new SSO instance. The following VMware KB describes this and I’ve also included the steps below: http://kb.vmware.com/kb/2048753
- Locate the vpxd.conf file which is located in:
- Create a copy of this file in case anything goes wrong. Now open this file in Notepad
- Search for “null” and you’ll see two fields that look like this:
- On both of these fields, change the null values to match below
- Save the file and close it
Now restart both the vCenter Server and vCenter Management Webservices services. Your vCenter Server should now be linked to the new SSO instance and should start up properly.
Next we need to re-link the vSphere Web Client to the new SSO instance. To do that, follow this procedure:
- CD to C:\Program Files\VMware\Infrastructure\vSphereWebClient\scripts
- Run the following command, replacing your vCenter Server name, admin username, and password:
client-repoint.bat https://vc5.corp.com:7444/lookupservice/sdk “admin@System-Domain” “SSO_pw1!”
Restart the vSphere Web Client service and you should be able to login to v Center Server with the user admin@System-Domain and the password you specified during installation. The default URL for the vSphere Web Client is: https://vc5.corp.com:9443/vsphere-client/
At this point, you should be able to login, but you should now see a message about vCenter being unable to connect to the inventory service. We’ll tackle that next…
To fix the Inventory service and re-link it to SSO, we need to perform a similar process:
- Open a command prompt and CD to C:\Program Files\VMware\Infrastructure\VirtualCenter Server\isregtool
- Run the following command, replacing your vCenter Server name in:
Now you can restart the Inventory Service and it should be re-linked to SSO. You’ll need to restart vCenter Server as well to pickup this change. We’re almost there!
After vCenter Server restarts, login to the vSphere Web Client using the admin@System-Domain credentials again. Although we have fixed all the links to SSO, in my case the Active Directory groups and permissions had been blown away, however the AD identity source was still there. So to add those back, use the following steps:
- On the left-hand panel, click on vCenter Home
- Click on vCenter Servers until Inventory List. Then click on your vCenter server name
- Click on manage along the top row and then choose the Permissions subsection
- Click on the + to add a new permission
- In my case, I was granting the AD group Domain Administrators the Administrator role in vC. So if you click the add button on the left hand pane, it will let you select your domain and then you can search for and add the group.
- Choose whichever role you would like to assign and make sure to do propagate to children
Once that is complete, log back out of the web client (or the vSphere client) and you should be able to login using Active Directory credentials! After that, in my case, I updated vCenter Update Manager and was able to proceed with my host updates.
So that was a bit of a long post, but I wanted to outline what happened and all of the steps I had to go through. Hopefully this helps out one of you if you ever hit this upgrade bug! This whole afternoon made me appreciate the SSO implementation in vSphere 5.5, which was completely re-written, as it is much easier to install and administer!
This post is a bit of a resurfacing as I have been disconnected from the community lately due to the craziness that has been work and my personal life for the past 6 months. I’m back now though and looking forward to reconnecting with everyone, the community, and the technology!
It’s hard to believe that I’ve been at RSA almost 2.5 years now in my current role as a technology consultant. I have learned a lot, worked with some great people, and most importantly, spoken with hundreds of EMC/RSA customers. I’ve also been on the leading edge of security technologies both for authentication and advanced analytics and threat detection/prevention. An exciting space to be sure! A big thanks goes out to all those on my direct team and countless others that have helped me throughout this journey.
All that being said, a recent opportunity has come along that I could not pass up. I am very excited to announce that starting October 1st, I will be moving over to a new role within EMC as a vSpecialist for the Northeast region. I am extremely excited to get back to my virtualization and storage roots, focusing on everything EMC and VMware. I am also honored and proud to be joining a team of this caliber. Between the technology, the customers, the partners, and the internal teams that I will be involved with, I can’t wait to get started. I’m also looking forward to participating in many more local VMUG’s and other events and hope to see everyone there!
As part of this new role, I will also be resuming my journey down the VCDX road and aiming for a defense at PEX 2014. So stay tuned for updates, blog posts, and more information around that as things progress.
More to come, but just wanted to share the news with everyone right now!
This post is a little bit overdue, but I wanted to report on my experience with the VCAP5-DCA exam that I took at VMworld US. The exam, much like its DCD counterpart, is 210 minutes. This one has 26 lab questions that you have to complete and each question can have multiple parts. There is partial credit so read each question (and its parts) thoroughly! A passing score is 300.
For studying, the first thing you’ll want to do is download the official blueprint from VMware:
This was a tough exam just in the sheer amount of tasks you are asked to do in a short amount of time. It basically compresses what you know as a VMware administrator and troubleshooter into 3.5 hours. Repetition of tasks on the blueprint and knowing where to go really helps here! If you don’t have the experience from your job, make sure you lab a lot here. You need to be quick! I ran out of time on the exam as I was running a command. I’m not sure it actually finished when the timer hit zero, but I went up until the last second here.
This exam definitely would have benefited from a dual monitor setup or even alt-tab! You end up switching between the exam questions and then an actual live terminal which is a jump box of sorts. This has all of your vSphere clients, remote desktop connections, SSH, and other connectivity tools. Additionally, the vSphere documentation set is on this VM. Familiarize yourself with the tools on this VM when you start and I would also recommend opening connections to all your important sources. This will save you time later on!
I’ve read mixed thoughts on looking at the documentation during the exam. I’m taking middle ground on this. If you know what to do or how to do something, but need to reference the syntax of a command, open the PDF! This happened to me twice and I actually planned on this before. There are some longer commands that I knew sometimes I don’t remember the exact argument order on, but I knew where to look! You don’t want to spend much time in the docs, but if it’s just for a quick reference and you know where to go, it can be very beneficial. However, if you don’t know how to do an objective, you will waste a lot of time going through the PDF’s trying to find a more general solution. Don’t fall into this trap!
One thing I did notice is that resources on the jumpbox VM were a little slow. I had to close down a few of the PDF’s and I had some duplicate RDP windows which I also closed, this helped to speed up performance. Thankfully I didn’t have any interface crashes this time like on DCD, but a few times switching between questions/lab was a bit slow and it made me a little nervous!
Overall I enjoyed the exam and thought it was one of the more fun formats for cert tests that I’ve taken. It’s definitely a mad rush to the finish, but I really liked how well the live lab format worked and the way you can structure the tasks around a scenario (building out a new datacenter, deploying a new app, etc.).
You will walk out of this one tired! I was mentally drained after focusing and running through the tasks over the 3.5 hours. It is a demanding test for sure!
Tips and Tricks
- Know each way (PowerCLI, command line, GUI) to do a task on the blueprint, but also have your preferred way. Yes, some things can only be accomplished via GUI or command line, but if not, do it your preferred way and do it fast!
- A great tip suggested by Tim Antonowicz (@timantz) to me at VMworld was to go through all the questions when you start and write down their objectives. Some tasks will build upon others you’ve previously completed and there are others that require you to wait while something completes. This way, you can group tasks together or know what you can move onto next while waiting.
- Be able to troubleshoot quickly. I’ve had practice with this in a production environment when your manager is breathing down your neck to get a server back up, but know where to look when things go wrong! Even if you haven’t had the job experience, do this in your lab. I’d even suggest letting a friend remote in and break a few things for you, then methodically go back and investigate/fix them.
- Write down the password for your system account. The password will be the same across all the different resources and it’s also displayed on the desktop. However, I found a few instances where I wasted time by having to move a window out of the way to see the password. If you’re like me, the stress of the exam makes you forget the password or you try typing in your own lab password which doesn’t quite work. Easy solution, write it down on your whiteboard!
- Make sure you know the other components (networking, storage, etc.) that can also affect a VMware environment. Familiarity with these and knowing a lot of the corner cases or lesser-used features will help you here!
- Chris Wahl (@chriswahl) put together an awesome DCA study sheet that you can check off each objective as you complete. This and the blueprint are a great way to start your studies:
- Gregg Robertson (@GreggRobertson5) has a good DCA5 objectives overview: http://thesaffageek.co.uk/vsphere-5-study-resources/vcap5-dca-objectives/
- Ed Grigson (@egrigson) has a good overview on changes between the DCA4 and DCA5 exams: http://www.vexperienced.co.uk/2012/04/12/vcap5-dca-whats-new/
- Jason Langer (@jaslanger) has great info up on different objectives: http://www.virtuallanger.com/vcap-dca-5/
- Josh Coen (@joshcoen) also goes into really good detail on the different objectives: http://www.valcolabs.com/vcap5-dca/
- Tim Antonowicz (@timantz) has a great post up on exam tips (including the one mentioned above) : http://whiteboardninja.wordpress.com/2012/09/19/vcap5-dca-testing-strategy-and-tips/
- As always, the crew at ProfesionalVMware put on a great series for VCAP5-DCA. Definitely watch/listen to all of the episodes to help prepare: http://professionalvmware.com/brownbags/
- Bas Raayman (@basraayman) has a good post up on his exam experience: http://basraayman.com/2012/08/30/vcap5-dca-my-test-experience/
One word. Patience When I finished up the exam, a note said results would come within 15 business days. A little over 40 business days later and mine finally came! I was very excited to receive the e-mail saying I had passed! It looks like there is a bit of a backlog over at the certification team right now though, so don’t be surprised if your results are delayed.
The Advanced track of the VMware datacenter virtualization certification ladder has been a fun one! I really enjoyed this exam and DCD. Now it’s onto VCDX, which I hope to be defending a defense at PEX in February 2013!
I received some awesome news this weekend. I am humbled and honored to be included among the vExperts for 2012. This is an award that recognizes members of what I believe is one the strongest communities across any vendor and technology. While the other names are far too many to list, to use a phrase, I am among giants. Check out the full list here: http://blogs.vmware.com/vmtn/2012/04/announcing-vexpert-2012-title-holders.html
This also serves as a reminder of how important the community is and how I enjoy giving back. Over the years I have had the support of many members from this group and I intend to do even more this year to give back. Make sure to hold me to that!
Congratulations to all the others for their hard work this year, it’s well deserved!
And lastly, a big thank you to VMware and all those involved in this great program!
As I’m sitting here on a semi-snowy New England Saturday, I’m reminded of a question that my good friend and fellow colleague, Mike Foley, always poses. Can we write a PowerShell script for this?
Automation and orchestration products, whether it be something like vCenter Orchestrator, or just simply leveraging PowerCLI for scripting, can really be of value to both a virtual environment and the team that manages it. My project for today was re-designing my lab infrastructure and included re-installing a lot of the vSphere components. I could make a list that is pages long of the manual configuration steps I’ve been running through. These steps are also prone to error. What if I’ve missed a setting on one host? Have I configured the portgroup differently on vSwitches? The point here is there’s a lot of room for error. Since I was aiming for consistency across my environment, an error here could come back to haunt me later.
In a previous life, I was a systems admin on a small team. I barely had time to manage day-to-day operations, let alone try to write automation scripts. Looking back on it though, my approach was error prone and with a little bit of up-front work, I could have saved myself hours.
A friend said to me recently, “Brian, I could automate myself out of a job. Why would I want to do that?”. On the surface, he made a good point, but I dove a little deeper with him. You would automate yourself out of what can sometimes be mindless or repetitive daily tasks. That is very true and one thing that PowerShell is great at! Wouldn’t it be great though if all these manual daily tasks were removed and then you could focus on more strategic, long-term projects that much more fully utilize your technology skillset? My friend was skeptical until he started thinking about this. He was eliminating portions of his job requirements, or making them a lot easier and less time consuming at least. Having a more strategic focus on projects sure sounded great though!
It’s ideas like these that make automation easier to grasp. Yes, it’s a time investment up front, no getting around that. However, the time you put in at the beginning, will be worth it in the long run!
Another advantage that you get through automation is increased security. We talk about a model in security where leveraging automation and orchestration can actually increase visibility for any unknown or potentially malicious actions that may be occurring. If you create a baseline of workflows that administrators or users can perform in an environment, you can build a foundation of consistency. Additionally, now that you have defined workflows that are known good, any actions outside of these stick out to your monitoring tools. It then makes it much easier to alert and report on these potential security risks.
I’ve only touched on a few of many reasons why automation should be given a serious consideration. I’ve been talking about this to more and more customers and I’d love if this blog post is even just a starting point for future conversations. Please feel free to chime in on this, how has automation helped your environment or made your job easier?
A little over a year ago, at VMworld 2010 in San Francisco, I took and passed the VCP 4 exam. This was my first VMware exam and at the time was based on vSphere 4.0. It was a hard test, no getting around that, but I felt it was pretty fair overall. As the test has now been upgraded to vSphere 5, I was curious to see how the exam had evolved over two generations (4.1 and 5). I’ve outlined some of my thoughts below and also included some resources that are useful for those studying for the exam.
(taken from VMware.com)
First let’s start with a quick discussion on the upgrade path, which has been somewhat of a debate in the community lately. VMware has a requirement of taking a certified, week-long course in addition to passing the VCP exam in order to become a VCP. This course is definitely a barrier to entry for some as the cost of it is around $2000.
Normally, if you are a VCP 3 or 4, you will need to take a “VMware vSphere: What’s New” course in addition to passing the exam to become a VCP 5. However, VMware has waived this requirement until February 29th, 2012 for VCP 4’s. So, if you’re a VCP 4, get out there and take the exam now!
VCP510 Exam Experience
I’ve taken a handful of industry certification tests from some of the major vendors (VMware, Cisco, Microsoft, etc.) and I think that VMware’s tests are some of the best for assessing real-world skills. To me, one of the marks of a good test is not just making you memorize numbers and specifications, but presenting questions that require you to use the knowledge that you’ve gained via studying in real-world situations. It’s one thing to memorize the entire vSphere 5 configuration maximums (you should be familiar with them nonetheless), but to make you go a little further and think how these affect other pieces in an environment is the more useful information.
Without breaking any of the NDA restrictions, I actually felt like the VCP 5 exam tested more on real-world knowledge and actual implementation questions than the VCP 4 test did. VMware did a good job of taking some realistic situations and then mapping the blueprint objectives and skills to these. I felt that the VCP 4 had a lot more of the straight memorization of numbers or more marketing/cloud intro material and this test moved away from that a bit. This was a welcome change as I feel the exam has a lot more practical value this way.
The test was 85 multiple-choice questions and you had 90 minutes to complete it. I felt like this was plenty of time and was done a little early, but I’m usually a quick test-taker so take that with a grain of salt.
I’ll say it straight out, this test is not easy. You absolutely need to have hands on experience using vSphere 5 or you will have a tough time with this test. As much information as there is in the PDF’s, there’s just something about knowing the interface and components from actually performing the tasks rather than just reading about them. This can also help you narrow down answers as you know which ones don’t fit based on actual usage. The PDF’s really only tell you one right answer whereas experience tells you which answers you can eliminate. For questions that have multiple answers that seem like they could be correct, this type of experience is valuable to help focus in on the best answer.
One mistake that I’ve heard a few people who were studying for this exam make is glossing over some of the installation/deployment sections. While it may seem like a lot of the focus of the exam topics are operational, you still have to know the install procedure for vCenter server. Likewise, know how each product is deployed, whether it be virtual appliances or installing on top of Windows. I’ll say this a few times, but don’t overlook some of the smaller details!
Another element that I like about this test is that it forces you to become familiar with some of the vSphere features you may not use on a daily basis. I used to work for a small company, so I didn’t get to use features such as NUMA, NPIV, or even FC storage. However, the exam tests you on features that are used across environments large and small. This means you get a much better exposure than you might normally get in your daily job. This is a perfect opportunity to get things working in your home lab. Or for those features that you can’t (I wish I had FC in my lab!), that’s when it’s time to hit the books and ask your peers!
Official Study Resources
Your first source of information should be the VCP 5 blueprint, located here:
Additionally, the blueprint links to all of the official VMware docs in PDF format. These are must reads and map directly to the blueprint! A direct link to the vSphere 5 Documentation Center is here: http://pubs.vmware.com/vsphere-50/index.jsp
http://mylearn.vmware.com/quiz.cfm?item=24908&ui=www_cert – VCP 5 mock exam. This is VMware’s official mock exam with some practice questions to try before taking the test.
Mastering VMware vSphere 4 – Scott Lowe
This is still the vSphere 4 version (Scott’s vSphere 5 update is coming the end of October), but it’s a great reference that covers a lot of the material on the blueprint in depth.
VMware vSphere 5 Clustering Technical Deepdive – Frank Denneman and Duncan Epping
This is the resource on HA/DRS and all things clustering. There have been major changes in HA between vSphere 4 and vSphere 5, so definitely check this one out to get up to speed!
Other Helpful Sites:
http://damiankarlson.com/category/vsphere-5/ – Damian has put together some great posts on some of the newer vSphere 5 features and also some VCP 5 resources.
http://blog.scottlowe.org/2009/11/27/understanding-npiv-and-npv/ – When I was first starting to learn some of the networking and FC components and how they fit into VMware, Scott’s post on NPIV was very helpful!
http://www.simonlong.co.uk/blog/vcp5-practice-exams/ – Simon has created both VCP 4 and VCP 5 practice tests. Definitely give these a try before the exam to see if you are on track with your studying.
I’ve linked to a few specific posts on these blogs, but just browse around and check the others out as they all have excellent vSphere-related content!
Also, if anyone has any resources to add to my list, please post them in the comments and I’ll be sure to include them!
Home Lab Tips
Nothing groundbreaking here, but just a few tips for those starting out with home labs and looking for studying advice.
Get two hosts if you can! You can even do this by virtualizing two instances of ESXi in VMware Workstation. Version 8 now supports nested 64-bit guests, so this opens a lot of possibilities! I have it running on my desktop and it works great. Either way though, try to build a setup with two hosts. There are just too many things you’ll miss out on by not having that second host.
Build out some common scenarios or designs and walk through the configuration of these. I’m currently studying for the VCAP-DCD, so I’ve been using some of those designs as examples and building them out. It always helps to have some examples like this where you can configure things end-to-end.
Lastly, break things! It’s your home lab, not production! Sometimes the situations when we learn the most are by fixing something that is broken. Try configuring something and if it doesn’t work, step through the process and figure out what happened and what was wrong. When you eventually fix the problem, you’ll have a much better understanding of the troubleshooting and of each component that you’ve worked with along the way.
I did pass the exam and am now a VCP 5. I have 5 years of hands-on experience administrating/using VMware and it still was not easy! Definitely make sure you know all of the topics on the blueprint. Also, don’t overlook things that you think may not be important. I think that practical knowledge is key here and not only being able to understand the concepts from the blueprint, but also how to use them. Finally, nothing beats hands-on! Get a home lab up and running if you haven’t already and play around with this stuff! It’s the best way to learn!
Best of luck to all those taking the test! I’d love to hear your thoughts!