High-level diagram illustrating that 2 certificates are used in app controller:
1st for https://(to app controller web site) in my case a wildcard certificate on the HTTPS listener for the published Web Site
2nd – for console connection on fixed port 2179 tied to Hyper-V Virtual Machine Management Service. This is a self-signed certificate on the Hyper-V host.
Remote Desktop Connection (RDP) also uses the self-signed certificate but it uses port (3389) You can dumb down the Remote Desktop Certificate Authentication to not use TLS and will still not connect using the Console (VMConnect).

The certificate used for console connection uses port a fixed port (2179) not port (3389) for Remote Desktop Connection  … (see diagram)

This blog will focus on the console connection (my preferred connection) because you can console into all Virtual Machines running on the Hyper-V host, each displayed in a seperate tab depending on the Browser version of course.

In this example ssp.domain.com is the self service portal (app controller) published via Forefront TMG 2010 using a HTTPS Publishing rule using a third-party wildcard certificate. Also the internal and external dns domains match for simplicity.

HTTPS Web Publishing Rule

From:                       Anywhere 
To:                         ssp.domain.com

HTTPS Listener:    
Authentication method:      Basic Authentication 
Validation Method:          Windows (Active Directory) 
Certificate:                *.domain.com
Public name:                ssp.domain.com 
Path:                       /*
Authentication Delegation:  No delegation, but client may authenticate directly 
Bridging:                   Redirect requests to SSL port: 443 
Users:                      All users

Verify that the AppController api virtual directory Authentication Method is set to Basic.

Also the WEB Server Certificate (a third-party wildcard certificate in my case) is exported to the TMG’s computer store. (see below for details regarding exporting and importing certificate)

Console access  to each Hyper-V Host(s) will require a external IP address per host because vmconnect uses a fixed port (2179).

Create a non web publishing rule (see image) Note: The published port is 2179, This will create a listener port 2179 on the Forefront TMG external interface that will map to the Hyper-V Host.

vmconnect uses a self-signed certificate tied to the Hyper-V Virtual Machine Management service. If you try to proceed you will end up with and empty window (see next image).

Certificate is not trusted result. You do not need to add this certificate on TMG, simply export and add to the Internet clients Trusted Certificate Store. See below.

Select View certificate… note that the certificate CA Root is not  trusted. Click The top Details tab and select Copy to File… This will launch the Certificate Export Wizard click Next > The default of DER encoded (.CER) is fine Next> …. for the File to Export enter the name of the host.cer and save in your Documents folder example: hv1.cer and select Finish. The export should be successful. To place the certificate into the Internet client’s Local Computer Trusted Root Certificate Authorities do the following:

Click Start – Search programs and files search box type “mmc”  and select File Add/Remove snap-in… and select Certificates Add> make sure to select Computer account Next> – Local computer – Finish and Click OK. Expand Certificates (Local Computer) and highlight Trusted Root Certification Authorities. Right-Click All Tasks – Import.. and place the certificate file into Trusted Root Certificate Authorities. (see image below)

Verify that the certificate appears in the Trusted Root Certification Authorities. If the hyper-v host certificate import is successful you should see a console screen appear

Additional Microsoft Links:

Configuring Certificates for Virtual Machine Connection
http://technet.microsoft.com/en-us/library/ff935311(WS.10).aspx
Advanced – replacing the self signed certificate with third party certificate
http://blogs.technet.com/b/hugofe/archive/2011/10/04/configuring-a-certificate-for-virtual-machine-connection-in-hyper-v-or-thru-scvmm.aspx
Read in a MS TechNet blog somewhere that wildcard * for hostname not supported 🙁