Installing external package on Azure ML (Error 0063)

Although there are number of packages already available on Azure ML Studio for various purposes like cleaning, feature selection, training, etc.

But sometimes while performing experiment we may need the packages that are not available on Azure ML studio. So we need to first upload that particular package/packages on the Azure ML and then install the same package for using it in the experiment.

Following are the steps to upload and install the package that are not already available/supported on/by the Azure ML:

  1. Depending upon the package that we need, download the package from their sources or install the package locally first using Command prompt or R studio(in case of R package) . For example :

If you need to install R package, you need to install this package locally first by using

install.packages("PackageName")

or you can directly download the zip file of that package from the CRAN.

  1. Build the zip file of the required packages that are downloaded from their respective sources or installed locally.
Important : Build the zip file with all dependencies of the package.
  1. Now again build the zip file of the zip file that we got from step 2. Sometimes we do miss this step and get a range of errors. But this is the most important step and package will not be installed without following this step.
  2. Upload this final zip file on the Azure ML by clicking on the '+' at the bottom of the Azure ML studio interface. Upload the package by going to DATASET > FROM LOCAL FILE. When you click on the FROM LOCAL FILE, following pop up will appear.You can choose the package from Local by clicking on the Choose File. Fill in the fields and select the different fields available on this popup based on the requirement.The zip file will be available in the datasets section of the Azure ML Studio(Screenshot).
  3. Now in order to install the package in the Azure ML, you need to drag and drop the zip file and Execute R script and connect them as shown in the below screenshot. And write the below r script in the Execute R script and execute.You can tune the parameters based on the requirement. Just in case you have multiple packages in the zip file then you have to install all the packages one by one using R script. Remember you need to install th dependencies first.

Is imbalanced data is really a problem?

In most of the real time problems(such as Fraud detection, Spam filtering, Disease screening, SaaS subscription churn, Advertising click-throughs, etc), the problem of imbalanced data occurs. The presence of imbalanced data in the real world scenario is not only common but expected. Imbalanced data is data with disproportionate distribution of classes.

For example : If you are working with a binary classification problem in which the ratio of Class A to Class B is 100:10 that is the number of samples of class A is more than class B. Due of this, the results overfit class A with high accuracy most of the times and you will feel frustrated when you found that all of the great results you thought you were getting turn out to be a lie. In such cases, machine learning does not produce satisfactory classifiers.

This will make the prediction difficult and there is a need to treat the imbalanced data first. There are several approaches to treat the imbalanced data and balance the classes.

You can change your data into a balanced form by performing artificial sampling. There are several methods that are described below:

Undersampling is the process by which observations belonging to majority class are randomly deleted to make the make the minority class observations nearly equal to majority class observations. Balanced data are good for classification, but you obviously loose information about appearance frequencies using this method, which is going to affect accuracy metrics themselves, as well as performance.

Oversampling is the process by which observations are generated belonging to minority class by randomly duplicating/replicating the observations of minority class. In this case, although there is no information loss but duplicating the minority data to increase the data will increase the likelihood of overfitting.

 

SMOTE(Synthetic Minority Over-sampling Technique) involves the introduction of new synthetic samples rather than creating new observations by replication. Although SMOTE is better than simple oversampling, still it loose its effectiveness when you have high dimensional data.

Through extensive experimentation, it is found that these algorithms are good enough to balance the data but there are several drawbacks due to which they are not effectiveness in some cases. So it is advised to use the hybrid of techniques rather than relying on the any of the single technique or you can modify the basic sampling approaches.

We do have a number of approaches that can help us to balance our balanced. But the question is do we really need to balance the data? Because when we are dealing with the imbalanced data, it is obvious to have imbalanced data. Although balancing the data will increase the accuracy, does it really provide a effective and robust classifier?

As we can see from the different techniques that they either remove the observations(which leads to loss of data) to balance the data or replicate or synthetically increase the data(which generate observations that are not genuine). It is really important to understand various facts before balancing the data.

 

Whether to balance the data or not is entirely dependent on the type of use case you are working on, reasons for the missing data or the type of ML algorithms that you are using for training.

 

  1. If you want to build a predictive model, using basic machine learning models such as Decision Tree, SVM, Naive Bayes, etc and using Area under curve, rank, etc as performance parameters then balanced data is a good choice.
  2. If you want to build a predictive model using a advanced machine learning framework i.e. something that determines sampling parameters via wrapper or a modification of a bagging framework that samples to class equivalence, then the representative sample is a good choice. As advanced machine learning framework take care of balancing training data on its own.
  3. If you want to build a representative model -- representative model can be a unsupervised model that describes the data rather than necessarily predicts -- then representative sample of data is a good choice.

 

When we have the unbalanced data it is not always necessary to balance that sample. There are several other things that should be considered before balancing the data. It is really important analysis the type of use case, model and data that you are using in the experiment.

 

“Error requesting http://xp0.sc/sitecore/admin/PopulateManagedSchema.aspx?indexes=all”

Stuck on the very last step on installation! This one was the worst of all. Too many things that could be wrong. I had to look at too many things to resolve this one

[------------------------------------------------------------------------- UpdateSolrSchema : SitecoreUrl ------------------------------------------------------------------------------]
[UpdateSolrSchema]:[Authenticating] xp0.sc/.../PopulateManagedSchema.aspx
Install-SitecoreConfiguration : Error requesting xp0.sc/.../PopulateManagedSchema.aspx The remote server returned an error: (500) Internal Server Error.
At [path to install file]\install.ps1:68 char:1
+ Install-SitecoreConfiguration @sitecoreParams
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Install-SitecoreConfiguration

[TIME] 00:03:11
Transcript stopped, output file is [output file path]
Invoke-SitecoreUrlTask : Error requesting xp0.sc/.../PopulateManagedSchema.aspx The remote server returned an error: (500) Internal Server Error.
At C:\Program Files\WindowsPowerShell\Modules\SitecoreInstallFramework\1.2.0\Public\Install-SitecoreConfiguration.ps1:253 char:21
+ & $entry.Task.Command @paramSet | Out-Default
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException

Solution

The issue with my installation was that some of the versions of .Net Framework were not installed. As suggested in Sitecore Community, the issue was resolved after I manually installed/enabled ASP, ASP.NET 3.5, ASP .NET 4.7 from Windows Add/Remove Features

Source: Sitecore Community

‘Database ‘sc9_Xdb.Collection.Shard1′ does not exist. Make sure that the name is entered correctly.’

I got this error while the post installation steps were running. So this one was particularly annoying as I was so confident that I did everything right.

I was getting the below error when running post installation sql script - 'Database 'sc9_Xdb.Collection.Shard1' does not exist. Make sure that the name is entered correctly.'

Solution

It turns out I didn't install a pre-requisite of Sitecore 9. Microsoft SQL Server Transact-SQL ScriptDom to be particular

Check out our Sitecore 9 Pre-requisites to download Microsoft SQL Server Transact-SQL ScriptDom: Sitecore 9 Pre-requisites

Source: Sitecore Community

Sitecore 9 Pre-Requisites

The following post will focus on the pre-requisites required for installing Sitecore 9 along with xConnect.

These should be applicable with Sitecore 9 Update 1 and Sitecore 9 Update 2 installation as well

This post is applicable for Windows 10 Environment and we will be generating a few Self Signed Certificates for our installation using Windows 10.

Here are the tools required:

  • Visual Studio 2015 or Higher
  • .Net Framework 4.6.2
  • SQL Version 2016 SP1 (Any version lower will cause issues with xConnect)
  • SOLR 6.6.2 (Sitecore recommended)
  • IIS Enabled
  • Web Platform Installer (WebPi 5.0)
  • Web Deploy 3.6
  • Microsoft SQL Server Data-Tier Application Framework (DACFx)

Let go through the prerequisites one by one.

Visual Studio

If you are installing sitecore for development puposes, you can get the latest version of Visual Studio Community (currently 2017) from Microsoft using a Developer Account.

Visual Studio 2017 Download

.Net Framework 4.6.2

You can download it directly from Microsoft

.Net Framework 4.6.2 Download

SQL Server 2016

Just like Visual Studio, for development purposes, you can get the SQL Server directly from Microsoft using a developer's account.

Microsoft SQL 2016 Download

Note: MS SQL Express has a limitation on the database size you can restore (10 GB). The MS SQL Developer Edition however does not have any such limitation.

SOLR 6.6.2

You can download SOLR 6.6.2 from Apache Archive

SOLR 6.6.2 Download

IIS Enabled

IIS is disabled by default in Windows 10. If you have not enabled IIS before, follow the steps below:

  • In Windows, access the Control Panel and click Add or Remove Programs.
  • In the Add or Remove Programs window, click Add/Remove Windows Components.
  • Select the Internet Information Services (IIS) check box, click Next, then click Finish.

Web Platform Installer (WebPi 5.0)

You can download the Web Platform Installer from Microsoft Website

Web Platform Installer 5.0 Download

Web Deploy 3.6

You can download Web Deploy 3.6 from Microsoft

Web Deploy 3.6 Download

Microsoft SQL Server Data-Tier Application Framework (DACFx)

You can download DACFx from Microsoft

 

Microsoft SQL Server Data-Tier Application Framework Download

“Error: .NET SqlClient Data Provider Msg 12809” You must remove all users with password before setting the containment property to NONE.

This is another issue that I faced initially while installing Sitecore 9.

Details: You must remove all users with password before setting the containment property to NONE.

Solution

In my case, this issue was related to incomplete or corrupt databases with the sitename already being present in SQL. This occurred because there was an error while I was installing Sitecore 9 earlier. So always check for existing databases before installing a new instance of Sitecore 9.

Check out our guide to uninstall Sitecore 9 instance: Uninstalling Sitecore 9

Source: Sitecore Community

Unable to parse [resolvepath(parameter(‘Package’))] – Exception calling “InvokeWithContext” with “2” argument(s)

This is one of the errors that I got while installing Sitecore 9 for the first time. Even after following the Sitecore 9 Installation guide step by step, I was getting the following Error

[---------------------------------------------- InstallWDP : WebDeploy -----------------------------------------------]
Install-SitecoreConfiguration : Unable to parse [resolvepath(parameter('Package'))] - Exception calling
"InvokeWithContext" with "2" argument(s): "The running command stopped because the preference variable
"ErrorActionPreference" or common parameter is set to Stop: Illegal characters in path."
At C:\resourcefiles\install.ps1:43 char:1
+ Install-SitecoreConfiguration @xconnectParams
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Install-SitecoreConfiguration

[TIME] 00:00:06
Transcript stopped, output file is C:\resourcefiles\xconnect-xp0.180628.log
GetConfigFunctionResult : Unable to parse [resolvepath(parameter('Package'))] - Exception calling "InvokeWithContext"
with "2" argument(s): "The running command stopped because the preference variable "ErrorActionPreference" or common
parameter is set to Stop: Illegal characters in path."
At C:\Program Files\WindowsPowerShell\Modules\SitecoreInstallFramework\1.2.1\Private\JsonConfiguration.ps1:182 char:20
+ ... return GetConfigFunctionResult -FunctionText $InputObject -Param ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,GetConfigFunctionResult

Solution

This issue was related to the format of the name of the package defined in the install.ps1 file. In my case, the package name was coming in two lines

Package Name Incorrect

instead of

Fixing this resolved my isssue. Listed below are some other reasons that might cause this:

  • When copying the install.ps1 script from the documentation, there were fancy quotes that had to be replaced with regular quotes
  • When copying the install.ps1 script from the documentation, make sure your "Package" lines for the Sitecore...sxwdp..zip files are on ONE line
  • If you put a special character in your password, use single quotes

Source: Sitecore Community

Setting Up SSL using Self Signed Certificate in Powershell

While installing Sitecore 9, I faced some issues while enabling SSL on the instance. I found few tutorials that explained how it can be done. The process included multiple steps, so I decided to write a powershell script to make it simple.

Just provide the Host Name and Website Name in IIS and the script will take care of:

  1. Creating a new Self Signed Certificate.
  2. Moving Certificate from Personal to Root Store.
  3. Adding 443 binding to the website in IIS.

Sources used:

Copying Certificate from one Store to Another

Setting Up SOLR with SSL in Windows 10

If you are trying to install Sitecore 9, you need to setup SOLR server and all the transfer of data between Sitecore and SOLR has to be secure.

This guide will help you setup SOLR with SSL on Windows 10 Device. We are using Powershell to create a Self Signed Certificate for SSL.

Getting SOLR 6.6.2

SOLR 6.6.2 is Sitecore recommended version of SOLR for Sitecore 9.

You can get it from SOLR Archive

SOLR 6.6.2 Download

Creating Certificates

  • Install latest version of Java Runtime Environment (JRE) on the System.
  • Make sure JAVA_HOME variable is setup in System Variables
  • Run the following command in Powershell as Administrator
  • Search for "Manage Computer Certificates" in start Menu
  • Open the Personal → Certificates node on the left pane, and right click on the newly created certificate.
  • Copy the certificate to Trusted Root Certification Authorities/Certificates
  • On the pop-up menu, select All Tasks → Export...:
    1. On the second wizard screen, select "Yes, export the private key" and click Next.
    2. Make sure only the "Include all certificates in the certification path if possible" option is selected, and click Next.
    3. Check the "Password" checkbox, and enter (and confirm) a password. Take note of this password. For this example we will use "secret". Click Next.
    4. Select a location to create the PFX file, give it a name (for this example, "solr_ssl.pfx") and click Next to finish the wizard.

Installing SOLR

  • Unzip the Solr installation file to get the Solr-xxx Folder. where xxx is Solr version. Note: Sitecore recommends v6.6.2 for Sitecore 9
  • Copy this Solr-xxx folder to some simple directory like C:/
  • We should not have Solr folder setup as C:/Solr-xxx.
  • Copy the solr_ssl.pfx created in the last step to this folder.
  • In a text editor, open the C:\Solr-xxx\bin\solr.in.cmd file and uncomment the following section:
  • Replace this section with the following code
  • Make sure to C:/solr-xxx is replaced with the path of the folder where Solr is installed
  • Test the Solr installation by running the following command in the Command Prompt:
  • Navigate tho https://localhost:8983 in Web browser, no Privacy warning should be thrown
  • Stop Solr in the Command Prompt by pressing Ctrl-C
  • Follow this guide Installing Solr as a service using NSSM to install Solr as a Windows Service