Merge branch 'master' into markdig-nolist

Author: MightyPen

docs/advanced-analytics/java/extension-java.md

@@ -4,7 +4,7 @@ description: Run Java code on SQL Server 2019 using the Java language extension.
 ms.prod: sql
 ms.technology: machine-learning
 
-ms.date: 11/29/2018  
+ms.date: 12/07/2018
 ms.topic: conceptual
 author: HeidiSteen
 ms.author: heidist
@@ -42,7 +42,7 @@ On Windows, we recommend installing the JDK under the default /Program Files/ fo
 
 ## Install on Linux
 
-You can install the [database engine and the Java extension together](../../linux/sql-server-linux-setup-machine-learning.md#chained-installation), or add Java support to an existing instance. The following examples add the Java extension to an existing installation.  
+You can install the [database engine and the Java extension together](../../linux/sql-server-linux-setup-machine-learning.md#install-all), or add Java support to an existing instance. The following examples add the Java extension to an existing installation.  
 
 ```bash
 # RedHat install commands

docs/advanced-analytics/java/howto-call-java-from-sql.md

@@ -4,21 +4,21 @@ description: Learn how to call Java classes from SQL Server stored procedures us
 ms.prod: sql
 ms.technology: machine-learning
 
-ms.date: 09/24/2018  
+ms.date: 12/07/2018
 ms.topic: conceptual
 author: HeidiSteen
 ms.author: heidist
 manager: cgronlun
 monikerRange: ">=sql-server-ver15||=sqlallproducts-allversions"
 ---
 
-# How to call Java from SQL Server 2019
+# How to call Java from SQL Server 2019 preview
 
 When using the [Java language extension](extension-java.md), the [sp_execute_external_script](https://docs.microsoft.com/sql/relational-databases/system-stored-procedures/sp-execute-external-script-transact-sql) system stored procedure is the interface used to call the Java runtime. Permissions on the database apply to Java code execution.
 
 This article explains implementation details for Java classes and methods that execute on SQL Server. Once you are familiar with these details, review the [Java sample](java-first-sample.md) as your next step.
 
-## Basic principles
+### Basic principles
 
 * Compiled custom Java classes must exist in .class files or .jar files in your Java classpath. The [CLASSPATH parameter](#set-classpath) provides the path to the compiled Java files. 
 
@@ -29,15 +29,15 @@ This article explains implementation details for Java classes and methods that e
 * "params" is used to pass parameters to a Java class. Calling a method that requires arguments is not supported, which makes parameters the only way to pass argument values to your method. 
 
 > [!Note]
-> This note restates supported and unsupported operations specific to Java in CTP 2.0.
+> This note restates supported and unsupported operations specific to Java in CTP 2.x.
 > * On the stored procedure, input parameters are supported. Output parameters are not.
 > * Streaming using the sp_execute_external_script parameter **@r_rowsPerRead** is not supported.
 > * Partitioning using **@input_data_1_partition_by_columns** is not supported.
 > * Parallel processing using **@parallel=1** is supported.
 
-## Call sp_execute_external_script
+### Call sp_execute_external_script
 
-The [sp_execute_external_script](https://docs.microsoft.com/sql/relational-databases/system-stored-procedures/sp-execute-external-script-transact-sql) system stored procedure is the interface used to call the Java runtime. The following example shows an sp_execute_external_script using the Java extension, and parameters for specifying path, script, and your custom code.
+Applicable to both Windows and Linux, the [sp_execute_external_script](https://docs.microsoft.com/sql/relational-databases/system-stored-procedures/sp-execute-external-script-transact-sql) system stored procedure is the interface used to call the Java runtime. The following example shows an sp_execute_external_script using the Java extension, and parameters for specifying path, script, and your custom code.
 
 ```sql
 DECLARE @myClassPath nvarchar(30)
@@ -57,7 +57,7 @@ EXEC sp_execute_external_script
 
 <a name="set-classpath"></a>
 
-## Set CLASSPATH
+### Set CLASSPATH
 
 Once you have compiled your Java class or classes and placed the .class file(s) or .jar files in your Java classpath, you have two options for providing the classpath to the SQL Server Java extension:
 
@@ -72,27 +72,27 @@ One approach for specifying a path to compiled code is by setting CLASSPATH as a
 
 Just as you created a system variable for the JDK executables, you can create a system variable for code paths. To do this, created a system environment variable called "CLASSPATH"
 
-## Class requirements
+### Class requirements
 
 In order for SQL Server to communicate with the Java runtime, you need to implement specific static variables in your class. SQL Server can then execute a method in the Java class and exchange data using the Java language extension.
 
 > [!Note]
 > Expect the implementation details to change in upcoming CTPs as we work to improve the experience for developers.
 
-## Method requirements
+### Method requirements
 To pass arguments, use the **@param** parameter in sp_execute_external_script. The method itself cannot have any arguments. The return type must be void.  
 
 ```java
 public static void test()  {}
 ```
 
-## Data inputs 
+### Data inputs 
 
 This section explains how to push data to Java from a SQL Server query using **InputDataSet** in sp_execute_external_script.
 
 For every input column your SQL query pushes to Java, you need to declare an array.
 
-### inputDataCol
+#### inputDataCol
 
 In the current version of the Java extension, the **inputDataColN** variable is required, where *N* is the column number. 
 
@@ -106,7 +106,7 @@ Example: `public static int[] inputDataCol1 = new int[1];`
 
 These array variables will be populated with input data from a SQL server query before execution of the Java program you are calling.
 
-### inputNullMap
+#### inputNullMap
 
 Null map is used by the extension to know which values are null. This variable will be populated with information about null values by SQL Server before execution of the user function.
 
@@ -116,30 +116,30 @@ The user only needs to initialize this variable (and the size of the array needs
 public static boolean[][] inputNullMap = new boolean[1][1];
 ```
 
-## Data outputs 
+### Data outputs 
 
 This section describes **OutputDataSet**, the output data sets returned from Java, which you can send to and persist in SQL Server.
 
 > [!Note]
 > Output parameters in [sp_execute_external_script](https://docs.microsoft.com/sql/relational-databases/system-stored-procedures/sp-execute-external-script-transact-sql) are not supported in this version.
 
-### outputDataColN
+#### outputDataColN
 
 Similar to **inputDataSet**, for every output column your Java program sends back to SQL Server, you must declare an array variable. All **outputDataCol** arrays should have the same length. You need to make sure this is initialized by the time the class execution finishes.
 
 ```java
 public static <type>[] outputDataColN = new <type>[]
 ```
 
-### numberofOutputCols
+#### numberofOutputCols
 
 Set this variable to the number of output data columns you expect to have when the user function finishes execution.
 
 ```java
 public static short numberofOutputCols = <expected number of output columns>;
 ```
 
-### outputNullMap
+#### outputNullMap
 
 Null map is used by the extension to indicate which values are null. We require this since primitive types don't support null. Currently, we also require the null map for String types, even though Strings can be null. Null values are indicated by "true".
 
@@ -148,6 +148,8 @@ This NullMap must be populated with the expected number of columns and rows you
 ```java
 public static boolean[][] outputNullMap
 ```
+<a name="create-external-library"></a>
+
 
 ## Next steps
 

docs/advanced-analytics/what-s-new-in-sql-server-machine-learning-services.md

@@ -4,7 +4,7 @@ description: New feature announcements for each release of SQL Server 2016 R Ser
 ms.prod: sql
 ms.technology: machine-learning
 
-ms.date: 11/06/2018  
+ms.date: 12/07/2018
 ms.topic: conceptual
 author: HeidiSteen
 ms.author: heidist
@@ -23,9 +23,10 @@ This release adds the top-requested features for R and Python machine learning o
 
 | Release | Feature update |
 |---------|----------------|
-| CTP 2.0 | Linux platform support for R and Python machine learning, plus the new Java extension. For help getting started, see [Install SQL Server Machine Learning Services on Linux](../linux/sql-server-linux-setup-machine-learning.md). |
-| CTP 2.0 | The [sp_execute_external_script](https://docs.microsoft.com/sql/relational-databases/system-stored-procedures/sp-execute-external-script-transact-sql) includes two new parameters that enable you to easily generate multiple models from partitioned data. Learn more in this tutorial, [Create partition-based models in R](tutorials/r-tutorial-create-models-per-partition.md). |
-| CTP 2.0 | Failover cluster support is now supported on Windows and Linux, assuming SQL Server Launchpad service is started on all nodes. For more information, see [SQL Server failover cluster installation](../sql-server/failover-clusters/install/sql-server-failover-cluster-installation.md). |
+| CTP 2.0 | Linux platform support for R and Python machine learning. Get started with [Install SQL Server Machine Learning Services on Linux](../linux/sql-server-linux-setup-machine-learning.md). |
+|   | [Java language extension](java/extension-java.md) on both Windows and Linux is new in SQL Server 2019 preview. You can make compiled Java code available to SQL Server by assigning permissions and setting the path. Client apps with access SQL Server can use data and run your code by calling [sp_execute_external_script](https://docs.microsoft.com/sql/relational-databases/system-stored-procedures/sp-execute-external-script-transact-sql), the same procedure used for R and Python integration on SQL Server. | 
+|  | The [sp_execute_external_script](https://docs.microsoft.com/sql/relational-databases/system-stored-procedures/sp-execute-external-script-transact-sql) introduces two new parameters that enable you to easily generate multiple models from partitioned data. Learn more in this tutorial, [Create partition-based models in R](tutorials/r-tutorial-create-models-per-partition.md). |
+|   | Failover cluster support is now supported on Windows and Linux, assuming SQL Server Launchpad service is started on all nodes. For more information, see [SQL Server failover cluster installation](../sql-server/failover-clusters/install/sql-server-failover-cluster-installation.md). |
 ::: moniker-end
 
 ::: moniker range=">=sql-server-2017||=sqlallproducts-allversions"

docs/big-data-cluster/big-data-cluster-create-apps.md

@@ -5,7 +5,7 @@ description: Deploy a Python or R script as an application on SQL Server 2019 bi
 author: TheBharath 
 ms.author: bharaths  
 manager: craigg
-ms.date: 12/06/2018
+ms.date: 12/07/2018
 ms.topic: conceptual
 ms.prod: sql
 ms.custom: seodec18
@@ -15,7 +15,7 @@ ms.custom: seodec18
 
 This article describes how to deploy and manage R and Python script as an application inside a SQL Server 2019 big data cluster (preview).
 
-R and Python applications are deployed and managed with the **mssqlctl-pre** command-line utility which is included in CTP 2.1. This article provides examples of how to deploy these R and Python scripts as apps from the command line.
+R and Python applications are deployed and managed with the **mssqlctl-pre** command-line utility which is included in CTP 2.2. This article provides examples of how to deploy these R and Python scripts as apps from the command line.
 
 ## Prerequisites
 
@@ -26,12 +26,12 @@ You must have a SQL Server 2019 big data cluster configured. For more informatio
 The **mssqlctl-pre** command-line utility is provided to preview the Python and R application deployment feature. Use the following command to install the utility:
 
 ```cmd
-pip3 install --extra-index-url https://private-repo.microsoft.com/python/ctp-2.1 mssqlctlpre
+pip3 install --extra-index-url https://private-repo.microsoft.com/python/ctp-2.2 mssqlctlpre
 ```
 
 ## Capabilities
 
-In CTP 2.1 you can create, delete, list, and run an R or Python application. The following table describes the application deployment commands that you can use with **mssqlctl-pre**.
+In CTP 2.2 you can create, delete, list, and run an R or Python application. The following table describes the application deployment commands that you can use with **mssqlctl-pre**.
 
 | Command | Description |
 |---|---|
@@ -51,15 +51,16 @@ The following sections describe these commands in more detail.
 
 ## Log in
 
-Before configuring R and Python applications, first log into your SQL Server big data cluster with the `mssqlctl-pre login` command. Specify the IP address (external) of the `service-proxy-lb` (for example: `https://ip-address:30777`) along with the user name and password to the cluster.
+Before configuring R and Python applications, first log into your SQL Server big data cluster with the `mssqlctl-pre login` command. Specify the external IP address of the `service-proxy-lb` or `service-proxy-nodeport` services (for example: `https://ip-address:30777`) along with the user name and password to the cluster.
+
+You can get the IP address of the **service-proxy-lb** or **service-proxy-nodeport** service by running this command in a bash or cmd window:
 
-You can get the IP address of the service-proxy-lb service by running this command in a bash or cmd window:
 ```bash 
 kubectl get svc service-proxy-lb -n <name of your cluster>
 ```
 
 ```bash
-mssqlctl-pre login -e https://<ip-address-of-service-proxy-lb> -u <user-name> -p <password>
+mssqlctl-pre login -e https://<ip-address-of-service-proxy-lb>:30777 -u <user-name> -p <password>
 ```
 
 ## Create an app

docs/big-data-cluster/big-data-cluster-release-notes.md

@@ -5,7 +5,7 @@ description: This article describes the latest updates and known issues for SQL
 author: rothja 
 ms.author: jroth 
 manager: craigg
-ms.date: 12/06/2018
+ms.date: 12/07/2018
 ms.topic: conceptual
 ms.prod: sql
 ms.custom: seodec18
@@ -17,12 +17,74 @@ This article provides the latest updates and known issues for the latest release
 
 | Release | Date |
 |---|---|
+| [CTP 2.2](#ctp22) | December 2018 |
 | [CTP 2.1](#ctp21) | November 2018 |
 | [CTP 2.0](#ctp20) | October 2018 |
 
-
 [!INCLUDE [Limited public preview note](../includes/big-data-cluster-preview-note.md)]
 
+## <a id="ctp22"></a> CTP 2.2 (December 2018)
+
+The following sections describe the new features and known issues for big data clusters in SQL Server 2019 CTP 2.2.
+
+### What's in the CTP 2.2 release?
+
+- Use SparkR from Azure Data Studio on a big data cluster.
+- Cluster Admin Portal accessed with `/portal` (**https://\<ip-address\>:30777/portal**).
+- Master pool service name changed from `service-master-pool-lb` and `service-master-pool-nodeport` to `endpoint-master-pool`.
+- New version of **mssqlctl** and updated images.
+- Miscellaneous bug fixes and improvements.
+
+### Known issues
+
+The following sections provide known issues for SQL Server big data clusters in CTP 2.2.
+
+#### Deployment
+
+- Upgrading a big data data cluster from a previous release is not supported. You must backup and delete any existing big data cluster before deploying the latest release. For more information, see [Upgrade to a new release](deployment-guidance.md#upgrade).
+
+- After deploying on AKS, you might see the following two warning events from the deployment. Both of these events are known issues, but they do not prevent you from successfully deploying the big data cluster on AKS.
+
+   `Warning  FailedMount: Unable to mount volumes for pod "mssql-storage-pool-default-1_sqlarisaksclus(c83eae70-c81b-11e8-930f-f6b6baeb7348)": timeout expired waiting for volumes to attach or mount for pod "sqlarisaksclus"/"mssql-storage-pool-default-1". list of unmounted volumes=[storage-pool-storage hdfs storage-pool-mlservices-storage hadoop-logs]. list of unattached volumes=[storage-pool-storage hdfs storage-pool-mlservices-storage hadoop-logs storage-pool-java-storage secrets default-token-q9mlx]`
+
+   `Warning  Unhealthy: Readiness probe failed: cat: /tmp/provisioner.done: No such file or directory`
+
+- If a big data cluster deployment fails, the associated namespace is not removed. This could result in an orphaned namespace on the cluster. A workaround is to delete the namespace manually before deploying a cluster with the same name.
+
+#### External tables
+
+- It is possible to create a data pool external table for a table that has unsupported column types. If you query the external table, you get a message similar to the following:
+
+   `Msg 7320, Level 16, State 110, Line 44 Cannot execute the query "Remote Query" against OLE DB provider "SQLNCLI11" for linked server "(null)". 105079; Columns with large object types are not supported for external generic tables.`
+
+- If you query a storage pool external table, you might get an error if the underlying file is being copied into HDFS at the same time.
+
+   `Msg 7320, Level 16, State 110, Line 157 Cannot execute the query "Remote Query" against OLE DB provider "SQLNCLI11" for linked server "(null)". 110806;A distributed query failed: One or more errors occurred.`
+
+#### Spark and notebooks
+
+- POD IP addresses may change in the Kubernetes environment as PODs restarts. In the scenario where the master-pod restarts, the Spark session may fail with `NoRoteToHostException`. This is caused by JVM caches that don't get refreshed with new IP addresses.
+
+- If you have Jupyter already installed and a separate Python on Windows, Spark notebooks might fail. To work around this issue, upgrade Jupyter to the latest version.
+
+- In a notebook, if you click the **Add Text** command, the text cell is added in preview mode rather than edit mode. You can click on the preview icon to toggle to edit mode and edit the cell.
+
+#### HDFS
+
+- If you right-click on a file in HDFS to preview it, you might see the following error:
+
+   `Error previewing file: File exceeds max size of 30MB`
+
+   Currently there is no way to preview files larger than 30 MB in Azure Data Studio.
+
+- Configuration changes to HDFS that involve changes to hdfs-site.xml are not supported.
+
+#### Security
+
+- The SA_PASSWORD is part of the environment and discoverable (for example in a cord dump file). You must reset the SA_PASSWORD on the master instance after deployment. This is not a bug but a security step. For more information on how to change the SA_PASSWORD in a Linux container, see [Change the SA password](../linux/quickstart-install-connect-docker.md#sapassword).
+
+- AKS logs may contain SA password for big data cluster deployments.
+
 ## <a id="ctp21"></a> CTP 2.1 (November 2018)
 
 The following sections describe the new features and known issues for big data clusters in SQL Server 2019 CTP 2.1.

docs/big-data-cluster/cluster-admin-portal.md

@@ -5,7 +5,7 @@ description: Learn how to use the cluster administration portal to monitor SQL S
 author: yualan
 ms.author: alayu
 manager: craigg
-ms.date: 12/06/2018
+ms.date: 12/07/2018
 ms.topic: conceptual
 ms.prod: sql
 ms.custom: seodec18
@@ -26,10 +26,10 @@ The cluster administration portal allows you to:
 
 Follow the [quickstart to deploy your big data cluster](quickstart-big-data-cluster-deploy.md) until you get to the **cluster administration portal** section. Once you have the big data cluster running with mssqlctl, follow these instructions:
 
-Once the controller pod is running, you can use the cluster administration portal to monitor the deployment. You can access the portal using the external IP address and port number for the `service-proxy-lb` (for example: **https://\<ip-address\>:30777**). Credentials for accessing the admin portal are the values of `CONTROLLER_USERNAME` and `CONTROLLER_PASSWORD` environment variables provided above.
+Once the controller pod is running, you can use the cluster administration portal to monitor the deployment. You can access the portal using the external IP address and port number for the `service-proxy-lb` (for example: **https://\<ip-address\>:30777/portal**). Credentials for accessing the admin portal are the values of `CONTROLLER_USERNAME` and `CONTROLLER_PASSWORD` environment variables provided above.
 
 > [!NOTE]
-> For CTP 2.1, There is a security warning when accessing the web page since it is using auto-generated SSL certificates.
+> For CTP 2.2, There is a security warning when accessing the web page since it is using auto-generated SSL certificates.
 
 ## Overview