Lokasi ngalangkungan proxy:   [ UP ]  
[Ngawartoskeun bug]   [Panyetelan cookie]                
Skip to content

Commit 4a1d89e

Browse files
v-albemiv-albemi
authored
Merge pull request #19929 from DaniBunny/20210804-bdc-cu12-library-details
library details to BDC CU12
2 parents 90637d0 + e57d58a commit 4a1d89e

3 files changed

Lines changed: 125 additions & 5 deletions

File tree

‎docs/big-data-cluster/concept-application-deployment.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -102,7 +102,7 @@ The followings are the target scenarios for app deploy:
102102

103103
In app deploy, BDC python runtime allows Python application inside the big data cluster to address variety of use cases such as machine learning inferencing, API serving and more.
104104

105-
Python 3.5 for Ubuntu 16.04 and Python 3.8 for Ubuntu 20.04.
105+
The app deploy Python runtime uses Python 3.8 on SQL Server Big Data Clusters CU10+.
106106

107107
In app deploy, `spec.yaml` is where you provide the information that controller needs to know to deploy your application. The following are the fields that can be specified:
108108

@@ -146,7 +146,7 @@ App deploy Python runtime doesn't support scheduling scenario. Once Python app i
146146

147147
In app deploy, BDC Python runtime allows R application inside the big data cluster to address variety of use cases such as machine learning inferencing, API serving and more.
148148

149-
The app deploy R runtime supports Microsoft R Open (MRO) 3.5.2.
149+
The app deploy R runtime uses Microsoft R Open (MRO) version 3.5.2 on SQL Server Big Data Clusters CU10+.
150150

151151
#### How to use it?
152152

‎docs/big-data-cluster/configure-bdc-postdeployment.md‎

Lines changed: 18 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,8 @@ titleSuffix: SQL Server big data clusters
44
description: Big Data Clusters Post-Deployment Configuration Overview
55
author: MikeRayMSFT
66
ms.author: mikeray
7-
ms.reviewer: rahul.ajmera
8-
ms.date: 02/11/2021
7+
ms.reviewer: dacoelho
8+
ms.date: 08/04/2021
99
ms.topic: reference
1010
ms.prod: sql
1111
ms.technology: big-data-cluster
@@ -24,37 +24,51 @@ Cluster, service, and resource scoped settings for Big Data Clusters can be conf
2424
## Step by Step: Configure BDC to meet your Spark workload requirements
2525

2626
### View the current configurations of the Big Data Cluster Spark service
27+
2728
The following example shows how to view the user configured settings of the Spark service. You can view all possible configurable settings, system-managed and all configurable settings, and pending settings through optional parameters. Visit [`azdata bdc spark` statement](../azdata/reference/reference-azdata-bdc-spark-statement.md) for more information.
2829

2930
```bash
3031
azdata bdc spark settings show
3132
```
3233
#### Sample output
33-
Spark Service
34+
35+
Spark Service
3436

3537
|Setting|Running Value|
3638
| --- | --- |
3739
|`spark-defaults-conf.spark.driver.cores`|`1` |
3840
|`spark-defaults-conf.spark.driver.memory`|`1664m` |
3941

4042
### Change the default number of cores and memory for the Spark driver across all resources with Spark (i.e. for the Spark service)
43+
4144
Update the default number of cores to 2 and default memory to 7424m for the Spark service.
4245

4346
```bash
4447
azdata bdc spark settings set --settings spark-defaults-conf.spark.driver.cores=2,spark-defaults-conf.spark.driver.memory=7424m
4548
```
4649

4750
### Change the default number of cores and memory for the Spark executors in the Storage Pool
51+
4852
Update the default number of executor cores to 4 for the Storage Pool.
4953

5054
```bash
5155
azdata bdc spark settings set --settings spark-defaults-conf.spark.executor.cores=4 --resource=storage-0
5256
```
5357

58+
### Configure additional paths to the default classpath of Spark applications
59+
60+
The ```/opt/hadoop/share/hadoop/tools/lib/``` path contains several libraries to be used by your spark applications, but the referred path is not loaded by default in the classpath of Spark applications. To enable this setting apply the following configuration pattern.
61+
62+
```bash
63+
azdata bdc hdfs settings set --settings hadoop-env.HADOOP_CLASSPATH="/opt/hadoop/share/hadoop/tools/lib/*"
64+
```
65+
5466
### View the pending settings changes staged in the big data cluster
67+
5568
View the pending settings changes for the Spark service only and across the entire big data cluster.
5669

5770
#### Pending Spark Service Settings
71+
5872
```bash
5973
azdata bdc spark settings show --filter-option=pending --include-details
6074
```
@@ -67,6 +81,7 @@ azdata bdc spark settings show --filter-option=pending --include-details
6781
|`spark-defaults-conf.spark.driver.memory`|`1664m`| `7424m` | `true` | `true` |
6882

6983
#### All Pending Settings
84+
7085
```bash
7186
azdata bdc settings show --filter-option=pending --include-details --recursive
7287
```

‎docs/big-data-cluster/release-notes-cumulative-update-12.md‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,7 @@ Reference Architecture White Papers for SQL Server Big Data Clusters can be foun
4545
* __Scala__: 2.11
4646
* __Python__: 3.7.2 (miniconda 4.5.12)
4747
* __R__: Microsoft R 3.5.2
48+
* __Spark SQL Connector__: [1.0.2](https://github.com/microsoft/sql-spark-connector)
4849

4950
## Embedded OSS component versions
5051

@@ -66,6 +67,110 @@ Reference Architecture White Papers for SQL Server Big Data Clusters can be foun
6667
| [Telegraf](https://docs.influxdata.com/telegraf/) | 1.16.1 |
6768
| [ZooKeeper](https://cwiki.apache.org/confluence/display/zookeeper) | 3.5.8 |
6869

70+
## Installed Python libraries
71+
72+
| Library | Version | Library | Version | Library | Version |
73+
|--|--|--|--|--|--|
74+
| revoscalepy | 9.4.7 | pycurl | 7.43.0.2 | jupyter-client | 5.2.4 |
75+
| microsoftml | 9.4.7 | pycrypto | 2.6.1 | jsonschema | 2.6.0 |
76+
| zipp | 0.5.2 | pycparser | 2.19 | Jinja2 | 2.10.1 |
77+
| zict | 1.0.0 | pycosat | 0.6.3 | jedi | 0.15.1 |
78+
| xlwt | 1.3.0 | py | 1.8.0 | jdcal | 1.4 |
79+
| XlsxWriter | 1.1.2 | ptyprocess | 0.6.0 | itsdangerous | 1.1.0 |
80+
| xlrd | 1.2.0 | psutil | 5.4.8 | ipywidgets | 7.4.2 |
81+
| wrapt | 1.11.2 | prompt-toolkit | 2.0.7 | ipython | 7.2.0 |
82+
| widgetsnbextension | 3.4.2 | prometheus-client | 0.7.1 | ipython-genutils | 0.2.0 |
83+
| wheel | 0.33.4 | pluggy | 0.12.0 | ipykernel | 4.9.0 |
84+
| Werkzeug | 0.15.5 | plotly | 4.0.0 | importlib-metadata | 0.19 |
85+
| webencodings | 0.5.1 | pkginfo | 1.5.0.1 | imagesize | 1.1.0 |
86+
| wcwidth | 0.1.7 | pip | 18.1 | imageio | 2.5.0 |
87+
| urllib3 | 1.24.2 | Pillow | 6.1.0 | idna | 2.8 |
88+
| unicodecsv | 0.14.1 | pickleshare | 0.7.5 | heapdict | 1.0.0 |
89+
| traitlets | 4.3.2 | pexpect | 4.7.0 | hdijupyterutils | 0.12.9 |
90+
| tornado | 6.0.3 | patsy | 0.5.1 | h5py | 2.9.0 |
91+
| toolz | 0.9.0 | pathlib2 | 2.3.3 | gmpy2 | 2.0.8 |
92+
| testpath | 0.4.2 | path.py | 11.5.0 | Flask | 1.1.1 |
93+
| terminado | 0.8.2 | partd | 0.3.9 | Flask-Cors | 3.0.8 |
94+
| tblib | 1.4.0 | parso | 0.5.1 | fastcache | 1.1.0 |
95+
| tables | 3.5.2 | pandocfilters | 1.4.2 | et-xmlfile | 1.0.1 |
96+
| sympy | 1.3 | pandasql | 0.7.3 | entrypoints | 0.3 |
97+
| statsmodels | 0.9.0 | pandas | 0.23.4 | docutils | 0.15.2 |
98+
| sqlparse | 0.2.4 | pandas-datareader | 0.7.0 | distributed | 1.28.1 |
99+
| SQLAlchemy | 1.2.15 | packaging | 19.1 | dill | 0.2.8.2 |
100+
| sphinxcontrib-websupport | 1.1.2 | openpyxl | 2.5.12 | defusedxml | 0.6.0 |
101+
| Sphinx | 1.8.2 | olefile | 0.46 | decorator | 4.4.0 |
102+
| sparkmagic | 0.12.6 | odo | 0.5.1 | datashape | 0.5.4 |
103+
| sortedcontainers | 2.1.0 | numpydoc | 0.8.0 | dask | 1.2.2 |
104+
| snowballstemmer | 1.9.0 | numpy | 1.15.4 | cytoolz | 0.9.0.1 |
105+
| six | 1.12.0 | numexpr | 2.6.9 | Cython | 0.29.2 |
106+
| setuptools | 41.0.1 | numba | 0.42.0 | cycler | 0.10.0 |
107+
| Send2Trash | 1.5.0 | notebook | 5.7.8 | cryptography | 2.5 |
108+
| seaborn | 0.9.0 | nltk | 3.4 | configobj | 5.0.6 |
109+
| scipy | 1.1.0 | networkx | 2.2 | conda | 4.5.12 |
110+
| scikit-learn | 0.20.2 | nbformat | 4.4.0 | cloudpickle | 0.6.1 |
111+
| scikit-image | 0.14.1 | nbconvert | 5.5.0 | Click | 7.0 |
112+
| ruamel-yaml | 0.15.46 | multipledispatch | 0.6.0 | chest | 0.2.3 |
113+
| retrying | 1.3.3 | msgpack | 0.6.1 | chardet | 3.0.4 |
114+
| requests | 2.21.0 | mpmath | 1.1.0 | cffi | 1.12.3 |
115+
| requests-kerberos | 0.12.0 | more-itertools | 7.2.0 | certifi | 2019.6.16 |
116+
| qtconsole | 4.5.3 | mock | 3.0.5 | Bottleneck | 1.2.1 |
117+
| pyzmq | 18.1.0 | mkl-random | 1.0.2 | bokeh | 1.0.3 |
118+
| PyYAML | 5.1.2 | mkl-fft | 1.0.10 | bleach | 3.1.0 |
119+
| PyWavelets | 1.0.3 | mistune | 0.8.4 | blaze | 0.11.3 |
120+
| pytz | 2019.2 | matplotlib | 3.0.2 | backports.os | 0.1.1 |
121+
| python-dateutil | 2.8.0 | MarkupSafe | 1.1.0 | backcall | 0.1.0 |
122+
| pytest | 4.0.2 | lxml | 4.3.0 | Babel | 2.7.0 |
123+
| PySocks | 1.7.0 | locket | 0.2.0 | azureml-model-management-sdk | 1.0.1b10 |
124+
| pyparsing | 2.4.2 | llvmlite | 0.27.0 | autovizwidget | 0.12.9 |
125+
| pyOpenSSL | 19.0.0 | liac-arff | 2.4.0 | attrs | 19.1.0 |
126+
| pyodbc | 4.0.25 | kiwisolver | 1.1.0 | atomicwrites | 1.3.0 |
127+
| pykerberos | 1.2.1 | jupyter | 1.0.0 | asn1crypto | 0.24.0 |
128+
| PyJWT | 1.7.1 | jupyter-core | 4.5.0 | alabaster | 0.7.12 |
129+
| Pygments | 2.4.2 | jupyter-console | 6.0.0 | adal | 1.2.2 |
130+
131+
## Installed R libraries
132+
133+
| Library | Version | Library | Version | Library | Version |
134+
|--|--|--|--|--|--|
135+
CompatibilityAPI | 1.1.0 | crayon | 1.3.4 | plogr | 0.2.0 |
136+
MicrosoftML | 9.4.7 | curl | 3.3 | plyr | 1.8.4 |
137+
RevoPemaR | 10.0.0 | datasets | 3.5.2 | png | 0.1-7 |
138+
RevoScaleR | 9.4.7 | dbplyr | 1.2.2 | promises | 1.0.1 |
139+
RevoTreeView | 10.0.0 | deployrRserve | 9.0.0 | psych | 1.8.10 |
140+
doRSR | 10.0.0 | digest | 0.6.18 | purrr | 0.2.5 |
141+
mrsdeploy | 1.1.3 | doParallel | 1.0.14 | r2d3 | 0.2.3 |
142+
sqlrutils | 1.0.0 | dplyr | 0.7.8 | rappdirs | 0.3.1 |
143+
BH | 1.69.0-1 | fansi | 0.4.0 | readr | 1.3.1 |
144+
DBI | 1.0.0 | foreach | 1.5.1 | reshape2 | 1.4.3 |
145+
KernSmooth | 2.23-15 | foreign | 0.8-71 | rlang | 0.3.1 |
146+
MASS | 7.3-51.1 | forge | 0.1.0 | rpart | 4.1-13 |
147+
Matrix | 1.2-15 | generics | 0.0.2 | rprojroot | 1.3-2 |
148+
MicrosoftR | 3.5.2 | glue | 1.3.0 | rstudioapi | 0.9.0 |
149+
R6 | 2.3.0 | grDevices | 3.5.2 | shiny | 1.2.0 |
150+
RUnit | 0.4.26 | graphics | 3.5.2 | sourcetools | 0.1.7 |
151+
Rcpp | 1.0.0 | grid | 3.5.2 | sparklyr | 0.9.4 |
152+
RevoIOQ | 10.0.1 | hms | 0.4.2 | spatial | 7.3-11 |
153+
RevoMods | 11.0.1 | htmltools | 0.3.6 | splines | 3.5.2 |
154+
RevoUtils | 11.0.2 | htmlwidgets | 1.3 | stats | 3.5.2 |
155+
RevoUtilsMath | 11.0.0 | httpuv | 1.4.5.1 | stats4 | 3.5.2 |
156+
askpass | 1.1 | httr | 1.4.0 | stringi | 1.2.4 |
157+
assertthat | 0.2.0 | iterators | 1.0.11 | stringr | 1.3.1 |
158+
backports | 1.1.3 | jsonlite | 1.5 | survival | 2.43-3 |
159+
base | 3.5.2 | later | 0.7.5 | sys | 2.1 |
160+
base64enc | 0.1-3 | lattice | 0.20-38 | tcltk | 3.5.2 |
161+
bindr | 0.1.1 | lazyeval | 0.2.1 | tibble | 2.0.1 |
162+
bindrcpp | 0.2.2 | magrittr | 1.5 | tidyr | 0.8.2 |
163+
boot | 1.3-20 | methods | 3.5.2 | tidyselect | 0.2.5 |
164+
broom | 0.5.1 | mgcv | 1.8-26 | tools | 3.5.2 |
165+
checkpoint | 0.4.4 | mime | 0.6 | utf8 | 1.1.4 |
166+
class | 7.3-14 | mnormt | 1.5-5 | utils | 3.5.2 |
167+
cli | 1.0.1 | nlme | 3.1-137 | withr | 2.1.2 |
168+
clipr | 0.5.0 | nnet | 7.3-12 | xml2 | 1.2.0 |
169+
cluster | 2.0.7-1 | openssl | 1.2.1 | xtable | 1.8-3 |
170+
codetools | 0.2-15 | parallel | 3.5.2 | yaml | 2.2.0 |
171+
compiler | 3.5.2 | pillar | 1.3.1 | | |
172+
config | 0.3 | pkgconfig | 2.0.2 | | |
173+
69174
## Next steps
70175

71176
For more information about [!INCLUDE[big-data-clusters-2019](../includes/ssbigdataclusters-ss-nover.md)], see [Introducing [!INCLUDE[big-data-clusters-2019](../includes/ssbigdataclusters-ver15.md)]](big-data-cluster-overview.md)

0 commit comments

Comments
 (0)