Skip to content

Commit

Permalink
[ISSUE-390]Optimize document details (#644)
Browse files Browse the repository at this point in the history
  • Loading branch information
binbinCheng authored Jan 4, 2023
1 parent 48a9020 commit 7028018
Show file tree
Hide file tree
Showing 56 changed files with 281 additions and 866 deletions.
36 changes: 18 additions & 18 deletions docs/about/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,11 @@ sidebar_position: 3
---


# 1. Recommended configuration of hardware and software environment
## 1. Recommended configuration of hardware and software environment

Linkis builds a layer of computing middleware between the upper application and the underlying engine. As an open source distributed computing middleware, it can be well deployed and run on Intel architecture servers and mainstream virtualization environments, and supports mainstream Linux operating system environments

## 1.1. Linux operating system version requirements
### 1.1. Linux operating system version requirements

| OS | Version |
| --- | --- |
Expand All @@ -19,11 +19,11 @@ Linkis builds a layer of computing middleware between the upper application and

> **Note:** The above Linux operating systems can run on physical servers and mainstream virtualization environments such as VMware, KVM, and XEN
## 1.2. Server recommended configuration
### 1.2. Server recommended configuration

Linkis supports 64-bit general-purpose hardware server platforms running on the Intel x86-64 architecture. The following recommendations are made for the server hardware configuration of the production environment:

### Production Environment
#### Production Environment

| **CPU** | **Memory** | **Disk type** | **Network** | **Number of instances** |
| --- | --- | --- | --- | --- |
Expand All @@ -34,7 +34,7 @@ Linkis supports 64-bit general-purpose hardware server platforms running on the
> - The above recommended configuration is the minimum configuration for deploying Linkis, and a higher configuration is strongly recommended for production environments
> - The hard disk size configuration is recommended to be 50GB+, and the system disk and data disk are separated
## 1.3. Software requirements
### 1.3. Software requirements

Linkis binary packages are compiled based on the following software versions:

Expand All @@ -53,14 +53,14 @@ Linkis binary packages are compiled based on the following software versions:
> **Note:**
> If the locally installed component version is inconsistent with the above, you need to modify the corresponding component version and compile the binary package yourself for installation.
## 1.4. Client web browser requirements
### 1.4. Client web browser requirements

Linkis recommends Chrome version 73 for front-end access


# 2. Common scenarios
## 2. Common scenarios

## 2.1 Open test mode
### 2.1 Open test mode
The development process requires a password-free interface, which can be replaced or appended to `linkis.properties`

| parameter name | default value | description |
Expand All @@ -71,7 +71,7 @@ The development process requires a password-free interface, which can be replace
![](./images/test-mode.png)


## 2.2 Login user settings
### 2.2 Login user settings
Apache Linkis uses configuration files to manage admin users by default, and this configuration can be replaced or appended to `linkis-mg-gateway.properties`. For multi-user access LDAP implementation.

| parameter name | default value | description |
Expand All @@ -82,7 +82,7 @@ Apache Linkis uses configuration files to manage admin users by default, and thi
![](./images/login-user.png)


## 2.3 LDAP Settings
### 2.3 LDAP Settings
Apache Linkis can access LDAP through parameters to achieve multi-user management, and this configuration can be replaced or added in `linkis-mg-gateway.properties`.

| parameter name | default value | description |
Expand All @@ -93,7 +93,7 @@ Apache Linkis can access LDAP through parameters to achieve multi-user managemen

![](./images/ldap.png)

## 2.4 Turn off resource checking
### 2.4 Turn off resource checking
Apache Linkis sometimes debugs exceptions when submitting tasks, such as: insufficient resources; you can replace or append this configuration in `linkis-cg-linkismanager.properties`.

| parameter name | default value | description |
Expand All @@ -102,7 +102,7 @@ Apache Linkis sometimes debugs exceptions when submitting tasks, such as: insuff

![](./images/resource-enable.png)

## 2.5 Enable engine debugging
### 2.5 Enable engine debugging
Apache Linkis EC can enable debugging mode, and this configuration can be replaced or added in `linkis-cg-linkismanager.properties`.

| parameter name | default value | description |
Expand All @@ -111,7 +111,7 @@ Apache Linkis EC can enable debugging mode, and this configuration can be replac

![](./images/engine-debug.png)

## 2.6 Hive metadata configuration
### 2.6 Hive metadata configuration
The public-service service of Apache Linkis needs to read hive metadata; this configuration can be replaced or appended in `linkis-ps-publicservice.properties`.

| parameter name | default value | description |
Expand All @@ -122,7 +122,7 @@ The public-service service of Apache Linkis needs to read hive metadata; this co

![](./images/hive-meta.png)

## 2.7 Linkis database configuration
### 2.7 Linkis database configuration
Apache Linkis access uses Mysql as data storage by default, you can replace or append this configuration in `linkis.properties`.

| parameter name | default value | description |
Expand All @@ -133,7 +133,7 @@ Apache Linkis access uses Mysql as data storage by default, you can replace or a

![](./images/linkis-db.png)

## 2.8 Linkis Session cache configuration
### 2.8 Linkis Session cache configuration
Apache Linkis supports using redis for session sharing; this configuration can be replaced or appended in `linkis.properties`.

| parameter name | default value | description |
Expand All @@ -145,7 +145,7 @@ Apache Linkis supports using redis for session sharing; this configuration can b

![](./images/redis.png)

## 2.9 Linkis module development configuration
### 2.9 Linkis module development configuration
When developing Apache Linkis, you can use this parameter to customize the database, Rest interface, and entity objects of the loading module; you can modify it in `linkis-ps-publicservice.properties`, and use commas to separate multiple modules.

| parameter name | default value | description |
Expand All @@ -157,7 +157,7 @@ When developing Apache Linkis, you can use this parameter to customize the datab

![](./images/deverlop-conf.png)

## 2.10 Linkis module development configuration
### 2.10 Linkis module development configuration
This parameter can be used to customize the route of loading modules during Apache Linkis development; it can be modified in `linkis.properties`, and commas are used to separate multiple modules.

| parameter name | default value | description |
Expand All @@ -166,7 +166,7 @@ This parameter can be used to customize the route of loading modules during Apac

![](./images/list-conf.png)

## 2.11 Linkis file system and material storage path
### 2.11 Linkis file system and material storage path
This parameter can be used to customize the route of loading modules during Apache Linkis development; it can be modified in `linkis.properties`, and commas are used to separate multiple modules.

| parameter name | default value | description |
Expand Down
24 changes: 12 additions & 12 deletions docs/about/introduction.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,17 +16,17 @@ Since the first release of Linkis in 2019, it has accumulated more than **700**

## Features

- **Support for diverse underlying computation storage engines**.
- **Support for diverse underlying computation storage engines**
Currently supported computation/storage engines: Spark, Hive, Python, Presto, ElasticSearch, MLSQL, TiSpark, JDBC, Shell, etc;
Computation/storage engines to be supported: Flink(Supported in version >=1.0.2), Impala, etc;
Supported scripting languages: SparkSQL, HiveQL, Python, Shell, Pyspark, R, Scala and JDBC, etc.
- **Powerful task/request governance capabilities**. With services such as Orchestrator, Label Manager and customized Spring Cloud Gateway, Linkis is able to provide multi-level labels based, cross-cluster/cross-IDC fine-grained routing, load balance, multi-tenancy, traffic control, resource control, and orchestration strategies like dual-active, active-standby, etc.
- **Support full stack computation/storage engine**. As a computation middleware, it will receive, execute and manage tasks and requests for various computation storage engines, including batch tasks, interactive query tasks, real-time streaming tasks and storage tasks;
- **Resource management capabilities**. ResourceManager is not only capable of managing resources for Yarn and Linkis EngineManger as in Linkis 0.X, but also able to provide label-based multi-level resource allocation and recycling, allowing itself to have powerful resource management capabilities across multiple Yarn clusters and multiple computation resource types;
- **Unified Context Service**. Generate Context ID for each **task**/request, associate and manage user and system resource files (JAR, ZIP, Properties, etc.), result set, parameter variable, function, etc., across user, system, and computing engine. Set in one place, automatic reference everywhere;
- **Unified materials**. System and user-level unified material management, which can be shared and transferred across users and systems.
- **Unified Data Source Manage**. Provides functions such as adding, deleting, checking, and modifying data sources of hive, es, mysql, and kafka types, version control, and connection testing.
- **Unified MetaData Manage**. Provides database, table, and partition queries for hive, es, mysql, and kafka metadata.
- **Powerful task/request governance capabilities** With services such as Orchestrator, Label Manager and customized Spring Cloud Gateway, Linkis is able to provide multi-level labels based, cross-cluster/cross-IDC fine-grained routing, load balance, multi-tenancy, traffic control, resource control, and orchestration strategies like dual-active, active-standby, etc.
- **Support full stack computation/storage engine** As a computation middleware, it will receive, execute and manage tasks and requests for various computation storage engines, including batch tasks, interactive query tasks, real-time streaming tasks and storage tasks;
- **Resource management capabilities** ResourceManager is not only capable of managing resources for Yarn and Linkis EngineManger as in Linkis 0.X, but also able to provide label-based multi-level resource allocation and recycling, allowing itself to have powerful resource management capabilities across multiple Yarn clusters and multiple computation resource types.
- **Unified Context Service** Generate Context ID for each **task**/request, associate and manage user and system resource files (JAR, ZIP, Properties, etc.), result set, parameter variable, function, etc., across user, system, and computing engine. Set in one place, automatic reference everywhere.
- **Unified materials** System and user-level unified material management, which can be shared and transferred across users and systems.
- **Unified Data Source Manage** Provides functions such as adding, deleting, checking, and modifying data sources of hive, es, mysql, and kafka types, version control, and connection testing.
- **Unified MetaData Manage** Provides database, table, and partition queries for hive, es, mysql, and kafka metadata.

## Supported engine types

Expand Down Expand Up @@ -56,15 +56,15 @@ Please refer to [Deployment_Documents](../deployment/deploy-quick.md) to do the
## Examples and Guidance
- [Engine Usage Guidelines](../engine-usage/overview.md)
- [API Documentation](../api/overview.md)
-

## Documentation

The documentation of linkis is in [Linkis-WebSite](https://github.com/apache/linkis-website)

## Architecture
Linkis services could be divided into three categories: computation governance services, public enhancement services and microservice governance services.
- The computation governance services, support the 3 major stages of processing a task/request: submission -> preparation -> execution;
- The public enhancement services, including the material library service, context service, and data source service;
- The computation governance services, support the 3 major stages of processing a task/request: submission -> preparation -> execution.
- The public enhancement services, including the material library service, context service, and data source service.
- The microservice governance services, including Spring Cloud Gateway, Eureka and Open Feign.

Below is the Linkis architecture diagram. You can find more detailed architecture docs in [Architecture](../architecture/overview.md).
Expand All @@ -90,7 +90,7 @@ More projects upcoming, please stay tuned.

## Contributing

Contributions are always welcomed, we need more contributors to build Linkis together. either code, or doc, or other supports that could help the community.
Contributions are always welcomed, we need more contributors to build Linkis together. either code, or doc or other supports that could help the community.
For code and documentation contributions, please follow the [contribution guide](/community/how-to-contribute).

## Contact Us
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ sidebar_position: 9
## 1. General
### 1.1 Requirements Background
     Linkis now performs load balancing based on the ribbon when it forwards services in the Gateway, but in some cases, there are some important business tasks that want to achieve service level isolation, if the service is based on the ribbon There will be problems in equilibrium. For example, tenant A wants his tasks to be routed to a specific Linkis-CG-Entrance service, so that when other instances are abnormal, the Entrance of service A will not be affected.
     In addition, tenants and isolation of support services can also quickly isolate an abnormal service and support scenarios such as grayscale upgrades.
In addition, tenants and isolation of support services can also quickly isolate an abnormal service and support scenarios such as grayscale upgrades.

### 1.2 Target
1. Support forwarding the service according to the routing label by parsing the label of the request
Expand Down
6 changes: 2 additions & 4 deletions docs/deployment/deploy-quick.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Linkis supports specifying the user who submits and executes. The linkis main pr

Take hadoop user as an example:

First check whether there is already a hadoop user in the system. If it already exists, you can directly authorize it; if not, create a user first, and then authorize.
First check whether there is already a hadoop user in the system. If it already exists, you can directly authorize it, if not, create a user first, and then authorize.

Check if a hadoop user already exists
```shell script
Expand Down Expand Up @@ -410,9 +410,7 @@ sh bin/linkis-cli -submitUser hadoop -engineType python-python2 -codeType python
If the verification fails, please refer to [Step 8] for troubleshooting
## 6 Installation of development tool IDE (Scriptis) (optional)
>After installing the Scripti tool, it can support writing SQL, Pyspark, HiveQL and other scripts online on the web page
For detailed instructions, see [Installation and Deployment of Tool Scriptis](integrated/install-scriptis)
After installing the Scripti tool, it can support writing SQL, Pyspark, HiveQL and other scripts online on the web page,For detailed instructions, see [Installation and Deployment of Tool Scriptis](integrated/install-scriptis)
## 7. Supported Engines
Expand Down
Loading

0 comments on commit 7028018

Please sign in to comment.