🌐 AI搜索 & 代理 主页
Skip to content

Commit d5d1ac7

Browse files
chillenbergerchillenberger
committed
Merge branch 'master' into dan-product-left-nav-update
2 parents 88ce70b + c5d8c6f commit d5d1ac7

File tree

43 files changed

+871
-262
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

43 files changed

+871
-262
lines changed

‎pgml-cms/docs/.gitbook/assets/architecture_1.png‎

82.7 KB
Loading

‎pgml-cms/docs/.gitbook/assets/architecture_2.png‎

117 KB
Loading

‎pgml-cms/docs/.gitbook/assets/architecture_3.png‎

118 KB
Loading

‎pgml-cms/docs/.gitbook/assets/performance_1.png‎

127 KB
Loading

‎pgml-cms/docs/.gitbook/assets/performance_2.png‎

49.7 KB
Loading

‎pgml-cms/docs/README.md‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -21,16 +21,18 @@ PostgresML allows you to take advantage of the fundamental relationship between
2121

2222
<figure><img src=".gitbook/assets/ml_system.svg" alt="Machine Learning Infrastructure (2.0) by a16z"><figcaption class="mt-2"><p>PostgresML handles all of the functions <a href="https://a16z.com/emerging-architectures-for-modern-data-infrastructure/">described by a16z</a></p></figcaption></figure>
2323

24-
These capabilities are primarily provided by two open-source software projects, that may be used independently, but are designed to be used with the rest of the Postgres ecosystem:
24+
These capabilities are primarily provided by two open-source software projects, that may be used independently, but are designed to be used together with the rest of the Postgres ecosystem:
2525

26-
* **pgml** - an open source extension for PostgreSQL. It adds support for GPUs and the latest ML & AI algorithms _inside_ the database with a SQL API and no additional infrastructure, networking latency, or reliability costs
27-
* **PgCat** - an open source pooler for PostgreSQL. It abstracts the scalability and reliability concerns of managing a distributed cluster of Postgres databases. Client applications connect only to the pooler, which handles load balancing, sharding, and failover, outside of any single database server.
26+
* [**pgml**](/docs/api/sql-extension/) - an open source extension for PostgreSQL. It adds support for GPUs and the latest ML & AI algorithms _inside_ the database with a SQL API and no additional infrastructure, networking latency, or reliability costs.
27+
* [**PgCat**](/docs/product/pgcat/) - an open source connection pooler for PostgreSQL. It abstracts the scalability and reliability concerns of managing a distributed cluster of Postgres databases. Client applications connect only to the pooler, which handles load balancing, sharding, and failover, outside of any single database server.
2828

2929
<figure><img src=".gitbook/assets/architecture.png" alt="PostgresML architectural diagram"><figcaption></figcaption></figure>
3030

31+
To learn more about how we designed PostgresML, take a look at our [architecture overview](/docs/resources/architecture/).
32+
3133
## Client SDK
3234

33-
The PostgresML team also provides [native language SDKs](https://github.com/postgresml/postgresml/tree/master/pgml-sdks/pgml) which implement best practices for common ML & AI applications. The JavaScript and Python SDKs are generated from the a core Rust library, which provides a uniform API, correctness and efficiency across all environments.
35+
The PostgresML team also provides [native language SDKs](/docs/api/client-sdk/) which implement best practices for common ML & AI applications. The JavaScript and Python SDKs are generated from the a core Rust library, which provides a uniform API, correctness and efficiency across all environments.
3436

3537
While using the SDK is completely optional, SDK clients can perform advanced machine learning tasks in a single SQL request, without having to transfer additional data, models, hardware or dependencies to the client application.
3638

‎pgml-cms/docs/SUMMARY.md‎

Lines changed: 15 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -15,9 +15,19 @@
1515
## API
1616

1717
* [Overview](api/apis.md)
18-
* [SQL Extension](api/sql-extension/README.md)
19-
* [pgml.deploy()](api/sql-extension/pgml.deploy.md)
18+
* [SQL extension](api/sql-extension/README.md)
2019
* [pgml.embed()](api/sql-extension/pgml.embed.md)
20+
* [pgml.transform()](api/sql-extension/pgml.transform/README.md)
21+
* [Fill-Mask](api/sql-extension/pgml.transform/fill-mask.md)
22+
* [Question answering](api/sql-extension/pgml.transform/question-answering.md)
23+
* [Summarization](api/sql-extension/pgml.transform/summarization.md)
24+
* [Text classification](api/sql-extension/pgml.transform/text-classification.md)
25+
* [Text Generation](api/sql-extension/pgml.transform/text-generation.md)
26+
* [Text-to-Text Generation](api/sql-extension/pgml.transform/text-to-text-generation.md)
27+
* [Token Classification](api/sql-extension/pgml.transform/token-classification.md)
28+
* [Translation](api/sql-extension/pgml.transform/translation.md)
29+
* [Zero-shot Classification](api/sql-extension/pgml.transform/zero-shot-classification.md)
30+
* [pgml.deploy()](api/sql-extension/pgml.deploy.md)
2131
* [pgml.chunk()](api/sql-extension/pgml.chunk.md)
2232
* [pgml.generate()](api/sql-extension/pgml.generate.md)
2333
* [pgml.predict()](api/sql-extension/pgml.predict/README.md)
@@ -29,16 +39,6 @@
2939
* [Data Pre-processing](api/sql-extension/pgml.train/data-pre-processing.md)
3040
* [Hyperparameter Search](api/sql-extension/pgml.train/hyperparameter-search.md)
3141
* [Joint Optimization](api/sql-extension/pgml.train/joint-optimization.md)
32-
* [pgml.transform()](api/sql-extension/pgml.transform/README.md)
33-
* [Fill Mask](api/sql-extension/pgml.transform/fill-mask.md)
34-
* [Question Answering](api/sql-extension/pgml.transform/question-answering.md)
35-
* [Summarization](api/sql-extension/pgml.transform/summarization.md)
36-
* [Text Classification](api/sql-extension/pgml.transform/text-classification.md)
37-
* [Text Generation](api/sql-extension/pgml.transform/text-generation.md)
38-
* [Text-to-Text Generation](api/sql-extension/pgml.transform/text-to-text-generation.md)
39-
* [Token Classification](api/sql-extension/pgml.transform/token-classification.md)
40-
* [Translation](api/sql-extension/pgml.transform/translation.md)
41-
* [Zero-shot Classification](api/sql-extension/pgml.transform/zero-shot-classification.md)
4242
* [pgml.tune()](api/sql-extension/pgml.tune.md)
4343
* [Client SDK](api/client-sdk/README.md)
4444
* [Collections](api/client-sdk/collections.md)
@@ -79,13 +79,15 @@
7979

8080
## Resources
8181

82+
* [Architecture](resources/architecture/README.md)
83+
* [Why PostgresML?](resources/architecture/why-postgresml.md)
8284
* [FAQs](resources/faqs.md)
8385
* [Data Storage & Retrieval](resources/data-storage-and-retrieval/tabular-data.md)
8486
* [Tabular data](resources/data-storage-and-retrieval/tabular-data.md)
8587
* [Documents](resources/data-storage-and-retrieval/documents.md)
8688
* [Partitioning](resources/data-storage-and-retrieval/partitioning.md)
8789
* [LLM based pipelines with PostgresML and dbt (data build tool)](resources/data-storage-and-retrieval/llm-based-pipelines-with-postgresml-and-dbt-data-build-tool.md)
88-
* [Benchmarks](resources/benchmarks/README.md)
90+
* [Benchmarks](resources/benchmarks/postgresml-is-8-40x-faster-than-python-http-microservices.md)
8991
* [PostgresML is 8-40x faster than Python HTTP microservices](resources/benchmarks/postgresml-is-8-40x-faster-than-python-http-microservices.md)
9092
* [Scaling to 1 Million Requests per Second](resources/benchmarks/million-requests-per-second.md)
9193
* [MindsDB vs PostgresML](resources/benchmarks/mindsdb-vs-postgresml.md)
@@ -97,8 +99,6 @@
9799
* [Contributing](resources/developer-docs/contributing.md)
98100
* [Distributed Training](resources/developer-docs/distributed-training.md)
99101
* [GPU Support](resources/developer-docs/gpu-support.md)
100-
* [Deploying PostgresML](resources/developer-docs/deploying-postgresml/README.md)
101-
* [Monitoring](resources/developer-docs/deploying-postgresml/monitoring.md)
102102
* [Self-hosting](resources/developer-docs/self-hosting/README.md)
103103
* [Pooler](resources/developer-docs/self-hosting/pooler.md)
104104
* [Building from source](resources/developer-docs/self-hosting/building-from-source.md)

‎pgml-cms/docs/api/apis.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,10 +18,11 @@ The PostgreSQL extension provides all of the ML & AI functionality, like trainin
1818

1919
The following functions are implemented and maintained by the PostgresML extension:
2020

21-
| Function name | Description |
21+
| Function | Description |
2222
|------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
2323
| [pgml.embed()](sql-extension/pgml.embed) | Generate embeddings inside the database using open source embedding models from Hugging Face. |
2424
| [pgml.transform()](sql-extension/pgml.transform/) | Download and run latest Hugging Face transformer models, like Llama, Mixtral, and many more to perform various NLP tasks like text generation, summarization, sentiment analysis and more. |
25+
| pgml.transform_stream() | Streaming version of [pgml.transform()](sql-extension/pgml.transform/). Retrieve tokens as they are generated by the LLM, decreasing time to first token. |
2526
| [pgml.train()](sql-extension/pgml.train/) | Train a machine learning model on data from a Postgres table or view. Supports XGBoost, LightGBM, Catboost and all Scikit-learn algorithms. |
2627
| [pgml.deploy()](sql-extension/pgml.deploy) | Deploy a version of the model created with pgml.train(). |
2728
| [pgml.predict()](sql-extension/pgml.predict/) | Perform real time inference using a model trained with pgml.train() on live application data. |
@@ -33,7 +34,7 @@ Together with standard database functionality provided by PostgreSQL, these func
3334

3435
The client SDK implements best practices and common use cases, using the PostgresML SQL functions and standard PostgreSQL features to do it. The SDK core is written in Rust, which manages creating and running queries, connection pooling, and error handling.
3536

36-
For each additional language we support (current JavaScript and Python), we create and publish language-native bindings. This architecture ensures all programming languages we support have identical APIs and similar performance when interacting with PostgresML.
37+
For each additional language we support (currently JavaScript and Python), we create and publish language-native bindings. This architecture ensures all programming languages we support have identical APIs and similar performance when interacting with PostgresML.
3738

3839
### Use cases
3940

‎pgml-cms/docs/api/client-sdk/tutorials/semantic-search-1.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,16 +6,16 @@ description: Example for Semantic Search
66

77
This tutorial demonstrates using the `pgml` SDK to create a collection, add documents, build a pipeline for vector search, make a sample query, and archive the collection when finished. In this tutorial we use [hkunlp/instructor-base](https://huggingface.co/hkunlp/instructor-base), a more advanced embeddings model that takes parameters when doing embedding and recall.
88

9-
[Link to full JavaScript implementation](../../../../../pgml-sdks/pgml/javascript/examples/question\_answering.js)
9+
[Link to full JavaScript implementation](https://github.com/postgresml/postgresml/blob/master/pgml-sdks/pgml/javascript/examples/question_answering.js)
1010

11-
[Link to full Python implementation](../../../../../pgml-sdks/pgml/python/examples/question\_answering.py)
11+
[Link to full Python implementation](https://github.com/postgresml/postgresml/blob/master/pgml-sdks/pgml/python/examples/question_answering.py)
1212

1313
## Imports and Setup
1414

1515
The SDK is imported and environment variables are loaded.
1616

1717
{% tabs %}
18-
{% tab title="JavasScript" %}
18+
{% tab title="JavaScript" %}
1919
```js
2020
const pgml = require("pgml");
2121
require("dotenv").config();

‎pgml-cms/docs/api/client-sdk/tutorials/semantic-search.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -8,16 +8,16 @@ description: >-
88

99
This tutorial demonstrates using the `pgml` SDK to create a collection, add documents, build a pipeline for vector search, make a sample query, and archive the collection when finished.
1010

11-
[Link to full JavaScript implementation](../../../../../pgml-sdks/pgml/javascript/examples/semantic\_search.js)
11+
[Link to full JavaScript implementation](https://github.com/postgresml/postgresml/blob/master/pgml-sdks/pgml/javascript/examples/semantic_search.js)
1212

13-
[Link to full Python implementation](../../../../../pgml-sdks/pgml/python/examples/semantic\_search.py)
13+
[Link to full Python implementation](https://github.com/postgresml/postgresml/blob/master/pgml-sdks/pgml/python/examples/semantic_search.py)
1414

1515
## Imports and Setup
1616

1717
The SDK is imported and environment variables are loaded.
1818

1919
{% tabs %}
20-
{% tab title="JavasScript" %}
20+
{% tab title="JavaScript" %}
2121
```js
2222
const pgml = require("pgml");
2323
require("dotenv").config();

0 commit comments

Comments
 (0)