more

postgresml · levkk · Apr 24, 2024 · Apr 23, 2024 · Apr 24, 2024 · Apr 24, 2024
commit cd61aaaa7df9634f10f2dfaf38fc870e0e3f6951
diff --git a/pgml-cms/docs/README.md b/pgml-cms/docs/README.md
@@ -28,7 +28,7 @@ These capabilities are primarily provided by two open-source software projects,
 
 <figure><img src=".gitbook/assets/architecture.png" alt="PostgresML architectural diagram"><figcaption></figcaption></figure>
 
-## SDK
+## Client SDK
 
 The PostgresML team also provides [native language SDKs](https://github.com/postgresml/postgresml/tree/master/pgml-sdks/pgml) which implement best practices for common ML & AI applications. The JavaScript and Python SDKs are generated from the a core Rust library, which provides a uniform API, correctness and efficiency across all environments.
 

diff --git a/pgml-cms/docs/SUMMARY.md b/pgml-cms/docs/SUMMARY.md
@@ -1,15 +1,16 @@
 # Table of contents
 
-## Introduction
+## Getting started
 
 * [Overview](README.md)
 * [Getting Started](introduction/getting-started/README.md)
   * [Create your database](introduction/getting-started/create-your-database.md)
   * [Connect your app](introduction/getting-started/connect-your-app.md)
-  * [Import your data](introduction/getting-started/import-your-data/README.md)
-    * [Logical replication](introduction/getting-started/import-your-data/logical-replication/README.md)
-    * [Foreign Data Wrappers](introduction/getting-started/import-your-data/foreign-data-wrappers.md)
-    * [COPY](introduction/getting-started/import-your-data/copy.md)
+* [Import your data](introduction/getting-started/import-your-data/README.md)
+  * [Logical replication](introduction/getting-started/import-your-data/logical-replication/README.md)
+  * [Foreign Data Wrappers](introduction/getting-started/import-your-data/foreign-data-wrappers.md)
+  * [Move data with COPY](introduction/getting-started/import-your-data/copy.md)
+  * [Migrate with pg_dump](introduction/getting-started/import-your-data/pg-dump.md)
 
 ## API
 
@@ -39,7 +40,7 @@
     * [Translation](api/sql-extension/pgml.transform/translation.md)
     * [Zero-shot Classification](api/sql-extension/pgml.transform/zero-shot-classification.md)
   * [pgml.tune()](api/sql-extension/pgml.tune.md)
-* [SDK](api/client-sdk/README.md)
+* [Client SDK](api/client-sdk/README.md)
   * [Overview](api/client-sdk/getting-started.md)
   * [Collections](api/client-sdk/collections.md)
   * [Pipelines](api/client-sdk/pipelines.md)

diff --git a/pgml-cms/docs/api/client-sdk/README.md b/pgml-cms/docs/api/client-sdk/README.md
@@ -1,4 +1,4 @@
-# Client SDKs
+# Client SDK
 
 ### Key Features
 

diff --git a/pgml-cms/docs/introduction/getting-started/README.md b/pgml-cms/docs/introduction/getting-started/README.md
@@ -4,14 +4,14 @@ description: Setup a database and connect your application to PostgresML
 
 # Getting Started
 
-A PostgresML deployment consists of multiple components working in concert to provide a complete Machine Learning platform. We provide a fully managed solution in our cloud, and document a self-hosted installation in our docs.
+A PostgresML deployment consists of multiple components working in concert to provide a complete Machine Learning platform. We provide a fully managed solution in [our cloud](create-your-database), and document a self-hosted installation in [Developer Docs](/docs/resources/developer-docs/quick-start-with-docker).
 
 * PostgreSQL database, with `pgml`, `pgvector` and many other extensions installed, including backups, metrics, logs, replicas and high availability
 * PgCat pooler to provide secure access and model load balancing across thousands of clients
-* A web application to manage deployed models and write experiments in SQL notebooks
+* A web application to manage deployed models and share experiments and analysis in SQL notebooks
 
-<figure><img src="../../.gitbook/assets/architecture.png" alt=""><figcaption></figcaption></figure>
+<figure class="m-3"><img src="../../.gitbook/assets/architecture.png" alt=""><figcaption></figcaption></figure>
 
 By building PostgresML on top of a mature database, we get reliable backups for model inputs and proven scalability without reinventing the wheel, so that we can focus on providing access to the latest developments in open source machine learning and artificial intelligence.
 
-This guide will help you get started with a generous free account, that includes access to GPU accelerated models and 5GB of storage, or you can skip to our Developer Docs to see how to run PostgresML locally with our Docker image.
+This guide will help you get started with a generous free account, that includes access to GPU accelerated models and 5 GB of storage, or you can skip to our [Developer Docs](/docs/resources/developer-docs/quick-start-with-docker) to see how to run PostgresML locally with our Docker image.
diff --git a/pgml-cms/docs/introduction/getting-started/connect-your-app.md b/pgml-cms/docs/introduction/getting-started/connect-your-app.md
@@ -4,16 +4,16 @@ description: PostgresML is compatible with all standard PostgreSQL clients
 
 # Connect your app
 
-You can connect to your database from any PostgresSQL-compatible client. PostgresML is intended to serve in the traditional role of an application database, along with it's extended role as an MLOps platform to make it easy to build and maintain AI applications together with your application data.
+You can connect to your PostgresML database from any PostgresSQL-compatible client. PostgresML can serve in the traditional role of an application database, along with it's extended role as an MLOps platform, to make it easy to build and maintain AI applications together with your application data.
 
-## SDK
+## Client SDK
 
-We provide a client SDK for JavaScript, Python and Rust. The SDK manages connections to the Postgres database and makes it easy to construct efficient queries for AI use cases, like managing a document collection for RAG, or building a chatbot. All of the ML & AI still happenening inside the database, with centralized operations, hardware and dependency management.
-
-The SDK are under rapid development to add new features and use cases, but we release non breaking changes with minor version updates in accordance with SemVer. It's easy to install into your existing application.
+We provide a client SDK for JavaScript, Python and Rust. The SDK manages connections to the database, and makes it easy to construct efficient queries for AI use cases, like managing RAG document collections, or building chatbots. All of the ML & AI still happens inside the database, with centralized operations, hardware and dependency management.
 
 ### Installation
 
+The SDK is available from npm and PyPI:
+
 {% tabs %}
 {% tab title="JavaScript" %}
 ```bash
@@ -32,6 +32,8 @@ Our SDK comes with zero additional dependencies. The core of the SDK is written
 
 ### Test the connection
 
+Once you have installed our SDK into your environment, you can test connectivity to our cloud with just a few lines of code:
+
 {% tabs %}
 {% tab title="JavaScript" %}
 ```javascript
@@ -82,9 +84,9 @@ async def main():
 {% endtab %}
 {% endtabs %}
 
-## Native language bindings
+## Native PostgreSQL libraries
 
-Using the SDK is optional. If you're comfortable with writing SQL, you can connect directly to the database using your favorite bindings or ORM:
+Using the SDK is completely optional. If you're comfortable with writing SQL, you can connect directly to the database using your favorite PostgreSQL client library or ORM:
 
 * C++: [libpqxx](https://www.tutorialspoint.com/postgresql/postgresql\_c\_cpp.htm)
 * C#: [Npgsql](https://github.com/npgsql/npgsql),[Dapper](https://github.com/DapperLib/Dapper), or [Entity Framework Core](https://github.com/dotnet/efcore)

diff --git a/pgml-cms/docs/introduction/getting-started/create-your-database.md b/pgml-cms/docs/introduction/getting-started/create-your-database.md
@@ -8,27 +8,25 @@ description: >-
 
 ## Sign up for an account
 
-Visit [https://postgresml.org/signup](https://postgresml.org/signup) to create a new account with your email, Google or Github authentication.
+Visit [https://postgresml.org/signup](https://postgresml.org/signup) to create a new account with your email, Google or GitHub.
 
 <div align="center" data-full-width="false">
-
-<figure><img src="../../.gitbook/assets/image (6).png" alt="Sign up" width="356"><figcaption></figcaption></figure>
-
+  <figure><img src="../../.gitbook/assets/image (6).png" alt="Sign up" width="356"><figcaption></figcaption></figure>
 </div>
 
 ## Select a plan <a href="#create-a-new-account" id="create-a-new-account"></a>
 
-Choose the type of GPU powered database deployment that is right for you.
+Choose the type of GPU powered database deployment that is right for you:
 
-* **Serverless** is the easiest way to get started. We offer a generous free tier with GPU access and 5GB of data storage.
-* **Dedicated** offers additional configuration options for more advanced use cases with established workloads and more predictable usage patterns.
+* **Serverless** is the easiest way to get started. We offer a generous free tier with GPU access and 5 GB of data storage
+* **Dedicated** offers additional configuration options for more advanced use cases with established workloads and more predictable usage patterns
 
 Click on **Get Started** under the plan of your choice.
 
 <figure><img src="../../.gitbook/assets/image (7).png" alt=""><figcaption></figcaption></figure>
 
-## Your database credentials  <a href="#create-a-new-account" id="create-a-new-account"></a>
+## Database access credentials  <a href="#create-a-new-account" id="create-a-new-account"></a>
 
-We'll automatically provision a set of database credentials and provide you with the `DATABASE_URL` connection string. You can connect to your database with `psql`, or any other PostgreSQL client library, or application.
+PostgresML Cloud automatically provisions database credentials and provides you with the `DATABASE_URL` connection string. You can connect to your database with `psql`, any other PostgreSQL client library, or application.
 
 <figure><img src="../../.gitbook/assets/Screenshot from 2023-11-27 23-21-36.png" alt=""><figcaption></figcaption></figure>
diff --git a/pgml-cms/docs/introduction/getting-started/import-your-data/README.md b/pgml-cms/docs/introduction/getting-started/import-your-data/README.md
@@ -6,13 +6,13 @@ Just like any PostgreSQL database, PostgresML can be configured as the primary a
 
 ## Primary database
 
-If you're intention is to use PostgresML as your primary database, your job here is done. You can use the connection credentials provided and start building your application on top of in-database AI right away.
+If your intention is to use PostgresML as your primary database, your job here is done. You can use the connection credentials provided and start building your application on top of in-database AI right away.
 
 ## [Logical replica](logical-replication/)
 
 If your primary database is hosted elsewhere, for example AWS RDS, or Azure Postgres, you can get your data replicated to PostgresML in real time using logical replication. 
 
-<figure><img src="../../../.gitbook/assets/logical_replication_1.png" alt=""><figcaption></figcaption></figure>
+<figure><img src="../../../.gitbook/assets/logical_replication_1.png" alt="Logical replication"><figcaption></figcaption></figure>
 
 Having access to your data immediately is very useful to
 accelerate your machine learning use cases and removes the need for moving data multiple times between microservices. Latency-sensitive applications should consider using this approach.
@@ -21,6 +21,14 @@ accelerate your machine learning use cases and removes the need for moving data
 
 Foreign data wrappers are a set of PostgreSQL extensions that allow making direct connections from inside the database directly to other databases, even if they aren't running on Postgres. For example, Postgres has foreign data wrappers for MySQL, S3, Snowflake and many others.
 
-<figure><img src="../../../.gitbook/assets/fdw_1.png" alt=""><figcaption></figcaption></figure>
+<figure><img src="../../../.gitbook/assets/fdw_1.png" alt="Foreign data wrappers"><figcaption></figcaption></figure>
 
 FDWs are useful when data access is infrequent and not latency-sensitive. For many use cases, like offline batch workloads and not very busy websites, this approach is suitable and easy to get started with.
+
+## [Move data with COPY](copy)
+
+`COPY` is a powerful PostgreSQL command to import data from a file format like CSV. Most data stores out there support exporting data using the CSV format, so moving data from your data source to PostgresML can almost always be done this way.
+
+## [Migrate with pg_dump](pg-dump)
+
+_pg_dump_ is a command-line PostgreSQL utility to migrate databases from one server to another. Databases of almost any size can be migrated with _pgdump_ quickly and safely.
diff --git a/pgml-cms/docs/introduction/getting-started/import-your-data/copy.md b/pgml-cms/docs/introduction/getting-started/import-your-data/copy.md
@@ -1,4 +1,4 @@
-# COPY
+# Move data with COPY
 
 Data that changes infrequently can be easily imported into PostgresML (and any other Postgres database) using `COPY`. All you have to do is export your data as a file, create a table in Postgres to store it, and import it using the command line (or your IDE of choice).
 

diff --git a/...ocs/introduction/getting-started/import-your-data/logical-replication/README.md b/...ocs/introduction/getting-started/import-your-data/logical-replication/README.md
@@ -77,19 +77,21 @@ Now that we have a list of tables we want to replicate, we need to make sure tho
 You can get the schema for your tables either by using a PostgreSQL client like pgAdmin or, more easily, by using _pg_dump_ and then importing it into PostgresML using _psql_:
 
 {% tabs %}
-{% tab title="Dumping the schema" %}
+{% tab title="Export the schema" %}
 
 ```bash
 pg_dump \
 	postgres://user:password@yyour-production-db.amazonaws.com:5432/prodution_db \
 	--schema-only \
+	--no-owner \
+	--no-privileges \
 	-t users \
 	-t blog_posts \
 > schema.sql
 ```
 
 {% endtab %}
-{% tab title="Importing the schema" %}
+{% tab title="Import the schema" %}
 
 ```bash
 psql \

diff --git a/pgml-cms/docs/introduction/getting-started/import-your-data/pg-dump.md b/pgml-cms/docs/introduction/getting-started/import-your-data/pg-dump.md
@@ -0,0 +1,45 @@
+# Migrate with pg_dump
+
+_pg_dump_ is a command-line PostgreSQL tool that can move data between PostgreSQL databases. If you're planning a migration from your database to PostgresML, _pg_dump_ is a good tool to get you going quickly.
+
+## Getting started
+
+If your database is reasonably small (10 GB or less), you can just run _pg_dump_ in one comand:
+
+{% tabs %}
+{% tab title="pg_dump" %}
+
+```bash
+pg_dump \
+	--no-owner \
+	--clean \
+	--no-privileges \
+  postgres://user:password@your-production-database.amazonaws.com/production_db | \
+psql postgres://user:password@sql.cloud.postgresml.org:6432/your_pgml_db
+```
+
+{% endtab %}
+{% endtabs %}
+
+This will take a few minutes, and once the command completes, all your data, including indexes, will be in your PostgresML database.
+
+## Migrating one table at a time
+
+If your database is larger, you can split the migration into multiple steps, migrating one or more tables at a time.
+
+{% tabs %}
+{% tab title="pg_dump" %}
+
+```bash
+pg_dump \
+	--no-owner \
+	--clean \
+	--no-privileges \
+	-t users \
+	-t orders \
+  postgres://user:password@your-production-database.amazonaws.com/production_db | \
+psql postgres://user:password@sql.cloud.postgresml.org:6432/your_pgml_db
+```
+
+{% endtab %}
+{% endtabs %}
diff --git a/pgml-cms/docs/product/cloud-database/README.md b/pgml-cms/docs/product/cloud-database/README.md
@@ -1,6 +1,6 @@
-# Cloud Database
+# AI database
 
-PostgresML Cloud databases can be deployed using three (3) configurations: serverless, dedicated and enterprise. Each has its advantages and are tailored for companies of all sizes.
+PostgresML cloud databases can be deployed using three (3) configurations: serverless, dedicated and enterprise. Each has its advantages and are tailored for companies of all sizes.
 
 <figure><img src="../../.gitbook/assets/image (5).png" alt=""><figcaption><p>Plans available on PostgresML Cloud</p></figcaption></figure>
 

diff --git a/pgml-dashboard/src/api/cms.rs b/pgml-dashboard/src/api/cms.rs
@@ -423,7 +423,7 @@ impl Collection {
         // Docs gets a home link added to the index
         match self.name.as_str() {
             "Docs" => {
-                index.push(IndexLink::new("Docs Home", indent_level).href("/docs"));
+                index.push(IndexLink::new("Documentation", indent_level).href("/docs"));
             }
             _ => {}
         }

diff --git a/pgml-dashboard/src/utils/markdown.rs b/pgml-dashboard/src/utils/markdown.rs
@@ -514,20 +514,20 @@ pub fn get_toc<'a>(root: &'a AstNode<'a>) -> anyhow::Result<Vec<TocLink>> {
                     }
                 };
 
-                println!("child: {:?}", sibling);
-
                 let text = if let NodeValue::Text(text) = &sibling.data.borrow().value {
                     Some(text.clone())
                 } else if let NodeValue::Link(_link) = &sibling.data.borrow().value {
-                    if let Some(child) = sibling.first_child() {
-                        if let NodeValue::Text(text) = &child.data.borrow().value {
-                                Some(text.clone())
-                        } else {
-                            None
-                        }
-                    } else {
-                        None
-                    }
+                    let text = sibling.children()
+                        .into_iter()
+                        .map(|child| {
+                            if let NodeValue::Text(text) = &child.data.borrow().value {
+                                text.clone()
+                            } else {
+                                "".to_string()
+                            }
+                        })
+                        .join("");
+                    Some(text)
                 } else {
                     None
                 };
-Original file line number
+Diff line change
@@ Expand Up @@
     <figure><img src=".gitbook/assets/architecture.png" alt="PostgresML architectural diagram"><figcaption></figcaption></figure>
-    ## SDK
+    ## Client SDK
     The PostgresML team also provides [native language SDKs](https://github.com/postgresml/postgresml/tree/master/pgml-sdks/pgml) which implement best practices for common ML & AI applications. The JavaScript and Python SDKs are generated from the a core Rust library, which provides a uniform API, correctness and efficiency across all environments.
@@ Expand Down @@