Data docs #1418
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| .terraform | ||
| *.lock.hcl | ||
| *.tfstate | ||
| *.tfstate.backup |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| # Terraform configuration for pgml-rds-proxy on EC2 | ||
|
|
||
| This is a sample Terraform deployment for running pgml-rds-proxy on EC2. This will spin up an EC2 instance | ||
| with a public IP and a working security group & install the community Docker runtime. | ||
|
|
||
| Once the instance is running, you can connect to it using the root key and run the pgml-rds-proxy Docker container | ||
| with the correct PostgresML `DATABASE_URL`. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,84 @@ | ||
| terraform { | ||
| required_providers { | ||
| aws = { | ||
| source = "hashicorp/aws" | ||
| version = "~> 5.46" | ||
| } | ||
| } | ||
|
|
||
| required_version = ">= 1.2.0" | ||
| } | ||
|
|
||
| provider "aws" { | ||
| region = "us-west-2" | ||
| } | ||
|
|
||
| data "aws_ami" "ubuntu" { | ||
| most_recent = true | ||
|
|
||
| filter { | ||
| name = "name" | ||
| values = ["ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-*"] | ||
| } | ||
|
|
||
| filter { | ||
| name = "virtualization-type" | ||
| values = ["hvm"] | ||
| } | ||
|
|
||
| owners = ["099720109477"] # Canonical | ||
| } | ||
|
|
||
| resource "aws_security_group" "pgml-rds-proxy" { | ||
| egress { | ||
| from_port = 0 | ||
| to_port = 0 | ||
| protocol = "-1" | ||
| cidr_blocks = ["0.0.0.0/0"] | ||
| ipv6_cidr_blocks = ["::/0"] | ||
| } | ||
|
|
||
| ingress { | ||
| from_port = 6432 | ||
| to_port = 6432 | ||
| protocol = "tcp" | ||
| cidr_blocks = ["0.0.0.0/0"] | ||
| ipv6_cidr_blocks = ["::/0"] | ||
| } | ||
|
|
||
| ingress { | ||
| from_port = 22 | ||
| to_port = 22 | ||
| protocol = "tcp" | ||
| cidr_blocks = ["0.0.0.0/0"] | ||
| ipv6_cidr_blocks = ["::/0"] | ||
| } | ||
| } | ||
|
|
||
| resource "aws_instance" "pgml-rds-proxy" { | ||
| ami = data.aws_ami.ubuntu.id | ||
| instance_type = "t3.micro" | ||
| key_name = var.root_key | ||
|
|
||
| root_block_device { | ||
| volume_size = 30 | ||
| delete_on_termination = true | ||
| } | ||
|
|
||
| vpc_security_group_ids = [ | ||
| "${aws_security_group.pgml-rds-proxy.id}", | ||
| ] | ||
|
|
||
| associate_public_ip_address = true | ||
| user_data = file("${path.module}/user_data.sh") | ||
| user_data_replace_on_change = false | ||
|
|
||
| tags = { | ||
| Name = "pgml-rds-proxy" | ||
| } | ||
| } | ||
|
|
||
| variable "root_key" { | ||
| type = string | ||
| description = "The name of the SSH Root Key you'd like to assign to this EC2 instance. Make sure it's a key you have access to." | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| #!/bin/bash | ||
| # | ||
| # Cloud init script to install Docker on an EC2 instance running Ubuntu 22.04. | ||
| # | ||
|
|
||
| sudo apt-get update | ||
| sudo apt-get install ca-certificates curl | ||
| sudo install -m 0755 -d /etc/apt/keyrings | ||
| sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc | ||
| sudo chmod a+r /etc/apt/keyrings/docker.asc | ||
|
|
||
| # Add the repository to Apt sources: | ||
| echo \ | ||
| "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \ | ||
| $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \ | ||
| sudo tee /etc/apt/sources.list.d/docker.list > /dev/null | ||
| sudo apt-get update | ||
|
|
||
| sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin | ||
| sudo groupadd docker | ||
| sudo usermod -aG docker ubuntu |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| *.md.bak |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -4,11 +4,11 @@ description: Setup a database and connect your application to PostgresML | |||||
|
|
||||||
| # Getting Started | ||||||
|
|
||||||
| A PostgresML deployment consists of multiple components working in concert to provide a complete Machine Learning platform. We provide a fully managed solution in our cloud, and document a self-hosted installation in our docs. | ||||||
|
||||||
|
|
||||||
| * PostgreSQL database, with `pgml`, `pgvector` and many other extensions installed, including backups, metrics, logs, replicas and high availability | ||||||
| * PgCat pooler to provide secure access and model load balancing across thousands of clients | ||||||
| * A web application to manage deployed models and write experiments in SQL notebooks | ||||||
|
||||||
| * A web application to manage deployed models and write experiments in SQL notebooks | |
| * A web application to manage deployed models and share experiments and analysis in SQL notebooks |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -4,13 +4,13 @@ description: PostgresML is compatible with all standard PostgreSQL clients | |||||
|
|
||||||
| # Connect your app | ||||||
|
|
||||||
| You can connect to your database from any PostgresSQL-compatible client. PostgresML is intended to serve in the traditional role of an application database, along with it's extended role as an MLOps platform to make it easy to build and maintain AI applications together with your application data. | ||||||
|
|
||||||
| ## SDK | ||||||
|
||||||
| ## SDK | |
| ## Client SDKs |
We've waffled between singular and plural. If we want to change to singular, we'll need to update in several other places, including urls.
Contributor
Author
There was a problem hiding this comment.
Not going to write Client SDKs ever again, that wording is just wrong. SDK are only usable on the client by definition.
Contributor
Author
There was a problem hiding this comment.
It's going to be singular going forward.
Contributor
There was a problem hiding this comment.
New PRs should use the existing style guide, for code or copy. We should make that update globally in a separate PR to preserve consistency.
Dealing with the separate issue, many users frequently ask about the difference between the SDK and Extension, and how they interact. I think you're confused about the difference between and SDK and API as commonly used, but those terms are not always 100% consistently used in industry, so we need some additional education about how we're using them.
Outdated
Contributor
There was a problem hiding this comment.
Suggested change
| We provide a client SDK for JavaScript, Python and Rust. The SDK manages connections to the Postgres database and makes it easy to construct efficient queries for AI use cases, like managing a document collection for RAG, or building a chatbot. All of the ML & AI still happenening inside the database, with centralized operations, hardware and dependency management. | |
| We provide a client SDK for JavaScript, Python and Rust. The SDK manages connections to the Postgres database and makes it easy to construct efficient queries for AI use cases, like managing a document collection for RAG, or building a chatbot. All of the ML & AI still happens inside the database, with centralized operations, hardware and dependency management. |
Outdated
Contributor
There was a problem hiding this comment.
Suggested change
| The SDK are under rapid development to add new features and use cases, but we release non breaking changes with minor version updates in accordance with SemVer. It's easy to install into your existing application. | |
| The SDKs are under rapid development to add new features and use cases, but we release non breaking changes with minor version updates in accordance with SemVer. It's easy to install into your existing application. |
Contributor
There was a problem hiding this comment.
Suggested change
| Our SDK comes with zero additional dependencies. The core of the SDK is written in Rust, and we provide language bindings and native packaging & distribution. | |
| Our SDK comes with zero additional dependencies, to provide the simplest and safest ML application deployment and maintenance possible. The core of the SDK is written in Rust, and we provide language bindings and native packaging & distribution. |
Contributor
There was a problem hiding this comment.
Suggested change
| If you need to write ad-hoc queries, you can use any of these popular tools to execute SQL queries directly on your database: | |
| If you need to write ad-hoc queries, or perform administrative functions, you can use any of these popular tools to execute SQL queries directly on your database: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -29,6 +29,6 @@ Click on **Get Started** under the plan of your choice. | |
|
|
||
| ## Your database credentials <a href="#create-a-new-account" id="create-a-new-account"></a> | ||
|
|
||
| We'll automatically provision a set of database credentials and provide you with the `DATABASE_URL` connection string. You can connect to your database with `psql`, or any other PostgreSQL client library, or application. | ||
|
|
||
| <figure><img src="../../.gitbook/assets/Screenshot from 2023-11-27 23-21-36.png" alt=""><figcaption></figcaption></figure> | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -1,22 +1,26 @@ | ||||||
| # Import your data | ||||||
|
|
||||||
| AI needs data, whether it's generating text with LLMs, creating embeddings, or training regression or classification models on customer data. | ||||||
|
|
||||||
| Just like any PostgreSQL database, PostgresML can be configured as the primary application database, a logical replica of your primary database, or with foreign data wrappers to query your primary database on demand. Depending on how frequently your data changes and your latency requirements, one approach is better than the other. | ||||||
|
|
||||||
| ## Primary database | ||||||
|
|
||||||
| If you're intention is to use PostgresML as your primary database, your job here is done. You can use the connection credentials provided and start building your application on top of in-database AI right away. | ||||||
|
||||||
| If you're intention is to use PostgresML as your primary database, your job here is done. You can use the connection credentials provided and start building your application on top of in-database AI right away. | |
| If your intention is to use PostgresML as your primary database, your job here is done. You can use the connection credentials provided and start building your application on top of in-database AI right away. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,75 @@ | ||
| # COPY | ||
|
|
||
| Data that changes infrequently can be easily imported into PostgresML (and any other Postgres database) using `COPY`. All you have to do is export your data as a file, create a table in Postgres to store it, and import it using the command line (or your IDE of choice). | ||
|
|
||
| ## Getting started | ||
|
|
||
| We'll be using CSV as our data format of choice. CSV is a supported mechanism for data transport in pretty much every database and system in existence, so you won't have any trouble finding the CSV export functionality in your current data store. | ||
|
|
||
| Let's use a simple CSV file with 3 columns as an example: | ||
|
|
||
| | Column | Data type | Example data | | ||
| | ---------------- | --------- | ------- | | ||
| | name | text | John | | ||
| | age | integer | 30 | | ||
| | is\_paying\_user | boolean | true | | ||
|
|
||
| ### Export data | ||
|
|
||
| If you're using a Postgres database already, you can export any table as CSV with just one command: | ||
|
|
||
| ```bash | ||
| psql \ | ||
| postgres://user:password@your-production-db.amazonaws.com \ | ||
| -c "\copy (SELECT * FROM users) TO '~/users.csv' CSV HEADER" | ||
| ``` | ||
|
|
||
| If you're using another data store, it will almost always provide a CSV export functionality. | ||
|
|
||
| ### Create table in PostgresML | ||
|
|
||
| Create a table in PostgresML with the correct schema: | ||
|
|
||
|
|
||
| {% tabs %} | ||
| {% tab title="SQL" %} | ||
|
|
||
| ```postgresql | ||
| CREATE TABLE users( | ||
| name TEXT, | ||
| age INTEGER, | ||
| is_paying_user BOOLEAN | ||
| ); | ||
| ``` | ||
|
|
||
| {% endtab %} | ||
| {% tab title="Output" %} | ||
|
|
||
| ``` | ||
| CREATE TABLE | ||
| ``` | ||
|
|
||
| {% endtab %} | ||
| {% endtabs %} | ||
|
|
||
| Data types should roughly match to what you have in your CSV file. If the data type is not known, you can always use `TEXT` and figure out what it is later with a few queries. Postgres also supports converting data types, as long as they are formatted correctly. | ||
|
|
||
| ### Import data | ||
|
|
||
| Once you have a table and your data exported as CSV, importing it can also be done with just one command: | ||
|
|
||
| ```bash | ||
| psql \ | ||
| postgres://user:password@sql.cloud.postgresml.org/your_pgml_database \ | ||
| -c "\copy your_table FROM '~/your_table.csv' CSV HEADER" | ||
| ``` | ||
|
|
||
| We took our export command and changed `TO` to `FROM`, and that's it. Make sure you're connecting to your PostgresML database when importing data. | ||
|
|
||
| ## Refresh data | ||
|
|
||
| If your data changed, repeat this process again. To avoid duplicate entries in your table, you can truncate (or delete) all rows beforehand: | ||
|
|
||
| ```sql | ||
| TRUNCATE your_table; | ||
| ``` |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is Cloud as opposed to Vector section