Merge branch 'master' into dan-qa-fixes

postgresml · chillenberger · Oct 12, 2023 · Oct 11, 2023 · Oct 12, 2023 · Oct 12, 2023
commit ee2a01259a1d9daa4d395ff3170d523604006993
diff --git a/README.md b/README.md
@@ -138,6 +138,10 @@ SELECT pgml.predict(
 # Installation
 PostgresML installation consists of three parts: PostgreSQL database, Postgres extension for machine learning and a dashboard app. The extension provides all the machine learning functionality and can be used independently using any SQL IDE. The dashboard app provides an easy to use interface for writing SQL notebooks, performing and tracking ML experiments and ML models.
 
+## Serverless Cloud
+
+If you want to check out the functionality without the hassle of Docker, [sign up for a free PostgresML account](https://postgresml.org/signup). You'll get a free database in seconds, with access to GPUs and state of the art LLMs.
+
 ## Docker
 
 ```
@@ -150,19 +154,14 @@ docker run \
     sudo -u postgresml psql -d postgresml
 ```
 
-For more details, take a look at our [Quick Start with Docker](https://postgresml.org/docs/guides/setup/quick_start_with_docker) documentation.
-
-## Serverless Cloud
-
-If you want to check out the functionality without the hassle of Docker, [sign up for a free PostgresML account](https://postgresml.org/signup). You'll get a free database in seconds, with access to GPUs and state of the art LLMs.
+For more details, take a look at our [Quick Start with Docker](https://postgresml.org/docs/guides/developer-docs/quick-start-with-docker) documentation.
 
 # Getting Started
 
 ## Option 1
 
-- On local installation, go to dashboard app at `http://localhost:8000/` to use SQL notebooks.
-
 - On the cloud console click on the **Dashboard** button to connect to your instance with a SQL notebook, or connect directly with tools listed below.
+- On local installation, go to dashboard app at `http://localhost:8000/` to use SQL notebooks.
 
 ## Option 2
 

diff --git a/docker/dashboard.sh b/docker/dashboard.sh
@@ -4,6 +4,7 @@ set -e
 export DATABASE_URL=postgres://postgresml:postgresml@127.0.0.1:5432/postgresml
 export DASHBOARD_STATIC_DIRECTORY=/usr/share/pgml-dashboard/dashboard-static
 export DASHBOARD_CONTENT_DIRECTORY=/usr/share/pgml-dashboard/dashboard-content
+export DASHBOARD_DOCS_DIRECTORY=/usr/share/pgml-docs
 export SEARCH_INDEX_DIRECTORY=/var/lib/pgml-dashboard/search-index
 export ROCKET_SECRET_KEY=$(openssl rand -hex 32)
 export ROCKET_ADDRESS=0.0.0.0

diff --git a/packages/postgresml-dashboard/build.sh b/packages/postgresml-dashboard/build.sh
@@ -25,7 +25,8 @@ rm "$deb_dir/release.sh"
   cargo build --release && \
   cp target/release/pgml-dashboard "$deb_dir/usr/bin/pgml-dashboard" && \
   cp -R content "$deb_dir/usr/share/pgml-dashboard/dashboard-content" && \
-  cp -R static "$deb_dir/usr/share/pgml-dashboard/dashboard-static" )
+  cp -R static "$deb_dir/usr/share/pgml-dashboard/dashboard-static" && \
+  cp -R ../pgml-docs "$deb_dir/usr/share/pgml-docs" )
 
 (cat ${SCRIPT_DIR}/DEBIAN/control | envsubst) > "$deb_dir/DEBIAN/control"
 (cat ${SCRIPT_DIR}/etc/systemd/system/pgml-dashboard.service | envsubst) > "$deb_dir/etc/systemd/system/pgml-dashboard.service"

diff --git a/packages/postgresml-dashboard/etc/systemd/system/pgml-dashboard.service b/packages/postgresml-dashboard/etc/systemd/system/pgml-dashboard.service
@@ -7,6 +7,7 @@ StartLimitIntervalSec=0
 Environment=RUST_LOG=info
 Environment=DASHBOARD_STATIC_DIRECTORY=/usr/share/pgml-dashboard/dashboard-static
 Environment=DASHBOARD_CONTENT_DIRECTORY=/usr/share/pgml-dashboard/dashboard-content
+Environment=DASHBOARD_DOCS_DIRECTORY=/usr/share/pgml-docs
 Environment=ROCKET_ADDRESS=0.0.0.0
 Environment=GITHUB_STARS=${GITHUB_STARS}
 Environment=SEARCH_INDEX_DIRECTORY=/var/lib/pgml-dashboard/search-index

diff --git a/pgml-dashboard/.env.development b/pgml-dashboard/.env.development
@@ -1,2 +1,3 @@
 DATABASE_URL=postgres:///pgml_dashboard_development
 DEV_MODE=true
+RUST_LOG=debug,tantivy=error,rocket=info
diff --git a/pgml-dashboard/Cargo.lock b/pgml-dashboard/Cargo.lock
diff --git a/pgml-dashboard/Cargo.toml b/pgml-dashboard/Cargo.toml
@@ -17,12 +17,14 @@ base64 = "0.21"
 comrak = "0.17"
 chrono = "0.4"
 csv-async = "1"
+convert_case = "0.6"
 dotenv = "0.15"
 env_logger = "0.10"
 itertools = "0.10"
 parking_lot = "0.12"
 lazy_static = "1.4"
 log = "0.4"
+markdown = "1.0.0-alpha.13"
 num-traits = "0.2"
 once_cell = "1.18"
 rand = "0.8"
@@ -39,6 +41,7 @@ sqlx = { version = "0.6.3", features = [ "runtime-tokio-rustls", "postgres", "js
 tantivy = "0.19"
 time = "0.3"
 tokio = { version = "1", features = ["full"] }
+url = "2.4"
 yaml-rust = "0.4"
 zoomies = { git="https://github.com/HyperparamAI/zoomies.git", branch="master" }
 pgvector = { version = "0.2.2", features = [ "sqlx", "postgres" ] }

diff --git a/pgml-dashboard/README.md b/pgml-dashboard/README.md
@@ -2,4 +2,4 @@
 
 PostgresML provides a dashboard with analytical views of the training data and model performance, as well as integrated notebooks for rapid iteration. It is primarily written in Rust using [Rocket](https://rocket.rs/) as a lightweight web framework and [SQLx](https://github.com/launchbadge/sqlx) to interact with the database.
 
-Please see the [quick start instructions](https://postgresml.org/user_guides/setup/quick_start_with_docker/) for general information on installing or deploying PostgresML. A [developer guide](https://postgresml.org/developer_guide/overview/) is also available for those who would like to contribute.
+Please see the [quick start instructions](https://postgresml.org/docs/guides/getting-started/sign-up) for general information on installing or deploying PostgresML. A [developer guide](https://postgresml.org/developer_guide/overview/) is also available for those who would like to contribute.
diff --git a/pgml-dashboard/content/docs/guides/dashboard/overview.md b/pgml-dashboard/content/docs/guides/dashboard/overview.md
@@ -1,6 +1,6 @@
 # Dashboard
 
-PostgresML comes with a web app to provide visibility into models and datasets in your database. If you're running [our Docker container](/docs/guides/setup/quick_start_with_docker/), you can view it running on [http://localhost:8000/](http://localhost:8000/).
+PostgresML comes with a web app to provide visibility into models and datasets in your database. If you're running [our Docker container](/docs/guides/developer-docs/quick-start-with-docker), you can view it running on [http://localhost:8000/](http://localhost:8000/).
 
 
 ## Generate example data

diff --git a/pgml-dashboard/content/docs/guides/setup/distributed_training.md b/pgml-dashboard/content/docs/guides/setup/distributed_training.md
@@ -22,7 +22,7 @@ psql \
     -f dump.sql
 ```
 
-If you're using our <a href="/docs/guides/setup/quick_start_with_docker">Docker</a> stack, you can import the data there:</p>
+If you're using our <a href="/docs/guides/developer-docs/quick-start-with-docker">Docker</a> stack, you can import the data there:</p>
 
 ```
 psql \

diff --git a/pgml-dashboard/content/docs/guides/setup/v2/installation.md b/pgml-dashboard/content/docs/guides/setup/v2/installation.md
@@ -10,7 +10,7 @@ The extension can be installed by compiling it from source, or if you're using U
 
 !!! tip
 
-If you're just looking to try PostgresML without installing it on your system, take a look at our [Quick Start with Docker](/docs/guides/setup/quick_start_with_docker) guide.
+If you're just looking to try PostgresML without installing it on your system, take a look at our [Quick Start with Docker](/docs/guides/developer-docs/quick-start-with-docker) guide.
 
 !!!
 

diff --git a/pgml-dashboard/src/api/docs.rs b/pgml-dashboard/src/api/docs.rs
@@ -24,54 +24,43 @@ async fn search(query: &str, index: &State<markdown::SearchIndex>) -> ResponseOk
     )
 }
 
-#[get("/docs/<path..>", rank = 10)]
-async fn doc_handler<'a>(path: PathBuf, cluster: &Cluster) -> Result<ResponseOk, Status> {
-    let guides = vec![
-        NavLink::new("Setup").children(vec![
-            NavLink::new("Installation").children(vec![
-                NavLink::new("v2").href("/docs/guides/setup/v2/installation"),
-                NavLink::new("Upgrade from v1.0 to v2.0")
-                    .href("/docs/guides/setup/v2/upgrade-from-v1"),
-                NavLink::new("v1").href("/docs/guides/setup/installation"),
-            ]),
-            NavLink::new("Quick Start with Docker")
-                .href("/docs/guides/setup/quick_start_with_docker"),
-            NavLink::new("Distributed Training").href("/docs/guides/setup/distributed_training"),
-            NavLink::new("GPU Support").href("/docs/guides/setup/gpu_support"),
-            NavLink::new("Developer Setup").href("/docs/guides/setup/developers"),
-        ]),
-        NavLink::new("Training").children(vec![
-            NavLink::new("Overview").href("/docs/guides/training/overview"),
-            NavLink::new("Algorithm Selection").href("/docs/guides/training/algorithm_selection"),
-            NavLink::new("Hyperparameter Search")
-                .href("/docs/guides/training/hyperparameter_search"),
-            NavLink::new("Preprocessing Data").href("/docs/guides/training/preprocessing"),
-            NavLink::new("Joint Optimization").href("/docs/guides/training/joint_optimization"),
-        ]),
-        NavLink::new("Predictions").children(vec![
-            NavLink::new("Overview").href("/docs/guides/predictions/overview"),
-            NavLink::new("Deployments").href("/docs/guides/predictions/deployments"),
-            NavLink::new("Batch Predictions").href("/docs/guides/predictions/batch"),
-        ]),
-        NavLink::new("Transformers").children(vec![
-            NavLink::new("Setup").href("/docs/guides/transformers/setup"),
-            NavLink::new("Pre-trained Models").href("/docs/guides/transformers/pre_trained_models"),
-            NavLink::new("Fine Tuning").href("/docs/guides/transformers/fine_tuning"),
-            NavLink::new("Embeddings").href("/docs/guides/transformers/embeddings"),
-        ]),
-        NavLink::new("Vector Operations").children(vec![
-            NavLink::new("Overview").href("/docs/guides/vector_operations/overview")
-        ]),
-        NavLink::new("Dashboard").href("/docs/guides/dashboard/overview"),
-        NavLink::new("Schema").children(vec![
-            NavLink::new("Models").href("/docs/guides/schema/models"),
-            NavLink::new("Snapshots").href("/docs/guides/schema/snapshots"),
-            NavLink::new("Projects").href("/docs/guides/schema/projects"),
-            NavLink::new("Deployments").href("/docs/guides/schema/deployments"),
-        ]),
-    ];
-
-    render(cluster, &path, guides, "Guides", &Path::new("docs")).await
+use rocket::fs::NamedFile;
+
+#[get("/docs/guides/.gitbook/assets/<path>", rank = 10)]
+pub async fn gitbook_assets(path: PathBuf) -> Option<NamedFile> {
+    let path = PathBuf::from(&config::docs_dir())
+        .join("docs/guides/.gitbook/assets/")
+        .join(path);
+
+    NamedFile::open(path).await.ok()
+}
+
+#[get("/docs/<path..>", rank = 5)]
+async fn doc_handler(path: PathBuf, cluster: &Cluster) -> Result<ResponseOk, Status> {
+    let root = PathBuf::from("docs/guides/");
+    let index_path = PathBuf::from(&config::docs_dir())
+        .join(&root)
+        .join("SUMMARY.md");
+    let contents = tokio::fs::read_to_string(&index_path).await.expect(
+        format!(
+            "could not read table of contents markdown: {:?}",
+            index_path
+        )
+        .as_str(),
+    );
+    let mdast = ::markdown::to_mdast(&contents, &::markdown::ParseOptions::default())
+        .expect("could not parse table of contents markdown");
+    let guides = markdown::parse_summary_into_nav_links(&mdast)
+        .expect("could not extract nav links from table of contents");
+    render(
+        cluster,
+        &path,
+        guides,
+        "Guides",
+        &Path::new("docs"),
+        &config::docs_dir(),
+    )
+    .await
 }
 
 #[get("/blog/<path..>", rank = 10)]
@@ -134,6 +123,7 @@ async fn blog_handler<'a>(path: PathBuf, cluster: &Cluster) -> Result<ResponseOk
         ],
         "Blog",
         &Path::new("blog"),
+        &config::blogs_dir(),
     )
     .await
 }
@@ -144,18 +134,32 @@ async fn render<'a>(
     mut nav_links: Vec<NavLink>,
     nav_title: &'a str,
     folder: &'a Path,
+    content: &'a str,
 ) -> Result<ResponseOk, Status> {
+    let mut path = path
+        .to_str()
+        .expect("path must convert to a string")
+        .to_string();
     let url = path.clone();
+    if path.ends_with("/") {
+        path.push_str("README");
+    }
 
     // Get the document content
-    let path = Path::new(&config::content_dir())
+    let path = Path::new(&content)
         .join(folder)
-        .join(&(path.to_str().unwrap().to_string() + ".md"));
+        .join(&(path.to_string() + ".md"));
 
     // Read to string
     let contents = match tokio::fs::read_to_string(&path).await {
-        Ok(contents) => contents,
-        Err(_) => return Err(Status::NotFound),
+        Ok(contents) => {
+            info!("loading markdown file: '{:?}", path);
+            contents
+        }
+        Err(err) => {
+            warn!("Error parsing markdown file: '{:?}' {:?}", path, err);
+            return Err(Status::NotFound);
+        }
     };
     let parts = contents.split("---").collect::<Vec<&str>>();
     let ((image, description), contents) = if parts.len() > 1 {
@@ -214,7 +218,7 @@ async fn render<'a>(
 
     // Handle navigation
     for nav_link in nav_links.iter_mut() {
-        nav_link.should_open(&url.to_str().unwrap().to_string());
+        nav_link.should_open(&url);
     }
 
     let user = if cluster.context.user.is_anonymous() {
@@ -244,7 +248,7 @@ async fn render<'a>(
 }
 
 pub fn routes() -> Vec<Route> {
-    routes![doc_handler, blog_handler, search]
+    routes![gitbook_assets, doc_handler, blog_handler, search]
 }
 
 #[cfg(test)]

diff --git a/pgml-dashboard/src/components/nav/template.html b/pgml-dashboard/src/components/nav/template.html
@@ -26,14 +26,14 @@
 
       <% if link.nav.is_some() { %>
       <ul class="nav flex-column justify-content-end flex-grow-1 ps-5">
-  
+
         <% for link in link.nav.unwrap().links.iter() {
           let active = if link.active {
             "active"
           } else {
             ""
           };
-  
+
           let aria = if link.active {
             r#"area-current="page""#
           } else {
@@ -48,10 +48,10 @@
             <span class="nav-link-name"><%= link.name %></span>
           </a>
         </li>
-  
+
         <% } %>
       </ul>
-  
+
       <% } %>
     </li>
     <% } %>

diff --git a/pgml-dashboard/src/components/navigation/navbar/marketing/template.html b/pgml-dashboard/src/components/navigation/navbar/marketing/template.html
@@ -14,7 +14,7 @@
         <div class="collapse navbar-collapse" id="navbarSupportedContent">
           <ul class="navbar-nav flex-grow-1 gap-4 me-auto mb-4 mb-lg-0">
             <li class="nav-item d-flex align-items-center">
-              <a class="nav-link p-0" href="/docs/guides/setup/quick_start_with_docker">Docs</a>
+              <a class="nav-link p-0" href="/docs/guides/developer-docs/quick-start-with-docker">Docs</a>
             </li>
             <% if !standalone_dashboard { %>
             <!-- <li class="nav-item d-flex align-items-center">

diff --git a/pgml-dashboard/src/components/navigation/navbar/web_app/template.html b/pgml-dashboard/src/components/navigation/navbar/web_app/template.html
@@ -50,7 +50,7 @@
             </li>
 
             <li class="nav-item d-flex align-items-center">
-              <a class="nav-link p-lg-0" href="/docs/guides/setup/quick_start_with_docker">Docs</a>
+              <a class="nav-link p-lg-0" href="/docs/guides/">Docs</a>
             </li>
 
             <div class="vr my-2 opacity-100 d-lg-block d-none" style="width: 2px"></div>
@@ -80,7 +80,7 @@
             </li>
 
             <li class="menu-item rounded-0 d-flex align-items-center">
-              <a href="/docs/guides/setup/quick_start_with_docker/">Docs</a>
+              <a href="/docs/guides//">Docs</a>
             </li>
 
             <li class="menu-item rounded-0 d-flex align-items-center">
Original file line number	Diff line number	Diff line change
Expand Up		@@ -2,4 +2,4 @@

		PostgresML provides a dashboard with analytical views of the training data and model performance, as well as integrated notebooks for rapid iteration. It is primarily written in Rust using [Rocket](https://rocket.rs/) as a lightweight web framework and [SQLx](https://github.com/launchbadge/sqlx) to interact with the database.

		Please see the [quick start instructions](https://postgresml.org/user_guides/setup/quick_start_with_docker/) for general information on installing or deploying PostgresML. A [developer guide](https://postgresml.org/developer_guide/overview/) is also available for those who would like to contribute.
		Please see the [quick start instructions](https://postgresml.org/docs/guides/getting-started/sign-up) for general information on installing or deploying PostgresML. A [developer guide](https://postgresml.org/developer_guide/overview/) is also available for those who would like to contribute.
-Original file line number
+Diff line change
@@ Expand Up / @@ -22,7 +22,7 @@ psql \ @@
         -f dump.sql
     ```
-    If you're using our <a href="/docs/guides/setup/quick_start_with_docker">Docker</a> stack, you can import the data there:</p>
+    If you're using our <a href="/docs/guides/developer-docs/quick-start-with-docker">Docker</a> stack, you can import the data there:</p>
     ```
     psql \
@@ Expand Down @@