🌐 AI搜索 & 代理 主页
Skip to content

Working chatbot streaming #1228

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
SilasMarvin merged 4 commits into from
Jan 3, 2024
Merged

Working chatbot streaming #1228

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
fda50c0
Updated api to stream
SilasMarvin Dec 5, 2023
668cafb
Working chatbot streaming
SilasMarvin Dec 8, 2023
aa82b6d
Added images and finalized models for chatbot
SilasMarvin Dec 14, 2023
685cc85
Take advantage of the new quantized_config option
SilasMarvin Jan 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Take advantage of the new quantized_config option
  • Loading branch information
@SilasMarvin
SilasMarvin committed Jan 3, 2024
commit 685cc854f69dcb042f245c53362995e342df9435
19 changes: 15 additions & 4 deletions pgml-dashboard/src/api/chatbot.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,17 +102,29 @@ impl ChatbotBrain {
Self::TekniumOpenHermes25Mistral7B => serde_json::json!({
"model": "TheBloke/OpenHermes-2.5-Mistral-7B-GPTQ",
"revision": "main",
"device_map": "auto"
"device_map": "auto",
"quantization_config": {
"bits": 4,
"max_input_length": 10000
}
}),
Self::GrypheMythoMaxL213b => serde_json::json!({
"model": "TheBloke/MythoMax-L2-13B-GPTQ",
"revision": "main",
"device_map": "auto"
"device_map": "auto",
"quantization_config": {
"bits": 4,
"max_input_length": 10000
}
}),
Self::Starling7b => serde_json::json!({
"model": "TheBloke/Starling-LM-7B-alpha-GPTQ",
"revision": "main",
"device_map": "auto"
"device_map": "auto",
"quantization_config": {
"bits": 4,
"max_input_length": 10000
}
}),
_ => unimplemented!(),
}
Expand Down Expand Up @@ -537,7 +549,6 @@ async fn process_message(
.map(|(_, context, metadata)| format!("\n\n#### Document {}: \n{}\n\n", metadata["id"], context))
.collect::<Vec<String>>()
.join("\n");
// let context = "".to_string();

let history_collection = Collection::new(
"ChatHistory",
Expand Down