Projects

Overview: This repository contains code samples demonstrating how to use the Azure Speech to Text service from Python, a part of the Azure Cognitive Services suite. Key features: - Sample code for converting speech to text using Azure Speech to Text - Demonstrates integration with Azure Cognitive Services - Provides a free tier for 5 hours of audio conversion per month Tech stack: Python Why it matters: This sample code showcases the ease of integrating Azure Cognitive Services into Python applications, making it a valuable resource for developers looking to add speech recognition capabilities to their projects.

Overview: This project is a PyTorch implementation of the Conformer model, focusing on end-to-end speech recognition on the LibriSpeech dataset. It includes a training script and usage instructions for both training from scratch and resuming from checkpoints. Key features: - PyTorch implementation of Conformer model - Training script for LibriSpeech dataset - Support for mixed precision training - Usage instructions for training from scratch, resuming from checkpoint, and smart batching Tech stack: Python Why it matters: This project provides a practical tool for speech recognition tasks, leveraging advanced neural network architectures to improve accuracy and efficiency in audio-to-text conversion. Repository: jreremy/conformer Primary language: Python GitHub stars: 29 GitHub description: Pytorch implementation of conformer with with training script for end-to-end speech recognition on the LibriSpeech dataset.

Overview: This is a project template for Svelte apps, designed to be shareable and easy to use. Key features: - Svelte app template - Uses Svelte for development - Includes a component template for easy component creation Tech stack: Svelte Why it matters: This template is useful for developers looking to quickly set up and share Svelte projects, making it easier to collaborate and deploy their applications.

### Overview: A proof-of-concept Android app utilizing KeenASR SDK, designed to demonstrate the functionality and capabilities of the SDK. ### Key features: - **KeenASR SDK Trial Version**: Includes a trial version of the KeenASR SDK, which will crash the app after 15 minutes. - **Trial SDK Licensing Agreement**: Agreed upon for access to the trial SDK. - **Randomly Chosen Phrases**: Experiment with predefined phrases to create language models and decoding graphs. - **Decoding Graph Creation**: Supports both real-time and pre-bundled decoding graphs for large vocabulary dictation tasks. ### Tech stack: Java ### Why it matters: This app serves as a practical demonstration of KeenASR SDK's capabilities, allowing developers to test and validate the SDK's functionality on an Android device.

### Overview: This project uses pre-trained BERT embeddings for transfer learning in the QA domain, specifically tailored for Stony Brook University (SBU). The system aims to improve the accuracy of question-answering systems by leveraging large datasets and fine-tuning on specific data. ### Key features: - **BERT Pre-trained Embeddings**: Utilizes pre-trained BERT embeddings for QA tasks. - **Fine-tuning on SBU Data**: Fine-tunes the BERT model on SBU-specific QA data. - **Multiway Classification and InferenceSent**: Techniques for context paragraph retrieval and evaluation. ### Tech stack: - TensorFlow - Spacy - Python 3.6 - NLTK ### Why it matters: The project addresses the challenge of obtaining relevant information from the internet by integrating pre-trained BERT embeddings with fine-tuned QA systems. This approach enhances the performance of QA systems on large datasets, improving their ability to answer natural language questions effectively.

Overview: pi-vlm-proxy 是一个为非多模态模型提供图片识别功能的扩展工具，允许这些模型通过描述图片来获取文字描述，而无需切换到支持视觉的模型。它通过自动隐藏工具和零额外开销确保了零厂商硬编码，主模型支持视觉时自动隐藏自己。 Key features: - 自动隐藏支持视觉的模型 - 通过描述图片自动调用多模态模型获取文字描述 - 配置文件权限固定，展示时自动脱敏 - 支持多种 OpenAI 兼容的模型，如火山 Ark、阶跃 StepFun、通义 Qwen-VL、智谱 GLM 等 - 无需手动切换模型，切换时自动同步状态 - 提供五个命令：`list`、`add`、`edit`、`remove`、`use` - 支持两种传图方式：本地路径和 base64 编码的截图 Tech stack: TypeScript Why it matters: 通过让非多模态模型能够识别图片，pi-vlm-proxy 提供了一种灵活且零厂商硬编码的方式，使得这些模型能够更好地适应各种应用场景，同时保持了模型的性能和效率。 Repository: lawrencewzen/pi-vlm-proxy Primary language: TypeScript GitHub stars: 29 GitHub description: 让非多模态模型（DeepSeek 等）把图片识别委托给任意 OpenAI 兼容的多模态模型。零厂商

### Overview: Human as Agent（人即智能体）是将工程师转变为模型的一种解决方案，通过OpenAI兼容的接口接入，为AI Agent 提效并收集工单。它为AI Agent 提供了零代码接入和实时反馈功能，同时确保涉密任务由人工工程师处理，确保合规性和数据安全。 ### Key features: - **OpenAI 兼容**：支持 `/v1/chat/completions` 和 `/v1/approvals`，提供异步受理和实时状态推送。 - **函数调用（tools）**：请求声明 `tools`，人类工程师照此执行并回填 `tool_calls`。 - **治理层**：分级策略引擎、审批异步化、质量验收分类、审计哈希链、合规报告、工程师评级。 - **多租户**：通过 `upstream_key` 路由租户，任务/审批/项目/规则数据隔离。 - **多工具安装**：一键生成 SKILL / AGENT / 规则，支持在线微调、本机全装、服务器端安装。 - **智能漂移**：支持 AI 任务的智能承接，涉密类任务锁死。 - **人机协作闭环**：工程师终端、多轮追问、知识库三级作用域、答复审核。 - **数据飞轮**：答复标注 → 私有评测集不可变快照 → SFT / 评测 JSONL 导出。 - **测试体系**：211 项

Overview: A simple script for computing CLIP-based scores using a trained DALL-e model. Key features: - Computes CLIP scores for compatibility between images and captions. - Supports cosine similarity and scaling by 100. - Measures compatibility on a scale of -100 to 100. - Outputs average scores for real and generated images. - Provides metrics like relative scores and scores based on the top CLIP score. Tech stack: Python Why it matters: This script is useful for evaluating the compatibility of images with captions, providing insights into the model's ability to generate realistic and contextually relevant images.

### Overview: This project focuses on training a model to spot specific keywords from input classes list using various architectures such as convolutional VGGNet, 1D convolutional network, and CTC model. ### Key features: - **Keyword Spotting**: The model is designed to identify and recognize specific keywords within a set of input classes. - **Model Architectures**: Utilizes Convolutional Neural Networks (CNNs), 1D Convolutional Networks, and CTC models. - **Data Augmentation**: Implements time shift addition of background noise, speed, and stretching of input frequency to enhance the training data. - **Model Parameters**: Configurable parameters such as seed, unknown percentage, silence percentage, validation percentage, testing percentage, model type, loss function, optimizer, learning rate, batch size, and augmentation techniques. ### Tech Stack: Python ### Why it Matters: This project showcases the effectiveness of different models and techniques in keyword spotting, demonstrating how various architectures can be optimized for specific tasks.

Overview: This project compares the performance of speech emotion recognition using two methods: a conventional Support Vector Machine (SVM) and a bidirectional long short-term memory (LSTM). The results are evaluated using speech data from the RAVDESS corpus. Key features: - Comparison of SVM and LSTM for speech emotion recognition - Evaluation using RAVDESS corpus - Use of conventional machine learning algorithms Tech stack: Python, Jupyter Notebook Why it matters: This project highlights the effectiveness of deep learning methods in speech emotion recognition, demonstrating that recent advancements in deep learning have surpassed traditional machine learning techniques in performance.

### Overview: Fluent GPT App is an open-source, multi-platform desktop application designed to bring the power of GPT models to your fingertips. Featuring a sleek Fluent interface, it offers a customizable chat experience across Windows, macOS, and Linux. ### Key Features: - **Cross-Platform Support**: Works seamlessly on Windows, macOS, and Linux. - **Tray Functionality**: Ability to minimize to the system tray for quick access. - **Custom Shortcut Activation**: Open the app with a custom keyboard shortcut. - **Multiple Chat Rooms**: Engage with different GPT models in separate chat rooms. - **Custom Instructions**: Tailor each chat room with specific instructions or guidelines. - **Integration with ChatGPT from OpenAI**: Use ChatGPT by obtaining a token from OpenAI. - **Custom GPT**: You can use your own GPT models by providing a URL to the model. - **Pin app**: Pin the app to the top of your screen for easy access. - **Shell support**: GPT can run shell commands on your computer. - **Quick Prompts**: Users can create their custom quick prompts to use them faster in the chat and overlay. - **Overlay Mode on Text Selection (macOS only)**: When users select text, the app will show a compact horizontal overlay with pre-created user's quick prompts. - **Sidebar Mode**: The app will switch to a compact vertical overlay that will show custom user prompts. - **Run Python Code**: The Fluent GPT App enables seamless integration and execution of Python code locally. - **Web Search

### Overview: A versatile Android keyboard designed for speech-to-text, offering real-time voice recording, processing queue, and customizable post-processing options. Compatible with OpenAI APIs, Groq, Mistral, and your own server, it supports multiple languages and auto-starts and auto-stops features for seamless integration. ### Key Features: - Real-time voice recording with amplitude visualization. - Processing queue for background transcription. - Cancelable latest dictation. - Offline-proof recovery for recordings. - Saved texts that persist across app restarts. - Configurable API endpoint, model, and language. - Multiple dictation languages with language keys. - Auto-start recording and auto-stop on silence. - Send after a pause (beta) for improved accuracy. - Voice input for other keyboards and apps. ### Tech Stack: Kotlin ### Why It Matters: This keyboard provides a user-friendly interface for real-time speech-to-text transcription, making it ideal for developers looking to integrate speech recognition into their applications. Its features cater to both beginners and advanced users, ensuring a smooth and efficient experience.

### Overview: This repository provides a comprehensive toolset for creating a dataset tailored for personalized text-to-speech models. It leverages the TTS (Text-to-Speech) GitHub repository to generate audio and transcript pairs, making it easier to train AI models that can convert written text into lifelike speech. ### Key Features: - **Audio and Transcript Extraction**: Automatically extracts audio and corresponding transcripts from the TTS repository. - **Personalized Dataset Creation**: Offers a way to create a dataset with audio and transcripts for model training. - **YouTube Audio Download**: Allows downloading audio from YouTube videos for personalized datasets. - **Audio Splitting**: Splits downloaded audio into smaller segments for efficient processing. - **Text Extraction**: Extracts text from audio segments using the `text_extraction` script. - **Dataset Structure**: The final dataset includes a `metadata.txt` file with audio filenames and a `wavs` folder containing audio files. ### Tech Stack: - Python ### Why It Matters: This repository simplifies the process of creating a dataset for text-to-speech models, making it easier to train AI systems that can generate lifelike speech from written text. It is particularly useful for researchers and developers looking to improve the accuracy and quality of their text-to-speech models.

Overview: A minimalistic vector database designed for fast similarity searches. Key features: - Efficient search in logarithmic time - Simple and minimalistic implementation Tech stack: Go Why it matters: Ideal for applications requiring fast and accurate similarity searches, such as recommendation systems, image search, or any scenario where vector similarity is a key metric. Repository: tobias-mayer/vector-db GitHub stars: 29 GitHub description: A simple vector database that can be used to search for similar vectors in logarithmic time.

Overview: Classy is a multi-platform Android, iOS, Desktop, and Web app that uses Compose Multiplatform with Clean Architecture for text-to-image generation and image captioning. Key features: - Compose Multiplatform - Clean Architecture - Image Captioning - Desktop support - HuggingFace's free inference API - Custom backend option Tech stack: Kotlin Why it matters: Classy leverages Kotlin's powerful libraries and Compose Multiplatform for a seamless, cross-platform development experience, making it accessible across multiple devices and platforms. The integration with HuggingFace's free API and the option to use a custom backend make it adaptable to various use cases, ensuring flexibility and performance.

Overview: Orbis Pictus is a tap-to-explore picture book application that generates pages in real time based on user interactions, without links or markup. It is inspired by the concept of an infinite "browser" where every page is an AI-generated image. Key features: - Infinite tap-to-explore navigation - Search and edit functionality - Page versions and breadcrumb navigation - Page-transition morphs (optional) - Tap caching (optional) - Photo upload (optional) - Web search grounding (optional) - Style and composition pickers (optional) - Idle-loop video (optional) Tech stack: TypeScript Why it matters: Orbis Pictus is an independent, open-source homage to flipbook.page, showcasing the potential of AI-generated images in creating a whole browsing experience without traditional links or markup.

### Overview: This project introduces a state-of-the-art biomedical multimodal benchmark designed for emergency care, focusing on enhancing clinical decision support with physiological waveforms. The benchmark is evaluated in two settings: patient discharge diagnoses and patient deterioration events, using a comprehensive dataset of patient data collected within a 90-minute interval. ### Key Features: - **Patient Discharge Diagnoses**: 1,428 patient discharge diagnoses. - **Patient Deterioration Events**: 15 patient deterioration events. - **Features**: Demographics, biometrics, vital parameter trends, laboratory value trends, ECG waveforms, and ECG features. - **Features Diversity**: Offers 1,443 target labels, significantly more than other datasets. - **Accessibility**: Open-source, promoting further research and collaboration. ### Tech Stack: Python ### Why It Matters: The benchmark demonstrates superior performance in both diagnostic and deterioration tasks, using ECG raw waveforms instead of ECG features, which improves performance in a statistically significant manner. This is the first statistically robust demonstration of the added value of raw ECG waveform input for clinically relevant prediction tasks such as diagnoses and deterioration prediction.

### Overview SACNet is a code repository for our paper "Alignment-Free RGBT Salient Object Detection: Semantics-guided Asymmetric Correlation Network and A Unified Benchmark," which was accepted at TMM 2024. This repository contains the latest research and code related to the proposed unaligned RGBT salient object detection dataset. ### Key Features - **Unaligned RGB-T Salient Object Detection Dataset**: Contains 2000 unaligned visible-thermal image pairs from various real-world scenes. - **Unified Benchmark**: Facilitates research on alignment-free RGBT SOD. - **Progressive Correlation Network**: A novel method for detecting salient objects in unaligned RGBT images. ### Tech Stack Python-based framework. ### Why It Matters This work provides a significant contribution to the field of salient object detection, particularly in the context of RGBT images, by offering a new dataset and a novel correlation network approach. The dataset and benchmark can be used to advance research and improve the accuracy of salient object detection in unaligned RGBT images. ### GitHub Stars 28 ### Repository Repository: Angknpng/SACNet ### Primary Language Python ### GitHub Description Alignment-Free RGBT Salient Object Detection: Semantics-guided Asymmetric Correlation Network and A Unified Benchmark ### Citing Our Work If you think our work is helpful, please cite ``` @article{Wang2024alignment, title={Alignment-Free RGBT Salient Object Detection: Semantics-guided

### Overview: BlossomData 是一个专为大模型训练设计的数据处理框架，旨在提供灵活、高效的数据处理解决方案。它内置丰富的算子库，支持不同模态、不同训练阶段的数据处理与合成。框架支持从单机环境无缝迁移到分布式环境，允许用户灵活扩展自定义算子，大幅降低数据处理流程的开发和计算成本。 ### Key features: - **内置算子库**: 支持多种模态和训练阶段的数据处理与合成。 - **灵活算子扩展**: 用户可以轻松定制和扩展算子库，以满足特定需求。 - **数据处理流程优化**: 从单机迁移到分布式环境，显著提高数据处理效率。 - **配置灵活**: 支持环境变量、配置文件和用户目录中的配置，方便用户自定义和管理。 ### Tech stack: Python ### Why it matters: BlossomData 是一个专为大模型训练设计的数据处理框架，旨在提供高效、灵活的数据处理解决方案。它内置丰富的算子库，支持不同模态、不同训练阶段的数据处理与合成。通过灵活的算子扩展和配置管理，BlossomData 使得数据处理流程的开发和计算成本大幅降低，从而提高了大模型训练的效率和质量。

Overview: This repository contains the official implementation of CELL-E: Biological Zero-Shot Text-to-Image Synthesis for Protein Localization Prediction. It uses various image encoders and a transformer to synthesize images for protein localization prediction. Key features: - Protein Threshold Image Encoder - Nucleus Image (Conditional Image) Encoder - CELL-E Transformer Tech stack: Python Why it matters: This repository provides a comprehensive solution for zero-shot protein localization prediction, leveraging zero-shot text-to-image synthesis techniques to enhance the accuracy and efficiency of protein localization in biological studies. GitHub stars: 28 Repository: BoHuangLab/Protein-Localization-Transformer