Michal/TuneRipper
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update docs: new README, IIS guide, and production web.config

Browse Source 
- Replaced README.md with a modern, production-focused version featuring quick start, architecture, workflow, troubleshooting, and security sections
- Added DEPLOYMENT_GUIDE_IIS_Windows11.md with step-by-step IIS deployment instructions, including SSL, CORS, and permissions
- Added version-controlled web.config files for API and Blazor app with HTTPS redirection, security headers, and SPA/static file support
- Moved detailed yt-dlp integration notes to README_UPDATED.md for technical reference
- Improves clarity, maintainability, and production readiness of documentation and deployment
This commit is contained in:
Michal Vrba
2025-12-12 15:31:03 +01:00
parent 3ecff7ca73
commit 54fb8c2c5e
6 changed files with 1096 additions and 380 deletions
Download Patch File Download Diff File Expand all files Collapse all files
TuneRipper.Api/README.md
-84
View File
@@ -1,84 +0,0 @@
# TuneRipper API
## Quick Start
### Starting the API
**Option 1: Using the start script (Windows)**
```cmd
cd TuneRipper.Api
start-api.bat
```
**Option 2: Using the start script (Linux/Mac)**
```bash
cd TuneRipper.Api
chmod +x start-api.sh
./start-api.sh
```
**Option 3: Manual start**
```cmd
cd TuneRipper.Api
dotnet run
```
### API Endpoints
Once running, the API will be available at:
- **HTTPS**: https://localhost:7160
- **HTTP**: http://localhost:5112
### Testing the API
1. **Start the API** using one of the methods above
2. **Open the HTTP test file**: `TuneRipper.Api/api-tests.http`
3. **Run the connectivity tests first** to verify the API is responding
4. **Use the main API tests** once connectivity is confirmed
### Troubleshooting
#### "This localhost page can't be found" errors:
1. **Check if API is running**:
- Look for console output showing "Now listening on: https://localhost:7160"
- If not running, start it using the methods above
2. **Try HTTP instead of HTTPS**:
- Use `http://localhost:5112` instead of `https://localhost:7160`
- Some development environments have HTTPS certificate issues
3. **Trust development certificates** (for HTTPS):
```cmd
dotnet dev-certs https --trust
```
4. **Check port availability**:
- Make sure ports 7160 and 5112 aren't used by other applications
- You can change ports in `Properties/launchSettings.json`
#### HTTP test file variables not working:
1. **Copy job IDs manually**:
- Run a POST request to start processing
- Copy the `fileName` value from the response
- Use it in the manual test sections
2. **Use the troubleshooting section**:
- Start with connectivity tests
- Use manual GUID replacements if variables fail
### API Documentation
When the API is running, you can access:
- **Root endpoint**: https://localhost:7160/ (lists available endpoints)
- **OpenAPI spec**: https://localhost:7160/openapi/v1.json
- **Health check**: https://localhost:7160/api/health
### Development
The API includes:
- Automatic reload on code changes (when using `dotnet run`)
- Detailed logging in the console
- CORS enabled for Blazor WebAssembly client
- OpenAPI/Swagger support for API documentation
TuneRipper.Api/README_UPDATED.md
-296
View File
@@ -1,296 +0,0 @@
# TuneRipper API - yt-dlp Integration
## Overview
The TuneRipper API has been updated to use `yt-dlp.exe` for real YouTube video processing with progress monitoring. Videos are downloaded and converted to MP3 files, stored in the `wwwroot/downloads` directory with a GUID-based filename format, and served directly through the API.
## Setup
### Prerequisites
1. **.NET 10** installed
2. **yt-dlp.exe** placed in `TuneRipper.Api/wwwroot/yt-dlp.exe`
3. **ffmpeg** available in PATH (required by yt-dlp for audio conversion)
### File Structure
```
TuneRipper.Api/
??? wwwroot/
? ??? yt-dlp.exe # YouTube downloader executable
? ??? downloads/ # Downloaded MP3 files (auto-created)
??? Controllers/
??? Services/
??? Program.cs
```
## Filename Format
Downloaded files use the structured format: **`{jobId}_{title}.mp3`**
### Format Breakdown:
- **`{jobId}`**: GUID identifier for the processing job (e.g., `550e8400-e29b-41d4-a716-446655440000`)
- **`_{title}`**: Sanitized video title with spaces replaced by underscores
- **`.mp3`**: Audio format extension
### Examples:
- `550e8400-e29b-41d4-a716-446655440000_Rick_Astley_Never_Gonna_Give_You_Up.mp3`
- `6ba7b810-9dad-11d1-80b4-00c04fd430c8_How_to_Build_a_Web_API.mp3`
- `6ba7b811-9dad-11d1-80b4-00c04fd430c8_Amazing_Guitar_Solo_Performance.mp3`
### Benefits:
- **Unique Identification**: GUID ensures no filename conflicts
- **Direct Job Mapping**: Easy file lookup using job ID
- **Human Readable**: Title part provides context
- **System Friendly**: Sanitized for filesystem compatibility
### Title Sanitization:
- Invalid filename characters are replaced with safe alternatives
- Spaces become underscores (`_`)
- Special characters: `< > | : * ? " /` are replaced with safe chars
- Maximum title length: 50 characters
- Fallback to "video" if title cannot be determined
## Key Changes
### 1. Program.cs Updates
- Added `UseStaticFiles()` middleware for serving downloaded files
- Added `AddHttpClient()` for VideoService HTTP requests
- Updated logging and endpoint information
### 2. VideoService Updates
- **GUID-Based Filenames**: Uses job ID as primary identifier in filename
- **Title Sanitization**: Converts video titles to safe filenames
- **Direct Job Mapping**: Files can be found using `{jobId}_*.*` pattern
- **Real yt-dlp Integration**: Uses actual yt-dlp.exe with progress monitoring
- **Progress Parsing**: Real-time progress from yt-dlp output streams
- **Error Handling**: Comprehensive error handling with 5-minute timeout
### 3. VideoController Updates
- Passes job ID to VideoService for filename generation
- Uses actual yt-dlp processing instead of simulation
- Stores downloaded file information with GUID-based naming
- Returns real download URLs for completed files
### 4. DownloadController Updates
- **Precise Job Lookup**: Uses `{jobId}_*.*` pattern for exact file matching
- **User-Friendly Downloads**: Converts filenames back to readable format
- **Example**: `550e8400-e29b-41d4-a716-446655440000_Rick_Astley_Never_Gonna_Give_You_Up.mp3` ? `Rick Astley Never Gonna Give You Up.mp3`
- **Reliable File Operations**: No ambiguity in file identification
- Serves actual MP3 files from `wwwroot/downloads/`
- Proper MIME type detection for audio files
## API Workflow
### 1. Start Processing
```http
POST /api/video/process
Content-Type: application/json
{
"url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ",
"format": "mp3",
"quality": "128"
}
```
**Response:**
```json
{
"success": true,
"message": "Processing started",
"fileName": "550e8400-e29b-41d4-a716-446655440000",
"downloadUrl": null,
"videoDetails": null
}
```
### 2. Monitor Progress
```http
GET /api/video/status/550e8400-e29b-41d4-a716-446655440000
```
**Response (Processing):**
```json
{
"percentageComplete": 45,
"status": "Processing",
"message": "Downloading audio...",
"videoDetails": {
"id": "dQw4w9WgXcQ",
"title": "Rick Astley - Never Gonna Give You Up",
"thumbnailUrl": "https://img.youtube.com/vi/dQw4w9WgXcQ/maxresdefault.jpg"
}
}
```
### 3. Get Final Result
```http
GET /api/video/result/550e8400-e29b-41d4-a716-446655440000
```
**Response:**
```json
{
"success": true,
"message": "Video processed successfully",
"fileName": "550e8400-e29b-41d4-a716-446655440000_Rick_Astley_Never_Gonna_Give_You_Up.mp3",
"downloadUrl": "/downloads/550e8400-e29b-41d4-a716-446655440000_Rick_Astley_Never_Gonna_Give_You_Up.mp3",
"videoDetails": { ... }
}
```
### 4. Download File
```http
GET /api/download/550e8400-e29b-41d4-a716-446655440000
```
**Response:** Binary MP3 file download with user-friendly filename
**Download Name:** `Rick Astley Never Gonna Give You Up.mp3`
**Alternative Static File Access:**
```http
GET /downloads/550e8400-e29b-41d4-a716-446655440000_Rick_Astley_Never_Gonna_Give_You_Up.mp3
```
## File Management
### Download Directory
- **Location**: `wwwroot/downloads/`
- **Naming Pattern**: `{jobId}_{title}.ext`
- **Job Lookup**: Files found using `Directory.GetFiles(dir, "{jobId}_*.*")`
- **User Download**: Converted back to readable names
- **Cleanup**: Files can be deleted by job ID
### File Serving Options
1. **API Download**: Controlled access via `/api/download/{jobId}` (recommended)
- Uses job ID for secure file lookup
- Returns user-friendly filename
- Proper error handling
2. **Static Files**: Direct access via `/downloads/{actual_filename}`
- Requires knowing the full filename
- Less secure but faster
3. **Content-Type**: Automatic detection based on file extension
### File Operations by Job ID
- **Find**: `GET /api/download/info/{jobId}` - Get file metadata
- **Download**: `GET /api/download/{jobId}` - Download the file
- **Delete**: `DELETE /api/download/{jobId}` - Remove the file
## Progress Monitoring
The VideoService monitors yt-dlp output and translates it to progress updates:
- **5-15%**: Fetching video details
- **15-70%**: Downloading audio (yt-dlp download progress × 0.7)
- **70-85%**: Converting to MP3 (ffmpeg processing)
- **85-95%**: Finalizing (cleanup)
- **100%**: Complete
## Error Handling
### Common Errors:
1. **yt-dlp.exe not found**: Ensure yt-dlp.exe is in wwwroot directory
2. **ffmpeg not available**: Install ffmpeg and add to PATH
3. **Invalid YouTube URL**: URL format validation
4. **Download timeout**: 5-minute timeout for processing
5. **File not found**: File may have been cleaned up or job ID invalid
### Error Response Format:
```json
{
"success": false,
"message": "Error description",
"fileName": null,
"downloadUrl": null,
"videoDetails": null
}
```
## Configuration
### appsettings.json
```json
{
"Logging": {
"LogLevel": {
"TuneRipper.Api.Services.VideoService": "Debug"
}
}
}
```
### yt-dlp Arguments Used:
- `--extract-audio`: Extract audio only
- `--audio-format mp3`: Convert to MP3
- `--audio-quality 0`: Best quality
- `--no-playlist`: Single video only
- `--no-warnings`: Clean output
- `--newline`: Line-by-line progress
- `--output "{jobId}_{title}.%(ext)s"`: Uses job ID-based filename
## Testing
### Manual Testing:
1. Start the API: `dotnet run --project TuneRipper.Api`
2. Use API testing tools or the Blazor client
3. Check logs for yt-dlp progress
4. Verify files in `wwwroot/downloads/` with GUID-based naming
### Example Test Files:
```
wwwroot/downloads/
??? 550e8400-e29b-41d4-a716-446655440000_Rick_Astley_Never_Gonna_Give_You_Up.mp3
??? 6ba7b810-9dad-11d1-80b4-00c04fd430c8_How_to_Build_a_Web_API.mp3
??? 6ba7b811-9dad-11d1-80b4-00c04fd430c8_Amazing_Guitar_Solo_Performance.mp3
```
### Health Check:
```http
GET /api/health
```
### Job Listing:
```http
GET /api/video/jobs
```
## Production Considerations
1. **File Storage**: Consider cloud storage for production
2. **Cleanup Strategy**: Implement automatic file cleanup/retention using job completion timestamps
3. **Security**: Job IDs provide secure file access (no directory traversal)
4. **Performance**: Consider queue-based processing for high load
5. **Monitoring**: Add detailed logging and metrics
6. **Updates**: Keep yt-dlp.exe updated for compatibility
7. **Scalability**: GUID-based naming prevents conflicts in distributed systems
## Troubleshooting
### Common Issues:
1. **Permissions**: Ensure API can write to wwwroot/downloads
2. **Antivirus**: yt-dlp.exe may be flagged by antivirus
3. **Network**: YouTube may block requests from some IPs
4. **Format Support**: Some videos may not support audio extraction
5. **Long Titles**: Titles are truncated to 50 characters for filesystem compatibility
### Debug Logging:
Enable debug logging for VideoService to see detailed yt-dlp output:
```json
{
"Logging": {
"LogLevel": {
"TuneRipper.Api.Services.VideoService": "Debug"
}
}
```
### File Lookup Issues:
- Verify job ID matches the GUID format
- Check that files exist with the `{jobId}_` prefix
- Look at directory contents if files are not found
- Ensure proper case sensitivity on Linux systems
### Advantages of GUID-Based Naming:
1. **Unique**: No filename conflicts possible
2. **Secure**: Job ID acts as access token
3. **Traceable**: Direct mapping from API calls to files
4. **Scalable**: Works in distributed environments
5. **Clean**: Easy cleanup by job ID pattern
TuneRipper.Api/web.config
+34
View File
@@ -0,0 +1,34 @@
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<location path="." inheritInChildApplications="false">
<system.webServer>
<handlers>
<add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModuleV2" resourceType="Unspecified" />
</handlers>
<aspNetCore processPath="dotnet" arguments=".\TuneRipper.Api.dll" stdoutLogEnabled="false" stdoutLogFile=".\logs\stdout" hostingModel="inprocess">
<environmentVariables>
<environmentVariable name="ASPNETCORE_ENVIRONMENT" value="Production" />
</environmentVariables>
</aspNetCore>
<rewrite>
<rules>
<rule name="Redirect to HTTPS" stopProcessing="true">
<match url=".*" />
<conditions>
<add input="{HTTPS}" pattern="off" ignoreCase="true" />
</conditions>
<action type="Redirect" url="https://{HTTP_HOST}/{R:0}" redirectType="Permanent" />
</rule>
</rules>
</rewrite>
<!-- Security headers -->
<httpProtocol>
<customHeaders>
<add name="X-Content-Type-Options" value="nosniff" />
<add name="X-Frame-Options" value="DENY" />
<add name="X-XSS-Protection" value="1; mode=block" />
</customHeaders>
</httpProtocol>
</system.webServer>
</location>
</configuration>