Inspect AI

This JavaScript code implements a VS Code extension for managing and executing tasks using the inspect-ai Python library. Here's a breakdown of its functionality:

Core Functionality:

• Task Management:
  • Scans workspace for Python files (.py) and Jupyter notebooks (.ipynb) containing @task decorators.
  • Provides a tree/list view to visualize tasks within files and folders.
  • Allows users to create new tasks with a template Python file.
  • Enables running and debugging individual tasks.
• Task Execution:
  • Utilizes the inspect-ai Python library for task execution.
  • Supports running and debugging tasks in both normal and single-sample debug modes.
  • Manages environment variables for inspect-ai through a .env file.
  • Provides a view to configure environment variables.
• Task Configuration:
  • Offers a view to configure task-specific parameters (e.g., epochs, limit, temperature).
  • Stores task configurations in the workspace state.
• Log Viewing:
  • Monitors for newly generated evaluation logs.
  • Provides options to view logs in HTML or text format.
  • Offers a dedicated Inspect View for log visualization.
• CodeLens:
  • Adds code lenses for running/debugging tasks directly within the code editor.

Key Components:

• InspectManager: Manages the availability of the inspect-ai library.
• WorkspaceTaskManager: Scans the workspace and maintains a list of available tasks.
• ActiveTaskManager: Tracks the currently active task based on editor selection.
• InspectEvalManager: Handles execution of tasks using inspect-ai.
• DebuggerManager: Manages debugging sessions for tasks.
• WorkspaceStateManager: Stores and retrieves workspace-specific configurations.
• WorkspaceEnvManager: Manages environment variables for inspect-ai.
• LogViewFileWatcher: Monitors for new evaluation logs.
• InspectLogviewManager: Manages the Inspect View for log visualization.

How It Works:

1. Activation: The extension activates on workspace open, initializes settings, and activates the Python extension.
2. Task Discovery: The WorkspaceTaskManager scans the workspace for files containing tasks.
3. Task View: The task outline (tree/list view) displays the discovered tasks.
4. Task Selection: The ActiveTaskManager tracks the task based on the active editor's selection.
5. Task Execution: Users can execute or debug the selected task through commands or code lenses.
6. Inspect Integration: The InspectEvalManager uses the inspect-ai library to execute tasks.
7. Configuration: Task configurations are stored in the workspace state, and users can adjust them in dedicated views.
8. Log Viewing: The LogViewFileWatcher detects new evaluation logs, and users can view them in the Inspect View or as text.
9. Environment Management: The WorkspaceEnvManager reads and updates the .env file, providing environment variables to inspect-ai.

Note: The code also includes various helper functions for managing webviews, JSON-RPC communication, file system operations, and error handling.