Sync Product co2api-todo from project files

2026-07-06 14:29:44 +02:00
parent d11b389d77
commit 8dde4d14f4

663
Product co2api-todo.-.md Normal file

@@ -0,0 +1,663 @@
<!-- codex-wiki-sync:c3ae91de43ce7d3edbe6dea2 -->
> Mirrored from `/mnt/DATA/Nextcloud/ADD ideas UG/Products/co2api/todo.txt`.
> Origin: `product:co2api`.
> Active tasks and changing state belong in Gitea issues; this wiki page is durable project context.
---
First draft
===========
1. Core Data Model Completion
Add APIKey model (UUID key, org association, creation, expiration, etc.)
Add Factor model with managed_by_role relationship
Add Airport model for API
Add Organization Billing Info model (planned, separate from org)
Add handy User model methods:
check_password / set_password (hashing)
is_active_now() (active + validity)
has_permission(permission_name)
get_organizations()
can_view_organization(org) (hierarchical rights)
2. Permissions & Roles
Helper method: check if a role has a permission (role-permission logic)
Role inheritance rules/templates (role hierarchies or templates)
Data Admin role & dataset access integration (dataset ownership, tag-based access)
Implement permission checks respecting organization hierarchy (parent → children orgs)
Admin takeover/view rights propagation across org hierarchy
Recursive org tree handling, traversal for nested org access rights
3. Authentication & Access Control
Implement login views with Flask-Login
Wire up role-based access control using decorators and permissions
Define and implement register_blueprints and core views
User session management and security (password hashing, active status)
4. API Keys
Finalize APIKey model details (UUID keys, expiration, rate limiting, rotation)
Design API keys management within organizations
Build views/endpoints for API key management (admin-only self-service)
5. Admin Interface
Set up Flask-Admin interface
Register models: User, Organization, Role, Permission, Membership, APIKey
Create scoped admin views restricted by organization and permissions
Add admin views for API key management
Create scripts to seed default roles and permissions
6. Organization Management
Implement organization hierarchy and sub-org support
Handle recursive org tree operations (list all nested sub-orgs)
Build APIs for managing org memberships, roles per user per org
Build APIs for managing roles and permissions dynamically
7. API Endpoints & Permissions
Define API admin role and permissions to manage API endpoints
Build interface for defining API logic and permissions dynamically (details TBD)
8. Testing
Unit tests for:
Permissions and roles logic
Decorators and view protections
User model methods
Organization hierarchy permissions
Integration tests for:
API endpoints
Authentication flows
Admin views access control
Suggested order:
Start with Core Data Model Completion → then Permissions & Roles → Authentication & Access Control → Organization Management → Admin Interface → API Keys → API Endpoints → Testing
1. What you might have missed
Audit Logging / Change Tracking:
Consider adding audit trails for critical actions—who changed what and when (especially for roles, permissions, org membership changes). This helps with debugging and compliance later.
Caching Layer for Permissions:
Permission checks can be expensive if done repeatedly (especially recursive org checks). Plan caching strategies early, maybe per-request cache or Redis, to avoid DB overload.
Event System / Hooks:
Hooks or events on role/permission changes (e.g., signals) could make the system extensible for notifications or syncing with external systems.
Multi-tenancy considerations:
You have org separation, but think about scaling with strict tenant isolation if needed (e.g., separate DB schemas).
Security Enhancements:
Rate limiting (not just API keys but login attempts)
Password reset / MFA (if you add Factor, might extend to MFA)
Session management (invalidate sessions on role change?)
Documentation & API Specs:
Plan docs or OpenAPI specs as you build API endpoints for clarity and future integration.
2. On testing: Should you do it now or later?
I strongly recommend TDD or at least writing tests continuously during development.
It helps prevent regressions as your permission system grows complex.
Catch subtle bugs early (especially recursive org permission checks).
Tests become living documentation for your access control rules.
It saves tons of debugging time later.
Start small — e.g., write unit tests for User methods, decorators, and the data model first. Then expand to integration tests.
3. Crazy ideas to make it better
Graph-based permissions:
Instead of traditional role/permission tables, consider modeling orgs, users, roles, permissions as a graph (e.g., using Neo4j or a graph DB). Traversing hierarchical permissions and inheritance becomes easier and more intuitive.
Visual Org & Role Management UI:
A drag-and-drop or tree UI for org hierarchies, roles, and permission assignments can be a huge usability boost.
Policy-as-Code:
Instead of static permission tables, encode complex rules in a policy language (like OPA/Rego). This enables dynamic, condition-based permission checks.
AI-assisted permission audits:
Use automated tools or ML to analyze permission assignments and suggest optimizations or detect risky permission grants.
Self-service org and role management:
Build workflows allowing org owners/admins to manage roles & permissions without dev intervention, with safeguards.
✅ Immediate Next Steps (High Value)
1. Extend Access Control Testing
Youve built great coverage around user_has_permission. Now:
🔍 Test the decorators like @permission_required, @role_required, etc., using mocked view functions.
🔐 Test view protection with Flask test client — ensure 403s are returned when access is denied.
2. Authentication Flow Tests
✅ Test login/logout logic with Flask-Login.
🔐 Add tests for protected routes using client.get(..., follow_redirects=True).
🔁 Test session handling, login-required redirects, and remember-me logic (if used).
3. Admin View Testing (if using Flask-Admin)
🚦 Log in as admin and test that admin views are accessible.
🔒 Ensure regular users are denied access to protected views.
🔄 Infrastructure and Quality
4. Fixtures + Seeding Utilities
⚙️ Write a seed_roles_and_permissions() function to preload your core roles (admin, owner, viewer, etc.).
🧪 Use this in tests and for initializing a dev environment.
5. CI Integration (if not yet set up)
🔁 Run your test suite automatically with GitHub Actions, GitLab CI, or similar.
✅ Validate migrations apply cleanly in a clean environment.
🧩 System Features / Enhancements
6. API Key Logic
Nows a good time to:
🔑 Add API key creation, lookup, revocation logic.
🧪 Test API key access with and without valid keys.
🚫 Ensure expired or revoked keys are denied.
7. Organization Tree + Inheritance
🌲 Add methods to fetch sub-orgs recursively (for permissions).
🔍 Write tests for hierarchy traversal, and whether a user has inherited access.
🤝 Introduce org ownership inheritance rules.
🧪 Bonus Ideas (If You Want More Depth)
🧰 Test role-permission editing flows (if you expose them via UI or API).
📄 Audit logging for permission changes.
🔒 Rate limiting or brute-force protection on login/API.
5. Scalability and Performance Tests
Once your core tests are in place, consider scaling the system by testing performance:
Database Load:
Test the system's ability to handle large numbers of users, permissions, and roles.
Simulate the creation of many organizations with permissions and users to ensure that the system can handle load.
Response Time for Permission Checks:
Profile the time it takes to check permissions, especially if the hierarchy grows deeply (many org levels or users).
6. Edge Case Testing
Test scenarios such as:
Invalid inputs for permissions and roles (e.g., empty names, invalid characters).
Simultaneous role assignments (do they conflict or behave as expected?).
Case sensitivity of permissions (should "view_reports" be different from "VIEW_REPORTS"?).
7. Review Database Migrations (if applicable)
If you're using SQLAlchemy migrations (e.g., Alembic), make sure:
The database schema changes are in sync with your model.
Run migration tests (e.g., simulate rolling back schema changes and verifying migrations).
8. Test Coverage
Run a full coverage report to make sure your tests cover all code paths (permissions, roles, organizations, users, and error handling).
Aim for high coverage in critical components, especially permission checks, org hierarchy logic, and user-role assignments.
9. Refactor the Circular Org Test for Robustness
Ensure that the circular reference check is tightly integrated into your ORM model logic.
Make the circular check part of the org creation process rather than a test. This will prevent you from running into issues when creating circular dependencies in any part of your app, not just in tests.
1. Permission System Improvements
Since you're working with complex permission checks, roles, and organizations, we should focus on:
Refactor the permission propagation logic: Ensure that the propagation of permissions across organizations and sub-organizations works as intended without any loopholes.
Role-based permission checks: Create more granular tests around how permissions are assigned through roles and memberships.
Permission Caching/Optimizing: If the permissions are checked frequently, consider caching permission results or optimizing them at the database level.
Priority: High
Why: Your permission system is central to your applications functionality and security.
2. Error Handling and Logging
Add robust error handling for edge cases (e.g., circular dependencies, invalid operations).
Implement logging: If not already done, adding proper logging can make your debugging and monitoring easier, especially for production deployments.
Priority: Medium
Why: Helps maintain reliability, especially in a production environment.
3. User Model Enhancements
Refactor user-related queries to improve performance (e.g., lazy loading related data only when necessary).
Enhance the soft delete functionality: Ensure that users who are "soft deleted" dont show up in lists but are still retained for audit purposes.
Priority: Medium
Why: Helps improve code clarity and ensures the user model scales better in production.
4. Security and Authentication Checks
Review and test password storage (even if hashed with Werkzog, check for potential security flaws).
Add two-factor authentication (optional but recommended for security-sensitive apps).
Token-based authentication (JWT or similar), if not implemented yet, can be useful for stateless authentication.
Priority: High
Why: Authentication and password handling are always critical for security.
5. Performance Optimizations
Analyze slow queries: Use SQLAlchemys EXPLAIN or logging to detect slow queries that could be optimized.
Optimize database indexing: For any columns that are frequently queried, especially email, user_id, and organization_id.
Batch processing for handling large data (if needed).
Priority: Medium
Why: Scalability and performance are important as your application grows.
6. Automated Testing Coverage
Expand test coverage: Ensure you have tests for all edge cases (e.g., non-existent organizations, empty permission checks, etc.).
End-to-end testing: If your system interacts with external APIs or services, ensure integration tests are in place.
Test data integrity: Simulate concurrent writes/updates to test consistency and correctness of the database.
Priority: High
Why: Ensuring that the application works under various scenarios and that new code does not break existing functionality.
7. Documentation
Document your models and functions: Ensure your code is well-documented (docstrings for classes and functions).
Write a readme (if not already done) for setup, configuration, and running tests.
Document key architectural decisions: What drove certain designs (e.g., circular dependency detection, user role assignments)?
Priority: Medium
Why: Documentation is essential for team collaboration and future developers.
8. Deployment Considerations
Prepare for deployment: If you're nearing production, review deployment configurations (e.g., Docker, cloud services, CI/CD pipelines).
Database migrations: Ensure proper migrations for any schema changes (Alembic can help with this).
Priority: Low
Why: You may be ready to move towards deployment but focus on core features first.
Prioritization Breakdown:
Critical Path: Permission System Improvements, Error Handling and Logging, Security
Core Functionality: User Model Enhancements, Performance Optimizations
Non-Critical: Test Coverage, Documentation, Deployment Considerations
Next Steps:
First Priority: Let's focus on improving your permission system and ensuring that error handling and logging are top-notch.
Second Priority: Work on security features (password storage, 2FA) and optimize the user model for performance.
Second draft
============
MVP Development Roadmap (Plain Text Version)
Date: 2025-07-31
------------------------------------------------------------
Shortened MVP Roadmap (Big Picture Overview)
Phase 1: Core Data & Permissions Infrastructure
- Implement core models (User, Organization, Role, Permission, Membership, APIKey, Airport)
- Support for organization hierarchy, permission propagation, and access rules
Phase 2: Authentication & Access Control
- Build login/logout flow with session management
- Role-based decorators and guards
- Secure views and APIs based on roles and permissions
Phase 3: Admin Interface (Flask-Admin)
- Add admin views for key models
- Restrict access based on org/permissions
- Support user/role/membership management
Phase 4: API Keys & Public Endpoints
- Implement API key model and validation
- Add endpoints for airport lookup and factor queries
- Secure with org-bound API keys and permissions
Phase 5: Testing & Deployment Preparation
- Write unit/integration tests
- Seed roles, permissions, users
- Prepare for deployment with migrations and basic CI
------------------------------------------------------------
Step-by-Step Detailed Task List
Phase 1: Core Data Model & Permissions
- API key ownership
- org admin can create, view, asign to key owner, renew, retire, (de-)activate, change dates ON ALL KEYS
- key owner can view, renew, retire (?), (de-)activate, change dates ON THEIR key
- org member can do what? nothing by default, they are only visible to the org admin
- Add Factor model (linked to org, managed_by_role)
- Add Airport model (IATA, ICAO, name, country, lat/lon)
- Add OrganizationBillingInfo (one-to-one or FK to org)
- Add utility methods to User:
- has_permission(permission_name)
- can_view_organization(org)
- get_organizations()
- Create org traversal utilities (e.g., get all sub-orgs)
- Implement Permission, Role, and Membership models
- Role templates: SysAdmin, DataAdmin, OrgOwner
- Add utility: role.has_permission(permission)
- Create seed function: default roles and permissions
Phase 2: Authentication & Access Control
- Implement login view with Flask-Login
- Add logout view and user session handling
- Enforce active user check via is_active_now
- Add decorators:
- @permission_required("…")
- @role_required("…")
- Register blueprints: auth, admin, api, org
- Create default error handlers (403, 404)
Phase 3: Admin Interface (Flask-Admin)
- Set up Flask-Admin
- Register models:
- User
- Organization
- Role
- Permission
- Membership
- APIKey
- Restrict views based on user org + permission
- Add inline editing for memberships and roles
- Conditionally hide sensitive fields (e.g., password_hash)
- Tree for org
- Send api key to ...
Phase 4: API Keys & Endpoints
- Implement APIKey generation, expiration, and revocation
- Create views for API key management (org-scoped)
- Secure APIs using API keys with rate limit (optional)
- Define endpoints:
- /api/airports → public IATA/ICAO search
- /api/factors → organization-specific factor lookup
- Require API key header (or param) and permission check
Phase 5: Testing, Fixtures, Deployment
- Write unit tests for:
- Decorators
- API key model behavior
- Integration tests for:
- Auth flows (login/logout)
- Protected routes
- API key access
- Seed data for dev env (orgs, users, roles)
- Add fixtures for test suite
- Run Alembic migrations
- Create .env.example, Dockerfile (optional)
- Add CI workflow for test runner (e.g., GitHub Actions)
Additional Improvements & Enhancements
Security
- Add rate limiting (login & API key)
- Add optional password reset logic
- Plan for MFA extension using Factor model
- Session invalidation on password change
Smart Enhancements (Optional)
- Add caching for permission/org traversal (per-request)
- Add audit logging on permission changes, key use, etc.
- Visual org hierarchy editor (tree view)
- Policy-as-code (e.g., Rego/OPA for dynamic permissions)
Immediate Next Steps
1. Models & Seeding
- Build APIKey, Airport, Factor, and BillingInfo models
- Add full User utility methods
- Implement org hierarchy logic and circular detection
- Write seeders for roles, orgs, and base permissions
2. Authentication & Access
- Create login/logout flow with session checks
- Implement @permission_required on dummy route
- Protect routes using is_active_now
3. Admin Panel
- Register models in Flask-Admin
- Add access checks
- Enable org-aware scoped views
Automatically log in the user after registering
Add CAPTCHA, email confirmation, etc. on registration - prob. means switching to usernames instead of email; allow both on login
Implement RBAC (role-based access control) using decorators or dependency injection (if FastAPI).
Consider Soft Deletion + Audit Logging for emissions data edits.
For time-dependency in emissions models, use a valid-from/valid-to range in the schema.
Use Pydantic models (FastAPI) or Marshmallow (Flask) to enforce schema integrity.
Consider making the API Key logic pluggable, so you can later add scopes, revocation, or analytics.
Third draft
===========
Core Development Priorities for MVP
________________________________________
1. Core Data Model Completion (High Priority)
Tasks:
1. Add APIKey Model
o Fields: UUID key, org association, creation date, expiration date, etc.
o Use db.Model to define the APIKey schema.
o Ensure secure handling of API keys, such as hashing the keys.
2. Add Factor Model
o Define a Factor model for multi-factor authentication (optional for MVP, but essential for future security improvements).
o Establish a managed_by_role relationship.
3. Add Airport Model
o Fields: airport_code, airport_name, location (city, country).
o Ensure this model is queryable for API usage.
4. Add Organization Billing Info Model
o Fields: planned amount, actual amount, due date, etc.
o Relate this to the Organization model via a foreign key.
5. Enhance User Model
o Methods:
 check_password: To securely verify user passwords.
 set_password: To securely hash and store user passwords.
 is_active_now: Checks if the user is currently active based on start_date, end_date, and is_active.
 has_permission: Check if the user has a specific permission.
 get_organizations: Get a list of organizations the user is associated with.
 can_view_organization: Verify if the user has access to a particular organization based on hierarchical permissions.
Priority: High This will lay the foundation for all other features, including authentication, roles, and permissions.
________________________________________
2. Permissions & Roles (High Priority)
Tasks:
1. Helper Method for Permission Check
o Implement a helper function to check if a role has a specific permission (role-permission logic).
2. Role Inheritance Rules/Templates
o Define role hierarchies or templates that determine inheritance.
o Data Admin → Dataset access permissions.
o Define roles like sys_admin, data_admin, user with specific access restrictions.
3. Role-based Access for Organization Hierarchy
o Implement logic to propagate permissions from parent → child organizations.
o Ensure recursive org tree handling to manage nested organizations.
o Build an admin takeover feature for sys-admin roles to control nested orgs.
4. Implement Permission Checks
o Add checks for whether a user can view/edit specific organizations based on the hierarchical relationship.
Priority: High These features are critical for the organization management and user access control logic.
________________________________________
3. Authentication & Access Control (High Priority)
Tasks:
1. Implement Login Views with Flask-Login
o Set up login and logout views using Flask-Login.
o Ensure password hashing with werkzeug.security.
2. Define and Implement Core Views (Registration)
o Implement registration functionality and wire up role-based access control (RBAC) using decorators.
3. Session Management & Security
o Implement session handling for active users.
o Ensure proper login required checks for protected routes.
4. Role-Based Access Control (RBAC) Decorators
o Implement decorators like @role_required or @permission_required to protect routes based on user roles and permissions.
Priority: High These are essential for ensuring proper user management and access control.
________________________________________
4. Admin Interface (Medium Priority)
Tasks:
1. Set up Flask-Admin
o Install and integrate Flask-Admin for user-friendly management.
o Register User, Organization, Role, Permission, Membership, and APIKey models in Flask-Admin.
2. Create Scoped Admin Views
o Restrict admin views based on organization and permissions.
o Allow admins to view and manage users and roles within their organization.
3. Seed Default Roles and Permissions
o Implement a seeding function to pre-load core roles (admin, user, etc.) and default permissions.
Priority: Medium Admin interface is needed but not urgent for MVP launch, and can be done after the basic roles and permissions are set up.
________________________________________
5. API Keys (Medium Priority)
Tasks:
1. Finalize APIKey Model
o Add UUID, expiration, rate-limiting, and other details.
o Implement CRUD operations to manage API keys.
2. Design API Key Management Views
o Create views for admins to generate and manage API keys.
o Allow API key creation and revocation.
Priority: Medium API key management is important but can be built after the core model and authentication are in place.
________________________________________
6. Organization Management (Medium Priority)
Tasks:
1. Implement Organization Hierarchy
o Define an organization model with parent-child relationships for recursive structure.
o Enable org creation, updating, deletion, and membership management.
2. Manage Org Memberships & Roles
o Build endpoints to manage users within organizations (assign roles, permissions, etc.).
o Allow admins to manage members at various levels (sys_admin, data_admin, etc.).
3. Organization Permissions
o Implement access control for users based on their organization hierarchy.
Priority: Medium Core functionality, but can be tackled after user roles and permissions are functional.
________________________________________
7. API Endpoints & Permissions (Medium Priority)
Tasks:
1. Define API Endpoints
o Focus on endpoints that query data for the organizations (e.g., GET requests for organizations, memberships, and roles).
o Exclude system management or interaction endpoints for now (as per your note).
2. Define Permissions for API Access
o Implement endpoint protection based on user roles and permissions.
Priority: Medium API endpoints are crucial for data querying and integration with external systems but can be done after the basic models are in place.
________________________________________
8. Testing (Ongoing Priority)
Tasks:
1. Unit Tests
o Test user model methods (check_password, set_password, is_active_now).
o Test role-based access control and permission checks.
o Test organization membership handling and recursive org tree access.
2. Integration Tests
o Test full API endpoint functionality (with and without valid API keys).
o Test login and session handling.
3. Testing User Authentication Flow
o Test login/logout, password verification, and protected views with Flask-Login.
4. Test Admin Views & Access Control
o Test admin views for restricted access, user creation, role management, etc.
Priority: High Testing is essential to ensure no regressions and to confirm permissions and authentication work as expected.
________________________________________
Additional Features (Post-MVP Considerations)
• Audit Logging: Implement change tracking for sensitive actions (user role changes, permissions updates, etc.).
• Caching for Permissions: Implement caching for permission checks to reduce the load on the database.
• Security Enhancements: Implement rate limiting, multi-factor authentication (optional), and session invalidation on role changes.
________________________________________
Next Steps (Immediate Focus)
1. Core Data Model Completion:
o Finish the User, APIKey, Factor, Airport, and Organization models.
2. Permissions & Roles:
o Focus on implementing role hierarchies and role-based access control across organizations.
3. Authentication & Access Control:
o Set up Flask-Login for login/logout functionality.
o Implement RBAC decorators and session management.
4. Testing:
o Start with unit tests for the User model and authentication flow, and test role-based access control.
Fourth draft
============
🚧 PARTIALLY DONE / NEEDS REVIEW
🧠 Utility Methods
• get_organizations() and can_view_organization() not found ❌
(should be implemented on User model)
🔐 Decorators
@role_required and @permission_required not yet defined ❌
(important for enforcing RBAC)
🏗️ Models Missing
• Factor ❌
• Airport ❌
• OrganizationBillingInfo ❌
________________________________________
🔄 REVISED ROADMAP (with Checked Status)
________________________________________
🔥 PHASE 1: Core Models & Permissions
🚧 IN PROGRESS:
• Factor, Airport, BillingInfo models
• User.get_organizations(), User.can_view_organization()
• Org hierarchy traversal utilities
• Seed script for default roles/permissions
________________________________________
🔐 PHASE 2: Role Logic & Propagation
• Role templates: sys_admin, org_owner, data_admin, etc.
• Role propagation down org tree
• Admin takeover for sys_admin
• Permission check logic via decorators
________________________________________
🔑 PHASE 3: Authentication Flow
🚧 TO DO:
• is_active_now check on User
• Registration view (if needed)
• Decorators: @role_required, @permission_required
• Secure protected routes
________________________________________
🛠️ PHASE 4: Admin Interface
• Flask-Admin setup
• Register models: User, Org, APIKey, etc.
• Scoped access + inline editing
________________________________________
🔐 PHASE 5: API Keys
• Base model exists
• Add rate limiting, revocation, expiration logic
• API key views for org admins
________________________________________
🌍 PHASE 6: Org Management
• Implement full org tree structure
• Circular checks
• Views for membership and org management
________________________________________
🌐 PHASE 7: API Endpoints
• /api/airports and /api/factors
• Auth via API key + permission check
________________________________________
🧪 PHASE 8: Testing (Ongoing)
• Unit tests for User, permission, APIKey
• Integration tests for auth, protected views
• Fixtures + seed data