diff --git a/Product co2api-todo.-.md b/Product co2api-todo.-.md new file mode 100644 index 0000000..25b4d91 --- /dev/null +++ b/Product co2api-todo.-.md @@ -0,0 +1,663 @@ + + +> Mirrored from `/mnt/DATA/Nextcloud/ADD ideas UG/Products/co2api/todo.txt`. +> Origin: `product:co2api`. +> Active tasks and changing state belong in Gitea issues; this wiki page is durable project context. + +--- +First draft +=========== + +1. Core Data Model Completion + Add APIKey model (UUID key, org association, creation, expiration, etc.) + Add Factor model with managed_by_role relationship + Add Airport model for API + Add Organization Billing Info model (planned, separate from org) + Add handy User model methods: + check_password / set_password (hashing) + is_active_now() (active + validity) + has_permission(permission_name) + get_organizations() + can_view_organization(org) (hierarchical rights) + +2. Permissions & Roles + Helper method: check if a role has a permission (role-permission logic) + Role inheritance rules/templates (role hierarchies or templates) + Data Admin role & dataset access integration (dataset ownership, tag-based access) + Implement permission checks respecting organization hierarchy (parent → children orgs) + Admin takeover/view rights propagation across org hierarchy + Recursive org tree handling, traversal for nested org access rights + +3. Authentication & Access Control + Implement login views with Flask-Login + Wire up role-based access control using decorators and permissions + Define and implement register_blueprints and core views + User session management and security (password hashing, active status) + +4. API Keys + Finalize APIKey model details (UUID keys, expiration, rate limiting, rotation) + Design API keys management within organizations + Build views/endpoints for API key management (admin-only self-service) + +5. Admin Interface + Set up Flask-Admin interface + Register models: User, Organization, Role, Permission, Membership, APIKey + Create scoped admin views restricted by organization and permissions + Add admin views for API key management + Create scripts to seed default roles and permissions + +6. Organization Management + Implement organization hierarchy and sub-org support + Handle recursive org tree operations (list all nested sub-orgs) + Build APIs for managing org memberships, roles per user per org + Build APIs for managing roles and permissions dynamically + +7. API Endpoints & Permissions + Define API admin role and permissions to manage API endpoints + Build interface for defining API logic and permissions dynamically (details TBD) + +8. Testing + Unit tests for: + Permissions and roles logic + Decorators and view protections + User model methods + Organization hierarchy permissions + Integration tests for: + API endpoints + Authentication flows + Admin views access control + +Suggested order: +Start with Core Data Model Completion → then Permissions & Roles → Authentication & Access Control → Organization Management → Admin Interface → API Keys → API Endpoints → Testing + + +1. What you might have missed +Audit Logging / Change Tracking: +Consider adding audit trails for critical actions—who changed what and when (especially for roles, permissions, org membership changes). This helps with debugging and compliance later. + +Caching Layer for Permissions: +Permission checks can be expensive if done repeatedly (especially recursive org checks). Plan caching strategies early, maybe per-request cache or Redis, to avoid DB overload. + +Event System / Hooks: +Hooks or events on role/permission changes (e.g., signals) could make the system extensible for notifications or syncing with external systems. + +Multi-tenancy considerations: +You have org separation, but think about scaling with strict tenant isolation if needed (e.g., separate DB schemas). + +Security Enhancements: + +Rate limiting (not just API keys but login attempts) + +Password reset / MFA (if you add Factor, might extend to MFA) + +Session management (invalidate sessions on role change?) + +Documentation & API Specs: +Plan docs or OpenAPI specs as you build API endpoints for clarity and future integration. + +2. On testing: Should you do it now or later? +I strongly recommend TDD or at least writing tests continuously during development. + +It helps prevent regressions as your permission system grows complex. + +Catch subtle bugs early (especially recursive org permission checks). + +Tests become living documentation for your access control rules. + +It saves tons of debugging time later. + +Start small — e.g., write unit tests for User methods, decorators, and the data model first. Then expand to integration tests. + +3. Crazy ideas to make it better +Graph-based permissions: +Instead of traditional role/permission tables, consider modeling orgs, users, roles, permissions as a graph (e.g., using Neo4j or a graph DB). Traversing hierarchical permissions and inheritance becomes easier and more intuitive. + +Visual Org & Role Management UI: +A drag-and-drop or tree UI for org hierarchies, roles, and permission assignments can be a huge usability boost. + +Policy-as-Code: +Instead of static permission tables, encode complex rules in a policy language (like OPA/Rego). This enables dynamic, condition-based permission checks. + +AI-assisted permission audits: +Use automated tools or ML to analyze permission assignments and suggest optimizations or detect risky permission grants. + +Self-service org and role management: +Build workflows allowing org owners/admins to manage roles & permissions without dev intervention, with safeguards. + +✅ Immediate Next Steps (High Value) +1. Extend Access Control Testing +You’ve built great coverage around user_has_permission. Now: + +🔍 Test the decorators like @permission_required, @role_required, etc., using mocked view functions. + +🔐 Test view protection with Flask test client — ensure 403s are returned when access is denied. + +2. Authentication Flow Tests +✅ Test login/logout logic with Flask-Login. + +🔐 Add tests for protected routes using client.get(..., follow_redirects=True). + +🔁 Test session handling, login-required redirects, and remember-me logic (if used). + +3. Admin View Testing (if using Flask-Admin) +🚦 Log in as admin and test that admin views are accessible. + +🔒 Ensure regular users are denied access to protected views. + +🔄 Infrastructure and Quality +4. Fixtures + Seeding Utilities +⚙️ Write a seed_roles_and_permissions() function to preload your core roles (admin, owner, viewer, etc.). + +🧪 Use this in tests and for initializing a dev environment. + +5. CI Integration (if not yet set up) +🔁 Run your test suite automatically with GitHub Actions, GitLab CI, or similar. + +✅ Validate migrations apply cleanly in a clean environment. + +🧩 System Features / Enhancements +6. API Key Logic +Now’s a good time to: + +🔑 Add API key creation, lookup, revocation logic. + +🧪 Test API key access with and without valid keys. + +🚫 Ensure expired or revoked keys are denied. + +7. Organization Tree + Inheritance +🌲 Add methods to fetch sub-orgs recursively (for permissions). + +🔍 Write tests for hierarchy traversal, and whether a user has inherited access. + +🤝 Introduce org ownership inheritance rules. + +🧪 Bonus Ideas (If You Want More Depth) +🧰 Test role-permission editing flows (if you expose them via UI or API). + +📄 Audit logging for permission changes. + +🔒 Rate limiting or brute-force protection on login/API. + +5. Scalability and Performance Tests +Once your core tests are in place, consider scaling the system by testing performance: + +Database Load: + +Test the system's ability to handle large numbers of users, permissions, and roles. + +Simulate the creation of many organizations with permissions and users to ensure that the system can handle load. + +Response Time for Permission Checks: + +Profile the time it takes to check permissions, especially if the hierarchy grows deeply (many org levels or users). + +6. Edge Case Testing +Test scenarios such as: + +Invalid inputs for permissions and roles (e.g., empty names, invalid characters). + +Simultaneous role assignments (do they conflict or behave as expected?). + +Case sensitivity of permissions (should "view_reports" be different from "VIEW_REPORTS"?). + +7. Review Database Migrations (if applicable) +If you're using SQLAlchemy migrations (e.g., Alembic), make sure: + +The database schema changes are in sync with your model. + +Run migration tests (e.g., simulate rolling back schema changes and verifying migrations). + +8. Test Coverage +Run a full coverage report to make sure your tests cover all code paths (permissions, roles, organizations, users, and error handling). + +Aim for high coverage in critical components, especially permission checks, org hierarchy logic, and user-role assignments. + +9. Refactor the Circular Org Test for Robustness +Ensure that the circular reference check is tightly integrated into your ORM model logic. + +Make the circular check part of the org creation process rather than a test. This will prevent you from running into issues when creating circular dependencies in any part of your app, not just in tests. + + + + +1. Permission System Improvements +Since you're working with complex permission checks, roles, and organizations, we should focus on: + +Refactor the permission propagation logic: Ensure that the propagation of permissions across organizations and sub-organizations works as intended without any loopholes. + +Role-based permission checks: Create more granular tests around how permissions are assigned through roles and memberships. + +Permission Caching/Optimizing: If the permissions are checked frequently, consider caching permission results or optimizing them at the database level. + +Priority: High +Why: Your permission system is central to your application’s functionality and security. + +2. Error Handling and Logging +Add robust error handling for edge cases (e.g., circular dependencies, invalid operations). + +Implement logging: If not already done, adding proper logging can make your debugging and monitoring easier, especially for production deployments. + +Priority: Medium +Why: Helps maintain reliability, especially in a production environment. + +3. User Model Enhancements +Refactor user-related queries to improve performance (e.g., lazy loading related data only when necessary). + +Enhance the soft delete functionality: Ensure that users who are "soft deleted" don’t show up in lists but are still retained for audit purposes. + +Priority: Medium +Why: Helps improve code clarity and ensures the user model scales better in production. + +4. Security and Authentication Checks +Review and test password storage (even if hashed with Werkzog, check for potential security flaws). + +Add two-factor authentication (optional but recommended for security-sensitive apps). + +Token-based authentication (JWT or similar), if not implemented yet, can be useful for stateless authentication. + +Priority: High +Why: Authentication and password handling are always critical for security. + +5. Performance Optimizations +Analyze slow queries: Use SQLAlchemy’s EXPLAIN or logging to detect slow queries that could be optimized. + +Optimize database indexing: For any columns that are frequently queried, especially email, user_id, and organization_id. + +Batch processing for handling large data (if needed). + +Priority: Medium +Why: Scalability and performance are important as your application grows. + +6. Automated Testing Coverage +Expand test coverage: Ensure you have tests for all edge cases (e.g., non-existent organizations, empty permission checks, etc.). + +End-to-end testing: If your system interacts with external APIs or services, ensure integration tests are in place. + +Test data integrity: Simulate concurrent writes/updates to test consistency and correctness of the database. + +Priority: High +Why: Ensuring that the application works under various scenarios and that new code does not break existing functionality. + +7. Documentation +Document your models and functions: Ensure your code is well-documented (docstrings for classes and functions). + +Write a readme (if not already done) for setup, configuration, and running tests. + +Document key architectural decisions: What drove certain designs (e.g., circular dependency detection, user role assignments)? + +Priority: Medium +Why: Documentation is essential for team collaboration and future developers. + +8. Deployment Considerations +Prepare for deployment: If you're nearing production, review deployment configurations (e.g., Docker, cloud services, CI/CD pipelines). + +Database migrations: Ensure proper migrations for any schema changes (Alembic can help with this). + +Priority: Low +Why: You may be ready to move towards deployment but focus on core features first. + +Prioritization Breakdown: +Critical Path: Permission System Improvements, Error Handling and Logging, Security + +Core Functionality: User Model Enhancements, Performance Optimizations + +Non-Critical: Test Coverage, Documentation, Deployment Considerations + +Next Steps: +First Priority: Let's focus on improving your permission system and ensuring that error handling and logging are top-notch. + +Second Priority: Work on security features (password storage, 2FA) and optimize the user model for performance. + + + +Second draft +============ + +MVP Development Roadmap (Plain Text Version) +Date: 2025-07-31 + +------------------------------------------------------------ + +Shortened MVP Roadmap (Big Picture Overview) + +Phase 1: Core Data & Permissions Infrastructure +- Implement core models (User, Organization, Role, Permission, Membership, APIKey, Airport) +- Support for organization hierarchy, permission propagation, and access rules + +Phase 2: Authentication & Access Control +- Build login/logout flow with session management +- Role-based decorators and guards +- Secure views and APIs based on roles and permissions + +Phase 3: Admin Interface (Flask-Admin) +- Add admin views for key models +- Restrict access based on org/permissions +- Support user/role/membership management + +Phase 4: API Keys & Public Endpoints +- Implement API key model and validation +- Add endpoints for airport lookup and factor queries +- Secure with org-bound API keys and permissions + +Phase 5: Testing & Deployment Preparation +- Write unit/integration tests +- Seed roles, permissions, users +- Prepare for deployment with migrations and basic CI + +------------------------------------------------------------ + +Step-by-Step Detailed Task List + +Phase 1: Core Data Model & Permissions +- API key ownership + - org admin can create, view, asign to key owner, renew, retire, (de-)activate, change dates ON ALL KEYS + - key owner can view, renew, retire (?), (de-)activate, change dates ON THEIR key + - org member can do what? nothing by default, they are only visible to the org admin +- Add Factor model (linked to org, managed_by_role) +- Add Airport model (IATA, ICAO, name, country, lat/lon) +- Add OrganizationBillingInfo (one-to-one or FK to org) +- Add utility methods to User: + - has_permission(permission_name) + - can_view_organization(org) + - get_organizations() +- Create org traversal utilities (e.g., get all sub-orgs) +- Implement Permission, Role, and Membership models +- Role templates: SysAdmin, DataAdmin, OrgOwner +- Add utility: role.has_permission(permission) +- Create seed function: default roles and permissions + +Phase 2: Authentication & Access Control + +- Implement login view with Flask-Login +- Add logout view and user session handling +- Enforce active user check via is_active_now +- Add decorators: + - @permission_required("…") + - @role_required("…") +- Register blueprints: auth, admin, api, org +- Create default error handlers (403, 404) + +Phase 3: Admin Interface (Flask-Admin) + +- Set up Flask-Admin +- Register models: + - User + - Organization + - Role + - Permission + - Membership + - APIKey +- Restrict views based on user org + permission +- Add inline editing for memberships and roles +- Conditionally hide sensitive fields (e.g., password_hash) +- Tree for org +- Send api key to ... + +Phase 4: API Keys & Endpoints + +- Implement APIKey generation, expiration, and revocation +- Create views for API key management (org-scoped) +- Secure APIs using API keys with rate limit (optional) +- Define endpoints: + - /api/airports → public IATA/ICAO search + - /api/factors → organization-specific factor lookup +- Require API key header (or param) and permission check + +Phase 5: Testing, Fixtures, Deployment + +- Write unit tests for: + - Decorators + - API key model behavior +- Integration tests for: + - Auth flows (login/logout) + - Protected routes + - API key access +- Seed data for dev env (orgs, users, roles) +- Add fixtures for test suite +- Run Alembic migrations +- Create .env.example, Dockerfile (optional) +- Add CI workflow for test runner (e.g., GitHub Actions) + +Additional Improvements & Enhancements + +Security +- Add rate limiting (login & API key) +- Add optional password reset logic +- Plan for MFA extension using Factor model +- Session invalidation on password change + +Smart Enhancements (Optional) +- Add caching for permission/org traversal (per-request) +- Add audit logging on permission changes, key use, etc. +- Visual org hierarchy editor (tree view) +- Policy-as-code (e.g., Rego/OPA for dynamic permissions) + +Immediate Next Steps + +1. Models & Seeding + - Build APIKey, Airport, Factor, and BillingInfo models + - Add full User utility methods + - Implement org hierarchy logic and circular detection + - Write seeders for roles, orgs, and base permissions + +2. Authentication & Access + - Create login/logout flow with session checks + - Implement @permission_required on dummy route + - Protect routes using is_active_now + +3. Admin Panel + - Register models in Flask-Admin + - Add access checks + - Enable org-aware scoped views + + +Automatically log in the user after registering +Add CAPTCHA, email confirmation, etc. on registration - prob. means switching to usernames instead of email; allow both on login + +Implement RBAC (role-based access control) using decorators or dependency injection (if FastAPI). +Consider Soft Deletion + Audit Logging for emissions data edits. +For time-dependency in emissions models, use a valid-from/valid-to range in the schema. +Use Pydantic models (FastAPI) or Marshmallow (Flask) to enforce schema integrity. +Consider making the API Key logic pluggable, so you can later add scopes, revocation, or analytics. + + + + +Third draft +=========== + +Core Development Priorities for MVP +________________________________________ +1. Core Data Model Completion (High Priority) +Tasks: +1. Add APIKey Model +o Fields: UUID key, org association, creation date, expiration date, etc. +o Use db.Model to define the APIKey schema. +o Ensure secure handling of API keys, such as hashing the keys. +2. Add Factor Model +o Define a Factor model for multi-factor authentication (optional for MVP, but essential for future security improvements). +o Establish a managed_by_role relationship. +3. Add Airport Model +o Fields: airport_code, airport_name, location (city, country). +o Ensure this model is queryable for API usage. +4. Add Organization Billing Info Model +o Fields: planned amount, actual amount, due date, etc. +o Relate this to the Organization model via a foreign key. +5. Enhance User Model +o Methods: + check_password: To securely verify user passwords. + set_password: To securely hash and store user passwords. + is_active_now: Checks if the user is currently active based on start_date, end_date, and is_active. + has_permission: Check if the user has a specific permission. + get_organizations: Get a list of organizations the user is associated with. + can_view_organization: Verify if the user has access to a particular organization based on hierarchical permissions. +Priority: High – This will lay the foundation for all other features, including authentication, roles, and permissions. +________________________________________ +2. Permissions & Roles (High Priority) +Tasks: +1. Helper Method for Permission Check +o Implement a helper function to check if a role has a specific permission (role-permission logic). +2. Role Inheritance Rules/Templates +o Define role hierarchies or templates that determine inheritance. +o Data Admin → Dataset access permissions. +o Define roles like sys_admin, data_admin, user with specific access restrictions. +3. Role-based Access for Organization Hierarchy +o Implement logic to propagate permissions from parent → child organizations. +o Ensure recursive org tree handling to manage nested organizations. +o Build an admin takeover feature for sys-admin roles to control nested orgs. +4. Implement Permission Checks +o Add checks for whether a user can view/edit specific organizations based on the hierarchical relationship. +Priority: High – These features are critical for the organization management and user access control logic. +________________________________________ +3. Authentication & Access Control (High Priority) +Tasks: +1. Implement Login Views with Flask-Login +o Set up login and logout views using Flask-Login. +o Ensure password hashing with werkzeug.security. +2. Define and Implement Core Views (Registration) +o Implement registration functionality and wire up role-based access control (RBAC) using decorators. +3. Session Management & Security +o Implement session handling for active users. +o Ensure proper login required checks for protected routes. +4. Role-Based Access Control (RBAC) Decorators +o Implement decorators like @role_required or @permission_required to protect routes based on user roles and permissions. +Priority: High – These are essential for ensuring proper user management and access control. +________________________________________ +4. Admin Interface (Medium Priority) +Tasks: +1. Set up Flask-Admin +o Install and integrate Flask-Admin for user-friendly management. +o Register User, Organization, Role, Permission, Membership, and APIKey models in Flask-Admin. +2. Create Scoped Admin Views +o Restrict admin views based on organization and permissions. +o Allow admins to view and manage users and roles within their organization. +3. Seed Default Roles and Permissions +o Implement a seeding function to pre-load core roles (admin, user, etc.) and default permissions. +Priority: Medium – Admin interface is needed but not urgent for MVP launch, and can be done after the basic roles and permissions are set up. +________________________________________ +5. API Keys (Medium Priority) +Tasks: +1. Finalize APIKey Model +o Add UUID, expiration, rate-limiting, and other details. +o Implement CRUD operations to manage API keys. +2. Design API Key Management Views +o Create views for admins to generate and manage API keys. +o Allow API key creation and revocation. +Priority: Medium – API key management is important but can be built after the core model and authentication are in place. +________________________________________ +6. Organization Management (Medium Priority) +Tasks: +1. Implement Organization Hierarchy +o Define an organization model with parent-child relationships for recursive structure. +o Enable org creation, updating, deletion, and membership management. +2. Manage Org Memberships & Roles +o Build endpoints to manage users within organizations (assign roles, permissions, etc.). +o Allow admins to manage members at various levels (sys_admin, data_admin, etc.). +3. Organization Permissions +o Implement access control for users based on their organization hierarchy. +Priority: Medium – Core functionality, but can be tackled after user roles and permissions are functional. +________________________________________ +7. API Endpoints & Permissions (Medium Priority) +Tasks: +1. Define API Endpoints +o Focus on endpoints that query data for the organizations (e.g., GET requests for organizations, memberships, and roles). +o Exclude system management or interaction endpoints for now (as per your note). +2. Define Permissions for API Access +o Implement endpoint protection based on user roles and permissions. +Priority: Medium – API endpoints are crucial for data querying and integration with external systems but can be done after the basic models are in place. +________________________________________ +8. Testing (Ongoing Priority) +Tasks: +1. Unit Tests +o Test user model methods (check_password, set_password, is_active_now). +o Test role-based access control and permission checks. +o Test organization membership handling and recursive org tree access. +2. Integration Tests +o Test full API endpoint functionality (with and without valid API keys). +o Test login and session handling. +3. Testing User Authentication Flow +o Test login/logout, password verification, and protected views with Flask-Login. +4. Test Admin Views & Access Control +o Test admin views for restricted access, user creation, role management, etc. +Priority: High – Testing is essential to ensure no regressions and to confirm permissions and authentication work as expected. +________________________________________ +Additional Features (Post-MVP Considerations) +• Audit Logging: Implement change tracking for sensitive actions (user role changes, permissions updates, etc.). +• Caching for Permissions: Implement caching for permission checks to reduce the load on the database. +• Security Enhancements: Implement rate limiting, multi-factor authentication (optional), and session invalidation on role changes. +________________________________________ +Next Steps (Immediate Focus) +1. Core Data Model Completion: +o Finish the User, APIKey, Factor, Airport, and Organization models. +2. Permissions & Roles: +o Focus on implementing role hierarchies and role-based access control across organizations. +3. Authentication & Access Control: +o Set up Flask-Login for login/logout functionality. +o Implement RBAC decorators and session management. +4. Testing: +o Start with unit tests for the User model and authentication flow, and test role-based access control. + + + + +Fourth draft +============ + +🚧 PARTIALLY DONE / NEEDS REVIEW +🧠 Utility Methods +• get_organizations() and can_view_organization() not found ❌ +(should be implemented on User model) +🔐 Decorators +• @role_required and @permission_required not yet defined ❌ +(important for enforcing RBAC) +🏗️ Models Missing +• Factor ❌ +• Airport ❌ +• OrganizationBillingInfo ❌ +________________________________________ +🔄 REVISED ROADMAP (with Checked Status) +________________________________________ +🔥 PHASE 1: Core Models & Permissions +🚧 IN PROGRESS: +• Factor, Airport, BillingInfo models +• User.get_organizations(), User.can_view_organization() +• Org hierarchy traversal utilities +• Seed script for default roles/permissions +________________________________________ +🔐 PHASE 2: Role Logic & Propagation +• Role templates: sys_admin, org_owner, data_admin, etc. +• Role propagation down org tree +• Admin takeover for sys_admin +• Permission check logic via decorators +________________________________________ +🔑 PHASE 3: Authentication Flow +🚧 TO DO: +• is_active_now check on User +• Registration view (if needed) +• Decorators: @role_required, @permission_required +• Secure protected routes +________________________________________ +🛠️ PHASE 4: Admin Interface +• Flask-Admin setup +• Register models: User, Org, APIKey, etc. +• Scoped access + inline editing +________________________________________ +🔐 PHASE 5: API Keys +• Base model exists +• Add rate limiting, revocation, expiration logic +• API key views for org admins +________________________________________ +🌍 PHASE 6: Org Management +• Implement full org tree structure +• Circular checks +• Views for membership and org management +________________________________________ +🌐 PHASE 7: API Endpoints +• /api/airports and /api/factors +• Auth via API key + permission check +________________________________________ +🧪 PHASE 8: Testing (Ongoing) +• Unit tests for User, permission, APIKey +• Integration tests for auth, protected views +• Fixtures + seed data