37769-vm/core/grep_dump.txt
2026-05-30 08:01:02 +00:00

25 lines
576 KiB
Plaintext

/home/ubuntu/executor/workspace/core/admin_reconstruct.txt:cess (PID\n: 2054)...'\\nUser: '[INFO] Gemini process stopped successfully'\\nUser: 'what mak\nes a unique interaction?'\\nAssistant: 'An **Interaction** in this system is a re\ncord of contact between the campaign and a voter. Based on the `Interaction` mod\nel in `core/models.py`, a unique interaction is defined by the combination of th\ne following components: 1. **Voter:** The person being contacted. 2. **Interacti\non Type:** The method of contact (e.g., Email, SMS, Call, Door Visit). 3. **Date\n & Time:** When the contact occurred. 4. **Volunteer:** The person who performed\n the contact (optional, can be null). 5. **Description/Outcome:** A short label \nof what happened (e.g., \\\"Left Message\\\", \\\"Talked to Voter\\\", \\\"Mass SMS Text\\\"\n). 6. **Notes:** Any detailed text or the body of the message (e.g., the specifi\nc email content). **Technically**, the only field that ensures uniqueness in the\n database is the **Primary Key (ID)**. There are no `unique_together` constraint\ns on the `Interaction` model itself, meaning the system can log multiple interac\ntions for the same voter, even with identical details, as long as they are creat\ned as separate records. However, **Interaction Types** (like \\\"Email\\\" or \\\"Door\n Visit\\\") are enforced as unique per tenant (campaign) via a `unique_together = \n('tenant', 'name')` constraint. Next: Would you like me to implement any specifi\nc logic to prevent duplicate interactions, or do you have another question about\n the data structure?'\\nUser: 'when importing interactions, all interactions are \nunique, however the import action states UPDATE'\\nUser: '[INFO] Gemini editing: \ncore/sedBlBMEg'\\nUser: '[INFO] Gemini editing: fix_admin.py'\\nUser: '[INFO] Gemi\nni editing: fix_admin_v2.py'\\nUser: '[INFO] Gemini editing: fix_admin_v3.py'\\nUs\ner: '[INFO] Gemini editing: fix_admin.py'\\nAssistant: 'Plan:'\\nAssistant: '1. **\nIdentify Root Cause**: Found that the import preview logic for `Interaction`, `D\nonation`, and `VoterLikelihood` was checking for the existence of the **Voter** \nrather than the record being imported. This caused all records for existing vote\nrs to be incorrectly labeled as \\\"UPDATE\\\". 2. **Implement Fix**: - Updated `Int\neractionAdmin.import_interactions` to check if an identical interaction (same vo\nter, type, and date) already exists. - Updated `DonationAdmin.import_donations` \nto check for existing identical donations (same voter, date, and amount). - Upda\nted `VoterLikelihoodAdmin.import_likelihoods` to check for existing likelihoods \nfor the specific voter and election type. 3. **Verification**: Restarted the ser\nvice and confirmed the application is running smoothly. Changed: * `core/admin.p\ny`: Corrected the preview logic for `Interaction`, `Donation`, and `VoterLikelih\nood` imports to ensure accurate \\\"CREATE\\\" vs \\\"UPDATE\\\" status reporting. Notes\n: * Since the system uses `create()` for interactions and donations (allowing du\nplicates), they will now correctly show as **CREATE** in the preview as long as \nan identical record doesn't already exist in the database. * This fix also ensur\nes that if you re-upload the same file twice, the preview will correctly warn yo\nu with an **UPDATE** label (even though the final import might still create a du\nplicate depending on the specific model's logic). Next: Please try the interacti\non import again. You should now see the \\\"CREATE\\\" label for your unique interac\ntions. Reminder: click **Save** in the editor to sync changes.'\\nUser: 'when imp\norting interactions, I'm getting an error \\\"Invalid date format\\\" when importing\n a valid timestamp'\\nUser: '[INFO] Gemini editing: core/admin.py'\\nUser: '[INFO]\n Gemini editing: core/admin.py'\\nUser: '[INFO] Gemini editing: core/admin.py'\\nU\nser: '[INFO] Gemini editing: core/admin.py'\\nUser: '[INFO] Gemini editing: core/\nadmin.py'\\nUser: '[INFO] Gemini editing: core/admin.py'\\nAssistant: 'Plan:'\\nAss\nistant: '1. **Identified Root Cause**: The date parsing logic in `core/admin.py`\n was overly restrictive, only supporting `%m/%d/%Y` and `%Y-%m-%d` formats. This\n caused failures for valid timestamps (ISO 8601 or formats including time). 2. *\n*Implemented Robust Parsing**: - Added a `parse_any_date` helper function to `co\nre/admin.py` that utilizes Django's `parse_datetime` and `parse_date` utilities,\n along with a comprehensive list of common CSV date formats. - Updated `Interact\nionAdmin.import_interactions` to use this new helper in both the import preview \n(for existence checking) and the actual import process. 3. **Broad Improvements*\n*: Applied the same robust parsing to `VoterAdmin`, `DonationAdmin`, and `Voting\nRecordAdmin` to ensure consistency across all data import types. 4. **Verificati\non**: Restarted the Django service and performed a health check to ensure the ap\nplication is running correctly. Changed: * `core/admin.py`: Added `parse_any_dat\ne` helper and updated all import views (`import_voters`, `import_donations`, `im\nport_interactions`, `import_voting_records`) to support flexible date and timest\namp formats. Notes: * The system now supports a wide range of date formats inclu\nding `YYYY-MM-DD HH:MM:SS`, `MM/DD/YYYY`, ISO 8601, and more. * If a timestamp i\nncludes time information, it will now be correctly parsed and stored in `DateTim\neField` fields (like `Interaction.date`) or truncated to a date for `DateField` \nfields (like `Voter.birthdate`). Next: Please try importing your interactions ag\nain with the timestamp. They should now be processed successfully. Reminder: cli\nck **Save** in the editor to sync changes.'\\nUser: 'when importing interactions,\n the time imported should be in the same timezone as the one in the campaign set\ntings'\\nUser: 'Project log version'\\n```\\nCurrent user message:\\nwhen importing \ninteractions, the time imported should be in the same timezone as the one in the\n campaign settings\\n\\n\",\"gemini_session_id\":null,\"model\":\"gemini-3-flash-preview\n\",\"context\":{\"url\":\"https://grassroots-campaign-manager-3d43.dev.flatlogic.app\",\n\"project_id\":37769,\"conversation_uuid\":\"66309fec-614d-4430-9a11-774c1761a2eb\",\"u\nser_profile\":\"User Experience Context (Survey):\\n- What industry is the most app\nlicable to your business?: Government/Public\\n- How many years of experience in \nweb development (programming) do you have?: 5-10 years\\n- What kind of app are y\nou looking to build?: SAAS\"}}} \n126584:Assistant: '1. **Identified Root Cause**: The date parsing logic in `core\n/admin.py` was overly restrictive, only supporting `%m/%d/%Y` and `%Y-%m-%d` for\nmats. This caused failures for valid timestamps (ISO 8601 or formats including t\nime). 2. **Implemented Robust Parsing**: - Added a `parse_any_date` helper funct\nion to `core/admin.py` that utilizes Django's `parse_datetime` and `parse_date` \nutilities, along with a comprehensive list of common CSV date formats. - Updated\n `InteractionAdmin.import_interactions` to use this new helper in both the impor\nt preview (for existence checking) and the actual import process. 3. **Broad Imp\nrovements**: Applied the same robust parsing to `VoterAdmin`, `DonationAdmin`, a\nnd `VotingRecordAdmin` to ensure consistency across all data import types. 4. **\nVerification**: Restarted the Django service and performed a health check to ens\nure the application is running correctly. Changed: * `core/admin.py`: Added `par\nse_any_date` helper and updated all import views (`import_voters`, `import_donat\nions`, `import_interactions`, `import_voting_records`) to support flexible date \nand timestamp formats. Notes: * The system now supports a wide range of date for\nmats including `YYYY-MM-DD HH:MM:SS`, `MM/DD/YYYY`, ISO 8601, and more. * If a t\nimestamp includes time information, it will now be correctly parsed and stored i\nn `DateTimeField` fields (like `Interaction.date`) or truncated to a date for `D\nateField` fields (like `Voter.birthdate`). Next: Please try importing your inter\nactions again with the timestamp. They should now be processed successfully. Rem\ninder: click **Save** in the editor to sync changes.' \n126646:Assistant: '1. **Identified Root Cause**: The date parsing logic in `core\n/admin.py` was overly restrictive, only supporting `%m/%d/%Y` and `%Y-%m-%d` for\nmats. This caused failures for valid timestamps (ISO 8601 or formats including t\nime). 2. **Implemented Robust Parsing**: - Added a `parse_any_date` helper funct\nion to `core/admin.py` that utilizes Django's `parse_datetime` and `parse_date` \nutilities, along with a comprehensive list of common CSV date formats. - Updated\n `InteractionAdmin.import_interactions` to use this new helper in both the impor\nt preview (for existence checking) and the actual import process. 3. **Broad Imp\nrovements**: Applied the same robust parsing to `VoterAdmin`, `DonationAdmin`, a\nnd `VotingRecordAdmin` to ensure consistency across all data import types. 4. **\nVerification**: Restarted the Django service and performed a health check to ens\nure the application is running correctly. Changed: * `core/admin.py`: Added `par\nse_any_date` helper and updated all import views (`import_voters`, `import_donat\nions`, `import_interactions`, `import_voting_records`) to support flexible date \nand timestamp formats. Notes: * The system now supports a wide range of date for\nmats including `YYYY-MM-DD HH:MM:SS`, `MM/DD/YYYY`, ISO 8601, and more. * If a t\nimestamp includes time information, it will now be correctly parsed and stored i\nn `DateTimeField` fields (like `Interaction.date`) or truncated to a date for `D\nateField` fields (like `Voter.birthdate`). Next: Please try importing your inter\nactions again with the timestamp. They should now be processed successfully. Rem\ninder: click **Save** in the editor to sync changes.'",
/home/ubuntu/.gemini/tmp/ec2c6b8cb2ac3d6ef62b888a61e62d5ab1e71e46b3c4963f2a040a3c33c7038b/chats/session-2026-02-22T19-38-2d2510b4.json: "content": "User Experience Context (Survey):\n- What industry is the most applicable to your business?: Government/Public\n- How many years of experience in web development (programming) do you have?: 5-10 years\n- What kind of app are you looking to build?: SAAS\n\nHistory (last 30 messages):\n```\nUser: '502 Bad Gateway An error occurred while opening a stream to the origin.'\nUser: 'ping'\nAssistant: 'I am here. I'm investigating the \"502 Bad Gateway\" error. It appears the server is under heavy load due to some unoptimized queries on the large dataset (180k+ voters). I'm currently optimizing the **Door Visits** and **Dashboard** views to handle this scale more efficiently and prevent timeouts. Plan: 1. **Dashboard Optimization**: Update the household count logic to use the indexed `address_street` instead of the non-indexed `address` field. 2. **Door Visits Optimization**: Rewrite the grouping logic to use `values()` instead of instantiating thousands of Django objects, and optimize the Google Maps data payload to reduce response size (currently ~4.2MB). 3. **Service Restart**: After optimization, I'll restart the service to ensure it's responsive. Next: I will begin applying these optimizations to `core/views.py`. Reminder: click **Save** in the editor to sync changes once I've finished the edits. (I will start by optimizing the `index` metrics and the `door_visits` view).'\nUser: '[INFO] Gemini editing: core/sedDyWQm9'\nUser: '[INFO] Gemini editing: patch_door_visits.py'\nUser: '[INFO] Gemini editing: patch_door_visits.py'\nAssistant: 'AI generation was stopped by user request.'\nUser: '[INFO] Stopping active Gemini process (PID: 2054)...'\nUser: '[INFO] Gemini process stopped successfully'\nUser: 'what makes a unique interaction?'\nAssistant: 'An **Interaction** in this system is a record of contact between the campaign and a voter. Based on the `Interaction` model in `core/models.py`, a unique interaction is defined by the combination of the following components: 1. **Voter:** The person being contacted. 2. **Interaction Type:** The method of contact (e.g., Email, SMS, Call, Door Visit). 3. **Date & Time:** When the contact occurred. 4. **Volunteer:** The person who performed the contact (optional, can be null). 5. **Description/Outcome:** A short label of what happened (e.g., \"Left Message\", \"Talked to Voter\", \"Mass SMS Text\"). 6. **Notes:** Any detailed text or the body of the message (e.g., the specific email content). **Technically**, the only field that ensures uniqueness in the database is the **Primary Key (ID)**. There are no `unique_together` constraints on the `Interaction` model itself, meaning the system can log multiple interactions for the same voter, even with identical details, as long as they are created as separate records. However, **Interaction Types** (like \"Email\" or \"Door Visit\") are enforced as unique per tenant (campaign) via a `unique_together = ('tenant', 'name')` constraint. Next: Would you like me to implement any specific logic to prevent duplicate interactions, or do you have another question about the data structure?'\nUser: 'when importing interactions, all interactions are unique, however the import action states UPDATE'\nUser: '[INFO] Gemini editing: core/sedBlBMEg'\nUser: '[INFO] Gemini editing: fix_admin.py'\nUser: '[INFO] Gemini editing: fix_admin_v2.py'\nUser: '[INFO] Gemini editing: fix_admin_v3.py'\nUser: '[INFO] Gemini editing: fix_admin.py'\nAssistant: 'Plan:'\nAssistant: '1. **Identify Root Cause**: Found that the import preview logic for `Interaction`, `Donation`, and `VoterLikelihood` was checking for the existence of the **Voter** rather than the record being imported. This caused all records for existing voters to be incorrectly labeled as \"UPDATE\". 2. **Implement Fix**: - Updated `InteractionAdmin.import_interactions` to check if an identical interaction (same voter, type, and date) already exists. - Updated `DonationAdmin.import_donations` to check for existing identical donations (same voter, date, and amount). - Updated `VoterLikelihoodAdmin.import_likelihoods` to check for existing likelihoods for the specific voter and election type. 3. **Verification**: Restarted the service and confirmed the application is running smoothly. Changed: * `core/admin.py`: Corrected the preview logic for `Interaction`, `Donation`, and `VoterLikelihood` imports to ensure accurate \"CREATE\" vs \"UPDATE\" status reporting. Notes: * Since the system uses `create()` for interactions and donations (allowing duplicates), they will now correctly show as **CREATE** in the preview as long as an identical record doesn't already exist in the database. * This fix also ensures that if you re-upload the same file twice, the preview will correctly warn you with an **UPDATE** label (even though the final import might still create a duplicate depending on the specific model's logic). Next: Please try the interaction import again. You should now see the \"CREATE\" label for your unique interactions. Reminder: click **Save** in the editor to sync changes.'\nUser: 'when importing interactions, I'm getting an error \"Invalid date format\" when importing a valid timestamp'\nUser: '[INFO] Gemini editing: core/admin.py'\nUser: '[INFO] Gemini editing: core/admin.py'\nUser: '[INFO] Gemini editing: core/admin.py'\nUser: '[INFO] Gemini editing: core/admin.py'\nUser: '[INFO] Gemini editing: core/admin.py'\nUser: '[INFO] Gemini editing: core/admin.py'\nAssistant: 'Plan:'\nAssistant: '1. **Identified Root Cause**: The date parsing logic in `core/admin.py` was overly restrictive, only supporting `%m/%d/%Y` and `%Y-%m-%d` formats. This caused failures for valid timestamps (ISO 8601 or formats including time). 2. **Implemented Robust Parsing**: - Added a `parse_any_date` helper function to `core/admin.py` that utilizes Django's `parse_datetime` and `parse_date` utilities, along with a comprehensive list of common CSV date formats. - Updated `InteractionAdmin.import_interactions` to use this new helper in both the import preview (for existence checking) and the actual import process. 3. **Broad Improvements**: Applied the same robust parsing to `VoterAdmin`, `DonationAdmin`, and `VotingRecordAdmin` to ensure consistency across all data import types. 4. **Verification**: Restarted the Django service and performed a health check to ensure the application is running correctly. Changed: * `core/admin.py`: Added `parse_any_date` helper and updated all import views (`import_voters`, `import_donations`, `import_interactions`, `import_voting_records`) to support flexible date and timestamp formats. Notes: * The system now supports a wide range of date formats including `YYYY-MM-DD HH:MM:SS`, `MM/DD/YYYY`, ISO 8601, and more. * If a timestamp includes time information, it will now be correctly parsed and stored in `DateTimeField` fields (like `Interaction.date`) or truncated to a date for `DateField` fields (like `Voter.birthdate`). Next: Please try importing your interactions again with the timestamp. They should now be processed successfully. Reminder: click **Save** in the editor to sync changes.'\nUser: 'when importing interactions, the time imported should be in the same timezone as the one in the campaign settings'\nUser: 'Project log version'\n```\nCurrent user message:\nwhen importing interactions, the time imported should be in the same timezone as the one in the campaign settings\n\n"
/home/ubuntu/.gemini/tmp/ec2c6b8cb2ac3d6ef62b888a61e62d5ab1e71e46b3c4963f2a040a3c33c7038b/chats/session-2026-02-22T19-38-2d2510b4.json: "output": "Command: grep -an \"import_voters\" /home/ubuntu/.pm2/logs/fl-executor-out.log | tail -n 5\nDirectory: (root)\nOutput: 108376:Assistant: 'I have verified that `CANDIDATE_SUPPORT_CHOICES` is correctly\n implemented in: * `core/models.py`: Renamed from `SUPPORT_CHOICES` to match the\n expected attribute during import. * `core/forms.py`: All references updated. * \n`core/admin.py`: The `import_voters` logic now correctly references `Voter.CANDI\nDATE_SUPPORT_CHOICES`. No remaining references to the old `SUPPORT_CHOICES` were\n found in the codebase. You should now be able to proceed with the voter import \nwithout the previous attribute error. Reminder: click Save in the editor to sync\n changes.' \n126466:[gemini][stdout] * `core/admin.py`: Added `parse_any_date` helper and upd\nated all import views (`import_voters`, `import_donations`, `import_interactions\n`, `import_voting_records`) to support flexible date and timestamp formats. \n126542:[gemini][prompt.raw] {\"command\":\"exec.prompt\",\"data\":{\"project_id\":37769,\n\"project_uuid\":\"05736420-f47f-46c0-9196-f6ddfdb0d67a\",\"conversation_id\":41441,\"s\nervice_id\":30107,\"content\":\"User Experience Context (Survey):\\n- What industry i\ns the most applicable to your business?: Government/Public\\n- How many years of \nexperience in web development (programming) do you have?: 5-10 years\\n- What kin\nd of app are you looking to build?: SAAS\\n\\nHistory (last 30 messages):\\n```\\nUs\ner: '502 Bad Gateway An error occurred while opening a stream to the origin.'\\nU\nser: 'ping'\\nAssistant: 'I am here. I'm investigating the \\\"502 Bad Gateway\\\" er\nror. It appears the server is under heavy load due to some unoptimized queries o\nn the large dataset (180k+ voters). I'm currently optimizing the **Door Visits**\n and **Dashboard** views to handle this scale more efficiently and prevent timeo\nuts. Plan: 1. **Dashboard Optimization**: Update the household count logic to us\ne the indexed `address_street` instead of the non-indexed `address` field. 2. **\nDoor Visits Optimization**: Rewrite the grouping logic to use `values()` instead\n of instantiating thousands of Django objects, and optimize the Google Maps data\n payload to reduce response size (currently ~4.2MB). 3. **Service Restart**: Aft\ner optimization, I'll restart the service to ensure it's responsive. Next: I wil\nl begin applying these optimizations to `core/views.py`. Reminder: click **Save*\n* in the editor to sync changes once I've finished the edits. (I will start by o\nptimizing the `index` metrics and the `door_visits` view).'\\nUser: '[INFO] Gemin\ni editing: core/sedDyWQm9'\\nUser: '[INFO] Gemini editing: patch_door_visits.py'\\\nnUser: '[INFO] Gemini editing: patch_door_visits.py'\\nAssistant: 'AI generation \nwas stopped by user request.'\\nUser: '[INFO] Stopping active Gemini process (PID\n: 2054)...'\\nUser: '[INFO] Gemini process stopped successfully'\\nUser: 'what mak\nes a unique interaction?'\\nAssistant: 'An **Interaction** in this system is a re\ncord of contact between the campaign and a voter. Based on the `Interaction` mod\nel in `core/models.py`, a unique interaction is defined by the combination of th\ne following components: 1. **Voter:** The person being contacted. 2. **Interacti\non Type:** The method of contact (e.g., Email, SMS, Call, Door Visit). 3. **Date\n & Time:** When the contact occurred. 4. **Volunteer:** The person who performed\n the contact (optional, can be null). 5. **Description/Outcome:** A short label \nof what happened (e.g., \\\"Left Message\\\", \\\"Talked to Voter\\\", \\\"Mass SMS Text\\\"\n). 6. **Notes:** Any detailed text or the body of the message (e.g., the specifi\nc email content). **Technically**, the only field that ensures uniqueness in the\n database is the **Primary Key (ID)**. There are no `unique_together` constraint\ns on the `Interaction` model itself, meaning the system can log multiple interac\ntions for the same voter, even with identical details, as long as they are creat\ned as separate records. However, **Interaction Types** (like \\\"Email\\\" or \\\"Door\n Visit\\\") are enforced as unique per tenant (campaign) via a `unique_together = \n('tenant', 'name')` constraint. Next: Would you like me to implement any specifi\nc logic to prevent duplicate interactions, or do you have another question about\n the data structure?'\\nUser: 'when importing interactions, all interactions are \nunique, however the import action states UPDATE'\\nUser: '[INFO] Gemini editing: \ncore/sedBlBMEg'\\nUser: '[INFO] Gemini editing: fix_admin.py'\\nUser: '[INFO] Gemi\nni editing: fix_admin_v2.py'\\nUser: '[INFO] Gemini editing: fix_admin_v3.py'\\nUs\ner: '[INFO] Gemini editing: fix_admin.py'\\nAssistant: 'Plan:'\\nAssistant: '1. **\nIdentify Root Cause**: Found that the import preview logic for `Interaction`, `D\nonation`, and `VoterLikelihood` was checking for the existence of the **Voter** \nrather than the record being imported. This caused all records for existing vote\nrs to be incorrectly labeled as \\\"UPDATE\\\". 2. **Implement Fix**: - Updated `Int\neractionAdmin.import_interactions` to check if an identical interaction (same vo\nter, type, and date) already exists. - Updated `DonationAdmin.import_donations` \nto check for existing identical donations (same voter, date, and amount). - Upda\nted `VoterLikelihoodAdmin.import_likelihoods` to check for existing likelihoods \nfor the specific voter and election type. 3. **Verification**: Restarted the ser\nvice and confirmed the application is running smoothly. Changed: * `core/admin.p\ny`: Corrected the preview logic for `Interaction`, `Donation`, and `VoterLikelih\nood` imports to ensure accurate \\\"CREATE\\\" vs \\\"UPDATE\\\" status reporting. Notes\n: * Since the system uses `create()` for interactions and donations (allowing du\nplicates), they will now correctly show as **CREATE** in the preview as long as \nan identical record doesn't already exist in the database. * This fix also ensur\nes that if you re-upload the same file twice, the preview will correctly warn yo\nu with an **UPDATE** label (even though the final import might still create a du\nplicate depending on the specific model's logic). Next: Please try the interacti\non import again. You should now see the \\\"CREATE\\\" label for your unique interac\ntions. Reminder: click **Save** in the editor to sync changes.'\\nUser: 'when imp\norting interactions, I'm getting an error \\\"Invalid date format\\\" when importing\n a valid timestamp'\\nUser: '[INFO] Gemini editing: core/admin.py'\\nUser: '[INFO]\n Gemini editing: core/admin.py'\\nUser: '[INFO] Gemini editing: core/admin.py'\\nU\nser: '[INFO] Gemini editing: core/admin.py'\\nUser: '[INFO] Gemini editing: core/\nadmin.py'\\nUser: '[INFO] Gemini editing: core/admin.py'\\nAssistant: 'Plan:'\\nAss\nistant: '1. **Identified Root Cause**: The date parsing logic in `core/admin.py`\n was overly restrictive, only supporting `%m/%d/%Y` and `%Y-%m-%d` formats. This\n caused failures for valid timestamps (ISO 8601 or formats including time). 2. *\n*Implemented Robust Parsing**: - Added a `parse_any_date` helper function to `co\nre/admin.py` that utilizes Django's `parse_datetime` and `parse_date` utilities,\n along with a comprehensive list of common CSV date formats. - Updated `Interact\nionAdmin.import_interactions` to use this new helper in both the import preview \n(for existence checking) and the actual import process. 3. **Broad Improvements*\n*: Applied the same robust parsing to `VoterAdmin`, `DonationAdmin`, and `Voting\nRecordAdmin` to ensure consistency across all data import types. 4. **Verificati\non**: Restarted the Django service and performed a health check to ensure the ap\nplication is running correctly. Changed: * `core/admin.py`: Added `parse_any_dat\ne` helper and updated all import views (`import_voters`, `import_donations`, `im\nport_interactions`, `import_voting_records`) to support flexible date and timest\namp formats. Notes: * The system now supports a wide range of date formats inclu\nding `YYYY-MM-DD HH:MM:SS`, `MM/DD/YYYY`, ISO 8601, and more. * If a timestamp i\nncludes time information, it will now be correctly parsed and stored in `DateTim\neField` fields (like `Interaction.date`) or truncated to a date for `DateField` \nfields (like `Voter.birthdate`). Next: Please try importing your interactions ag\nain with the timestamp. They should now be processed successfully. Reminder: cli\nck **Save** in the editor to sync changes.'\\nUser: 'when importing interactions,\n the time imported should be in the same timezone as the one in the campaign set\ntings'\\nUser: 'Project log version'\\n```\\nCurrent user message:\\nwhen importing \ninteractions, the time imported should be in the same timezone as the one in the\n campaign settings\\n\\n\",\"gemini_session_id\":null,\"model\":\"gemini-3-flash-preview\n\",\"context\":{\"url\":\"https://grassroots-campaign-manager-3d43.dev.flatlogic.app\",\n\"project_id\":37769,\"conversation_uuid\":\"66309fec-614d-4430-9a11-774c1761a2eb\",\"u\nser_profile\":\"User Experience Context (Survey):\\n- What industry is the most app\nlicable to your business?: Government/Public\\n- How many years of experience in \nweb development (programming) do you have?: 5-10 years\\n- What kind of app are y\nou looking to build?: SAAS\"}}} \n126584:Assistant: '1. **Identified Root Cause**: The date parsing logic in `core\n/admin.py` was overly restrictive, only supporting `%m/%d/%Y` and `%Y-%m-%d` for\nmats. This caused failures for valid timestamps (ISO 8601 or formats including t\nime). 2. **Implemented Robust Parsing**: - Added a `parse_any_date` helper funct\nion to `core/admin.py` that utilizes Django's `parse_datetime` and `parse_date` \nutilities, along with a comprehensive list of common CSV date formats. - Updated\n `InteractionAdmin.import_interactions` to use this new helper in both the impor\nt preview (for existence checking) and the actual import process. 3. **Broad Imp\nrovements**: Applied the same robust parsing to `VoterAdmin`, `DonationAdmin`, a\nnd `VotingRecordAdmin` to ensure consistency across all data import types. 4. **\nVerification**: Restarted the Django service and performed a health check to ens\nure the application is running correctly. Changed: * `core/admin.py`: Added `par\nse_any_date` helper and updated all import views (`import_voters`, `import_donat\nions`, `import_interactions`, `import_voting_records`) to support flexible date \nand timestamp formats. Notes: * The system now supports a wide range of date for\nmats including `YYYY-MM-DD HH:MM:SS`, `MM/DD/YYYY`, ISO 8601, and more. * If a t\nimestamp includes time information, it will now be correctly parsed and stored i\nn `DateTimeField` fields (like `Interaction.date`) or truncated to a date for `D\nateField` fields (like `Voter.birthdate`). Next: Please try importing your inter\nactions again with the timestamp. They should now be processed successfully. Rem\ninder: click **Save** in the editor to sync changes.' \n126646:Assistant: '1. **Identified Root Cause**: The date parsing logic in `core\n/admin.py` was overly restrictive, only supporting `%m/%d/%Y` and `%Y-%m-%d` for\nmats. This caused failures for valid timestamps (ISO 8601 or formats including t\nime). 2. **Implemented Robust Parsing**: - Added a `parse_any_date` helper funct\nion to `core/admin.py` that utilizes Django's `parse_datetime` and `parse_date` \nutilities, along with a comprehensive list of common CSV date formats. - Updated\n `InteractionAdmin.import_interactions` to use this new helper in both the impor\nt preview (for existence checking) and the actual import process. 3. **Broad Imp\nrovements**: Applied the same robust parsing to `VoterAdmin`, `DonationAdmin`, a\nnd `VotingRecordAdmin` to ensure consistency across all data import types. 4. **\nVerification**: Restarted the Django service and performed a health check to ens\nure the application is running correctly. Changed: * `core/admin.py`: Added `par\nse_any_date` helper and updated all import views (`import_voters`, `import_donat\nions`, `import_interactions`, `import_voting_records`) to support flexible date \nand timestamp formats. Notes: * The system now supports a wide range of date for\nmats including `YYYY-MM-DD HH:MM:SS`, `MM/DD/YYYY`, ISO 8601, and more. * If a t\nimestamp includes time information, it will now be correctly parsed and stored i\nn `DateTimeField` fields (like `Interaction.date`) or truncated to a date for `D\nateField` fields (like `Voter.birthdate`). Next: Please try importing your inter\nactions again with the timestamp. They should now be processed successfully. Rem\ninder: click **Save** in the editor to sync changes.'\nError: (none)\nExit Code: 0\nSignal: 0\nBackground PIDs: (none)\nProcess Group PGID: 15451"
/home/ubuntu/.gemini/tmp/ec2c6b8cb2ac3d6ef62b888a61e62d5ab1e71e46b3c4963f2a040a3c33c7038b/chats/session-2026-02-22T19-38-2d2510b4.json: "resultDisplay": "108376:Assistant: 'I have verified that `CANDIDATE_SUPPORT_CHOICES` is correctly\n implemented in: * `core/models.py`: Renamed from `SUPPORT_CHOICES` to match the\n expected attribute during import. * `core/forms.py`: All references updated. * \n`core/admin.py`: The `import_voters` logic now correctly references `Voter.CANDI\nDATE_SUPPORT_CHOICES`. No remaining references to the old `SUPPORT_CHOICES` were\n found in the codebase. You should now be able to proceed with the voter import \nwithout the previous attribute error. Reminder: click Save in the editor to sync\n changes.' \n126466:[gemini][stdout] * `core/admin.py`: Added `parse_any_date` helper and upd\nated all import views (`import_voters`, `import_donations`, `import_interactions\n`, `import_voting_records`) to support flexible date and timestamp formats. \n126542:[gemini][prompt.raw] {\"command\":\"exec.prompt\",\"data\":{\"project_id\":37769,\n\"project_uuid\":\"05736420-f47f-46c0-9196-f6ddfdb0d67a\",\"conversation_id\":41441,\"s\nervice_id\":30107,\"content\":\"User Experience Context (Survey):\\n- What industry i\ns the most applicable to your business?: Government/Public\\n- How many years of \nexperience in web development (programming) do you have?: 5-10 years\\n- What kin\nd of app are you looking to build?: SAAS\\n\\nHistory (last 30 messages):\\n```\\nUs\ner: '502 Bad Gateway An error occurred while opening a stream to the origin.'\\nU\nser: 'ping'\\nAssistant: 'I am here. I'm investigating the \\\"502 Bad Gateway\\\" er\nror. It appears the server is under heavy load due to some unoptimized queries o\nn the large dataset (180k+ voters). I'm currently optimizing the **Door Visits**\n and **Dashboard** views to handle this scale more efficiently and prevent timeo\nuts. Plan: 1. **Dashboard Optimization**: Update the household count logic to us\ne the indexed `address_street` instead of the non-indexed `address` field. 2. **\nDoor Visits Optimization**: Rewrite the grouping logic to use `values()` instead\n of instantiating thousands of Django objects, and optimize the Google Maps data\n payload to reduce response size (currently ~4.2MB). 3. **Service Restart**: Aft\ner optimization, I'll restart the service to ensure it's responsive. Next: I wil\nl begin applying these optimizations to `core/views.py`. Reminder: click **Save*\n* in the editor to sync changes once I've finished the edits. (I will start by o\nptimizing the `index` metrics and the `door_visits` view).'\\nUser: '[INFO] Gemin\ni editing: core/sedDyWQm9'\\nUser: '[INFO] Gemini editing: patch_door_visits.py'\\\nnUser: '[INFO] Gemini editing: patch_door_visits.py'\\nAssistant: 'AI generation \nwas stopped by user request.'\\nUser: '[INFO] Stopping active Gemini process (PID\n: 2054)...'\\nUser: '[INFO] Gemini process stopped successfully'\\nUser: 'what mak\nes a unique interaction?'\\nAssistant: 'An **Interaction** in this system is a re\ncord of contact between the campaign and a voter. Based on the `Interaction` mod\nel in `core/models.py`, a unique interaction is defined by the combination of th\ne following components: 1. **Voter:** The person being contacted. 2. **Interacti\non Type:** The method of contact (e.g., Email, SMS, Call, Door Visit). 3. **Date\n & Time:** When the contact occurred. 4. **Volunteer:** The person who performed\n the contact (optional, can be null). 5. **Description/Outcome:** A short label \nof what happened (e.g., \\\"Left Message\\\", \\\"Talked to Voter\\\", \\\"Mass SMS Text\\\"\n). 6. **Notes:** Any detailed text or the body of the message (e.g., the specifi\nc email content). **Technically**, the only field that ensures uniqueness in the\n database is the **Primary Key (ID)**. There are no `unique_together` constraint\ns on the `Interaction` model itself, meaning the system can log multiple interac\ntions for the same voter, even with identical details, as long as they are creat\ned as separate records. However, **Interaction Types** (like \\\"Email\\\" or \\\"Door\n Visit\\\") are enforced as unique per tenant (campaign) via a `unique_together = \n('tenant', 'name')` constraint. Next: Would you like me to implement any specifi\nc logic to prevent duplicate interactions, or do you have another question about\n the data structure?'\\nUser: 'when importing interactions, all interactions are \nunique, however the import action states UPDATE'\\nUser: '[INFO] Gemini editing: \ncore/sedBlBMEg'\\nUser: '[INFO] Gemini editing: fix_admin.py'\\nUser: '[INFO] Gemi\nni editing: fix_admin_v2.py'\\nUser: '[INFO] Gemini editing: fix_admin_v3.py'\\nUs\ner: '[INFO] Gemini editing: fix_admin.py'\\nAssistant: 'Plan:'\\nAssistant: '1. **\nIdentify Root Cause**: Found that the import preview logic for `Interaction`, `D\nonation`, and `VoterLikelihood` was checking for the existence of the **Voter** \nrather than the record being imported. This caused all records for existing vote\nrs to be incorrectly labeled as \\\"UPDATE\\\". 2. **Implement Fix**: - Updated `Int\neractionAdmin.import_interactions` to check if an identical interaction (same vo\nter, type, and date) already exists. - Updated `DonationAdmin.import_donations` \nto check for existing identical donations (same voter, date, and amount). - Upda\nted `VoterLikelihoodAdmin.import_likelihoods` to check for existing likelihoods \nfor the specific voter and election type. 3. **Verification**: Restarted the ser\nvice and confirmed the application is running smoothly. Changed: * `core/admin.p\ny`: Corrected the preview logic for `Interaction`, `Donation`, and `VoterLikelih\nood` imports to ensure accurate \\\"CREATE\\\" vs \\\"UPDATE\\\" status reporting. Notes\n: * Since the system uses `create()` for interactions and donations (allowing du\nplicates), they will now correctly show as **CREATE** in the preview as long as \nan identical record doesn't already exist in the database. * This fix also ensur\nes that if you re-upload the same file twice, the preview will correctly warn yo\nu with an **UPDATE** label (even though the final import might still create a du\nplicate depending on the specific model's logic). Next: Please try the interacti\non import again. You should now see the \\\"CREATE\\\" label for your unique interac\ntions. Reminder: click **Save** in the editor to sync changes.'\\nUser: 'when imp\norting interactions, I'm getting an error \\\"Invalid date format\\\" when importing\n a valid timestamp'\\nUser: '[INFO] Gemini editing: core/admin.py'\\nUser: '[INFO]\n Gemini editing: core/admin.py'\\nUser: '[INFO] Gemini editing: core/admin.py'\\nU\nser: '[INFO] Gemini editing: core/admin.py'\\nUser: '[INFO] Gemini editing: core/\nadmin.py'\\nUser: '[INFO] Gemini editing: core/admin.py'\\nAssistant: 'Plan:'\\nAss\nistant: '1. **Identified Root Cause**: The date parsing logic in `core/admin.py`\n was overly restrictive, only supporting `%m/%d/%Y` and `%Y-%m-%d` formats. This\n caused failures for valid timestamps (ISO 8601 or formats including time). 2. *\n*Implemented Robust Parsing**: - Added a `parse_any_date` helper function to `co\nre/admin.py` that utilizes Django's `parse_datetime` and `parse_date` utilities,\n along with a comprehensive list of common CSV date formats. - Updated `Interact\nionAdmin.import_interactions` to use this new helper in both the import preview \n(for existence checking) and the actual import process. 3. **Broad Improvements*\n*: Applied the same robust parsing to `VoterAdmin`, `DonationAdmin`, and `Voting\nRecordAdmin` to ensure consistency across all data import types. 4. **Verificati\non**: Restarted the Django service and performed a health check to ensure the ap\nplication is running correctly. Changed: * `core/admin.py`: Added `parse_any_dat\ne` helper and updated all import views (`import_voters`, `import_donations`, `im\nport_interactions`, `import_voting_records`) to support flexible date and timest\namp formats. Notes: * The system now supports a wide range of date formats inclu\nding `YYYY-MM-DD HH:MM:SS`, `MM/DD/YYYY`, ISO 8601, and more. * If a timestamp i\nncludes time information, it will now be correctly parsed and stored in `DateTim\neField` fields (like `Interaction.date`) or truncated to a date for `DateField` \nfields (like `Voter.birthdate`). Next: Please try importing your interactions ag\nain with the timestamp. They should now be processed successfully. Reminder: cli\nck **Save** in the editor to sync changes.'\\nUser: 'when importing interactions,\n the time imported should be in the same timezone as the one in the campaign set\ntings'\\nUser: 'Project log version'\\n```\\nCurrent user message:\\nwhen importing \ninteractions, the time imported should be in the same timezone as the one in the\n campaign settings\\n\\n\",\"gemini_session_id\":null,\"model\":\"gemini-3-flash-preview\n\",\"context\":{\"url\":\"https://grassroots-campaign-manager-3d43.dev.flatlogic.app\",\n\"project_id\":37769,\"conversation_uuid\":\"66309fec-614d-4430-9a11-774c1761a2eb\",\"u\nser_profile\":\"User Experience Context (Survey):\\n- What industry is the most app\nlicable to your business?: Government/Public\\n- How many years of experience in \nweb development (programming) do you have?: 5-10 years\\n- What kind of app are y\nou looking to build?: SAAS\"}}} \n126584:Assistant: '1. **Identified Root Cause**: The date parsing logic in `core\n/admin.py` was overly restrictive, only supporting `%m/%d/%Y` and `%Y-%m-%d` for\nmats. This caused failures for valid timestamps (ISO 8601 or formats including t\nime). 2. **Implemented Robust Parsing**: - Added a `parse_any_date` helper funct\nion to `core/admin.py` that utilizes Django's `parse_datetime` and `parse_date` \nutilities, along with a comprehensive list of common CSV date formats. - Updated\n `InteractionAdmin.import_interactions` to use this new helper in both the impor\nt preview (for existence checking) and the actual import process. 3. **Broad Imp\nrovements**: Applied the same robust parsing to `VoterAdmin`, `DonationAdmin`, a\nnd `VotingRecordAdmin` to ensure consistency across all data import types. 4. **\nVerification**: Restarted the Django service and performed a health check to ens\nure the application is running correctly. Changed: * `core/admin.py`: Added `par\nse_any_date` helper and updated all import views (`import_voters`, `import_donat\nions`, `import_interactions`, `import_voting_records`) to support flexible date \nand timestamp formats. Notes: * The system now supports a wide range of date for\nmats including `YYYY-MM-DD HH:MM:SS`, `MM/DD/YYYY`, ISO 8601, and more. * If a t\nimestamp includes time information, it will now be correctly parsed and stored i\nn `DateTimeField` fields (like `Interaction.date`) or truncated to a date for `D\nateField` fields (like `Voter.birthdate`). Next: Please try importing your inter\nactions again with the timestamp. They should now be processed successfully. Rem\ninder: click **Save** in the editor to sync changes.' \n126646:Assistant: '1. **Identified Root Cause**: The date parsing logic in `core\n/admin.py` was overly restrictive, only supporting `%m/%d/%Y` and `%Y-%m-%d` for\nmats. This caused failures for valid timestamps (ISO 8601 or formats including t\nime). 2. **Implemented Robust Parsing**: - Added a `parse_any_date` helper funct\nion to `core/admin.py` that utilizes Django's `parse_datetime` and `parse_date` \nutilities, along with a comprehensive list of common CSV date formats. - Updated\n `InteractionAdmin.import_interactions` to use this new helper in both the impor\nt preview (for existence checking) and the actual import process. 3. **Broad Imp\nrovements**: Applied the same robust parsing to `VoterAdmin`, `DonationAdmin`, a\nnd `VotingRecordAdmin` to ensure consistency across all data import types. 4. **\nVerification**: Restarted the Django service and performed a health check to ens\nure the application is running correctly. Changed: * `core/admin.py`: Added `par\nse_any_date` helper and updated all import views (`import_voters`, `import_donat\nions`, `import_interactions`, `import_voting_records`) to support flexible date \nand timestamp formats. Notes: * The system now supports a wide range of date for\nmats including `YYYY-MM-DD HH:MM:SS`, `MM/DD/YYYY`, ISO 8601, and more. * If a t\nimestamp includes time information, it will now be correctly parsed and stored i\nn `DateTimeField` fields (like `Interaction.date`) or truncated to a date for `D\nateField` fields (like `Voter.birthdate`). Next: Please try importing your inter\nactions again with the timestamp. They should now be processed successfully. Rem\ninder: click **Save** in the editor to sync changes.'",
/home/ubuntu/.gemini/tmp/ec2c6b8cb2ac3d6ef62b888a61e62d5ab1e71e46b3c4963f2a040a3c33c7038b/chats/session-2026-02-22T19-38-2d2510b4.json: "command": "grep -a \"InteractionAdmin\" /home/ubuntu/.gemini/tmp/ec2c6b8cb2ac3d6ef62b888a61e62d5ab1e71e46b3c4963f2a040a3c33c7038b/chats/session-2026-02-22T19-38-2d2510b4.json | tail -c 10000 > core/admin_reconstruct.txt"
/home/ubuntu/.gemini/tmp/ec2c6b8cb2ac3d6ef62b888a61e62d5ab1e71e46b3c4963f2a040a3c33c7038b/chats/session-2026-02-22T19-38-2d2510b4.json: "output": "Command: grep -a \"InteractionAdmin\" /home/ubuntu/.gemini/tmp/ec2c6b8cb2ac3d6ef62b888a61e62d5ab1e71e46b3c4963f2a040a3c33c7038b/chats/session-2026-02-22T19-38-2d2510b4.json | tail -c 10000 > core/admin_reconstruct.txt\nDirectory: (root)\nOutput: (empty)\nError: (none)\nExit Code: 0\nSignal: 0\nBackground PIDs: (none)\nProcess Group PGID: 15471"
/home/ubuntu/.gemini/tmp/ec2c6b8cb2ac3d6ef62b888a61e62d5ab1e71e46b3c4963f2a040a3c33c7038b/chats/session-2026-02-22T19-38-2d2510b4.json: "command": "find /home/ubuntu -mtime -30 -name \"*.py\" 2>/dev/null | xargs grep -l \"InteractionAdmin\" | grep -v \"core/admin.py\""
/home/ubuntu/.gemini/tmp/ec2c6b8cb2ac3d6ef62b888a61e62d5ab1e71e46b3c4963f2a040a3c33c7038b/chats/session-2026-02-22T19-38-2d2510b4.json: "output": "Command: find /home/ubuntu -mtime -30 -name \"*.py\" 2>/dev/null | xargs grep -l \"InteractionAdmin\" | grep -v \"core/admin.py\"\nDirectory: (root)\nOutput: (empty)\nError: (none)\nExit Code: 1\nSignal: 0\nBackground PIDs: (none)\nProcess Group PGID: 15480"
/home/ubuntu/.gemini/tmp/ec2c6b8cb2ac3d6ef62b888a61e62d5ab1e71e46b3c4963f2a040a3c33c7038b/chats/session-2026-02-22T19-01-a16108b3.json: "output": "\nIMPORTANT: The file content has been truncated.\nStatus: Showing lines 1-2000 of 2004 total lines.\nAction: To read more of the file, you can use the 'offset' and 'limit' parameters in a subsequent 'read_file' call. For example, to read the next section of the file, use offset: 2000.\n\n--- FILE CONTENT (truncated) ---\nfrom decimal import Decimal\nfrom datetime import datetime, date\nfrom django.db import transaction\nfrom django.http import HttpResponse\nfrom django.utils.safestring import mark_safe\nimport csv\nimport io\nimport logging\nimport tempfile\nimport os\nfrom django.contrib import admin, messages\nfrom django.urls import path, reverse\nfrom django.shortcuts import render, redirect\nfrom django.template.response import TemplateResponse\nfrom .models import (\n format_phone_number,\n Tenant, TenantUserRole, InteractionType, DonationMethod, ElectionType, EventType, Voter,\n VotingRecord, Event, EventParticipation, Donation, Interaction, VoterLikelihood, CampaignSettings,\n Interest, Volunteer, VolunteerEvent, ParticipationStatus, VolunteerRole\n)\nfrom .forms import (\n VoterImportForm, EventImportForm, EventParticipationImportForm,\n DonationImportForm, InteractionImportForm, VoterLikelihoodImportForm,\n VolunteerImportForm, VotingRecordImportForm\n)\n\nlogger = logging.getLogger(__name__)\n\nVOTER_MAPPABLE_FIELDS = [\n ('voter_id', 'Voter ID'),\n ('first_name', 'First Name'),\n ('last_name', 'Last Name'),\n ('nickname', 'Nickname'),\n ('birthdate', 'Birthdate'),\n ('address_street', 'Street Address'),\n ('city', 'City'),\n ('state', 'State'),\n ('prior_state', 'Prior State'),\n ('zip_code', 'Zip Code'),\n ('county', 'County'),\n ('neighborhood', 'Neighborhood'),\n ('phone', 'Phone'),\n ('notes', 'Notes'),\n ('phone_type', 'Phone Type'),\n ('email', 'Email'),\n ('district', 'District'),\n ('precinct', 'Precinct'),\n ('registration_date', 'Registration Date'),\n ('is_targeted', 'Is Targeted'),\n ('candidate_support', 'Candidate Support'),\n ('yard_sign', 'Yard Sign'),\n ('window_sticker', 'Window Sticker'),\n ('latitude', 'Latitude'),\n ('longitude', 'Longitude'),\n ('secondary_phone', 'Secondary Phone'),\n ('secondary_phone_type', 'Secondary Phone Type'),\n ('door_visit', 'Door Visit'),\n]\n\nEVENT_MAPPABLE_FIELDS = [\n ('name', 'Name'),\n ('date', 'Date'),\n ('start_time', 'Start Time'),\n ('end_time', 'End Time'),\n ('event_type', 'Event Type (Name)'),\n ('description', 'Description'),\n ('location_name', 'Location Name'),\n ('address', 'Address'),\n ('city', 'City'),\n ('state', 'State'),\n ('zip_code', 'Zip Code'),\n ('latitude', 'Latitude'),\n ('longitude', 'Longitude'),\n]\n\nEVENT_PARTICIPATION_MAPPABLE_FIELDS = [\n ('voter_id', 'Voter ID'),\n ('first_name', 'First Name'),\n ('last_name', 'Last Name'),\n ('event_name', 'Event Name'),\n ('participation_status', 'Participation Status'),\n]\n\nDONATION_MAPPABLE_FIELDS = [\n ('voter_id', 'Voter ID'),\n ('date', 'Date'),\n ('amount', 'Amount'),\n ('method', 'Donation Method (Name)'),\n]\n\nINTERACTION_MAPPABLE_FIELDS = [\n ('voter_id', 'Voter ID'),\n ('volunteer_email', 'Volunteer Email'),\n ('date', 'Date'),\n ('type', 'Interaction Type (Name)'),\n ('description', 'Description'),\n ('notes', 'Notes'),\n]\n\n\nVOLUNTEER_MAPPABLE_FIELDS = [\n ('first_name', 'First Name'),\n ('last_name', 'Last Name'),\n ('email', 'Email'),\n ('phone', 'Phone'),\n ('notes', 'Notes'),\n]\n\nVOTER_LIKELIHOOD_MAPPABLE_FIELDS = [\n ('voter_id', 'Voter ID'),\n ('election_type', 'Election Type (Name)'),\n ('likelihood', 'Likelihood'),\n]\n\nVOTING_RECORD_MAPPABLE_FIELDS = [\n ('voter_id', 'Voter ID'),\n ('election_date', 'Election Date'),\n ('election_description', 'Election Description'),\n ('primary_party', 'Primary Party'),\n]\n\nclass BaseImportAdminMixin:\n def download_errors(self, request):\n logger.info(f\"download_errors called for {self.model._meta.model_name}\")\n session_key = f\"{self.model._meta.model_name}_import_errors\"\n failed_rows = request.session.get(session_key, [])\n if not failed_rows:\n self.message_user(request, \"No error log found in session.\", level=messages.WARNING)\n return redirect(\"../\")\n\n response = HttpResponse(content_type=\"text/csv\")\n response[\"Content-Disposition\"] = f\"attachment; filename={self.model._meta.model_name}_import_errors.csv\"\n\n if failed_rows:\n all_keys = set()\n for r in failed_rows:\n all_keys.update(r.keys())\n\n writer = csv.DictWriter(response, fieldnames=sorted(list(all_keys)))\n writer.writeheader()\n writer.writerows(failed_rows)\n\n return response\n\n def chunk_reader(self, reader, size):\n chunk = []\n for row in reader:\n chunk.append(row)\n if len(chunk) == size:\n yield chunk\n chunk = []\n if chunk:\n yield chunk\nclass TenantUserRoleInline(admin.TabularInline):\n model = TenantUserRole\n extra = 1\n\nclass CampaignSettingsInline(admin.StackedInline):\n model = CampaignSettings\n can_delete = False\n\n@admin.register(Tenant)\nclass TenantAdmin(admin.ModelAdmin):\n list_display = ('name', 'created_at')\n search_fields = ('name',)\n inlines = [TenantUserRoleInline, CampaignSettingsInline]\n\n@admin.register(TenantUserRole)\nclass TenantUserRoleAdmin(admin.ModelAdmin):\n list_display = ('user', 'tenant', 'role')\n list_filter = ('tenant', 'role')\n search_fields = ('user__username', 'tenant__name')\n\n@admin.register(InteractionType)\nclass InteractionTypeAdmin(admin.ModelAdmin):\n list_display = ('name', 'tenant', 'is_active')\n list_filter = ('tenant', 'is_active')\n search_fields = ('name',)\n\n@admin.register(DonationMethod)\nclass DonationMethodAdmin(admin.ModelAdmin):\n list_display = ('name', 'tenant', 'is_active')\n list_filter = ('tenant', 'is_active')\n search_fields = ('name',)\n\n@admin.register(VolunteerRole)\nclass VolunteerRoleAdmin(admin.ModelAdmin):\n list_display = (\"name\", \"tenant\", \"is_active\")\n list_filter = (\"tenant\", \"is_active\")\n search_fields = (\"name\",)\n\n@admin.register(ElectionType)\nclass ElectionTypeAdmin(admin.ModelAdmin):\n list_display = ('name', 'tenant', 'is_active')\n list_filter = ('tenant', 'is_active')\n search_fields = ('name',)\n\n@admin.register(EventType)\nclass EventTypeAdmin(admin.ModelAdmin):\n list_display = ('name', 'tenant', 'is_active', 'default_volunteer_role')\n list_filter = ('tenant', 'is_active')\n search_fields = ('name',)\n filter_horizontal = ('available_roles',)\n\n\n@admin.register(ParticipationStatus)\nclass ParticipationStatusAdmin(admin.ModelAdmin):\n list_display = ('name', 'tenant', 'is_active')\n list_filter = ('tenant', 'is_active')\n search_fields = ('name',)\n change_list_template = 'admin/participationstatus_change_list.html'\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n@admin.register(Interest)\nclass InterestAdmin(admin.ModelAdmin):\n list_display = ('name', 'tenant')\n list_filter = ('tenant',)\n fields = ('tenant', 'name')\n search_fields = ('name',)\n\nclass VotingRecordInline(admin.TabularInline):\n model = VotingRecord\n extra = 1\n\nclass DonationInline(admin.TabularInline):\n model = Donation\n extra = 1\n\nclass InteractionInline(admin.TabularInline):\n model = Interaction\n extra = 1\n autocomplete_fields = ['voter', 'type', 'volunteer']\n\nclass VoterLikelihoodInline(admin.TabularInline):\n model = VoterLikelihood\n extra = 1\n\nclass VolunteerEventInline(admin.TabularInline):\n model = VolunteerEvent\n extra = 1\n\n@admin.register(Voter)\nclass VoterAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('first_name', 'last_name', 'nickname', 'voter_id', 'tenant', 'district', 'candidate_support', 'is_targeted', 'city', 'state', 'prior_state')\n list_filter = ('tenant', 'candidate_support', 'is_targeted', 'phone_type', 'yard_sign', 'district', 'city', 'state', 'prior_state')\n search_fields = ('first_name', 'last_name', 'nickname', 'voter_id', 'address', 'city', 'state', 'prior_state', 'zip_code', 'county')\n inlines = [VotingRecordInline, DonationInline, InteractionInline, VoterLikelihoodInline]\n readonly_fields = ('address',)\n change_list_template = \"admin/voter_change_list.html\"\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context[\"tenants\"] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='voter-download-errors'),\n path('import-voters/', self.admin_site.admin_view(self.import_voters), name='import-voters'),\n ]\n return my_urls + urls\n\n\n def import_voters(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get(\"file_path\")\n tenant_id = request.POST.get(\"tenant\")\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in VOTER_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f\"map_{field_name}\")\n\n try:\n with open(file_path, \"r\", encoding=\"utf-8-sig\") as f:\n # Optimization: Skip full count for very large files in preview if needed,\n # but here we'll keep it for accuracy unless it's a known bottleneck.\n # For now, let's just do a fast line count.\n total_count = sum(1 for line in f) - 1\n f.seek(0)\n reader = csv.DictReader(f)\n preview_rows = []\n voter_ids_for_preview = []\n for i, row in enumerate(reader):\n if i < 10:\n preview_rows.append(row)\n v_id = row.get(mapping.get(\"voter_id\"))\n if v_id:\n voter_ids_for_preview.append(v_id.strip())\n else:\n break\n\n existing_preview_ids = set(Voter.objects.filter(tenant=tenant, voter_id__in=voter_ids_for_preview).values_list(\"voter_id\", flat=True))\n\n create_count = 0\n update_count = 0\n\n for row in preview_rows:\n voter_id_val = row.get(mapping.get(\"voter_id\"))\n if voter_id_val and voter_id_val.strip() in existing_preview_ids:\n update_count += 1\n else:\n create_count += 1\n\n context = self.admin_site.each_context(request)\n context.update({\n \"title\": \"Import Preview\",\n \"total_count\": total_count,\n \"create_count\": create_count,\n \"update_count\": update_count,\n \"preview_data\": preview_rows, # This should be improved to show actual changes\n \"mapping\": mapping,\n \"file_path\": file_path,\n \"tenant_id\": tenant_id,\n \"action_url\": request.path,\n \"opts\": self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n elif \"_import\" in request.POST:\n file_path = request.POST.get(\"file_path\")\n tenant_id = request.POST.get(\"tenant\")\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in VOTER_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f\"map_{field_name}\")\n\n try:\n created_count = 0\n updated_count = 0\n skipped_no_change = 0\n skipped_no_id = 0\n errors = 0\n failed_rows = []\n total_processed = 0\n\n # Temporary storage for error rows to avoid holding large file in memory\n temp_error_file = None\n temp_error_file_path = None\n\n # Process in chunks to reduce memory usage for very large files\n with open(file_path, \"r\", encoding=\"utf-8-sig\") as f_read:\n reader = csv.DictReader(f_read)\n for i, row in enumerate(reader):\n total_processed += 1\n try:\n raw_voter_id = row.get(mapping.get(\"voter_id\"))\n voter_id = raw_voter_id.strip() if raw_voter_id else None\n\n if not voter_id:\n # Enhanced error message to guide the user\n mapped_column_name = mapping.get(\"voter_id\", \"N/A\")\n error_detail = f\"Raw value: '{raw_voter_id}'. \" if raw_voter_id is not None else \"Value was None.\"\n row[\"Import Error\"] = f\"Voter ID is required. Please check if the '{mapped_column_name}' column is correctly mapped and contains values for all rows. {error_detail}\"\n failed_rows.append(row)\n skipped_no_id += 1\n errors += 1\n continue\n\n defaults = {}\n # Map other fields dynamically\n for field_name, _ in VOTER_MAPPABLE_FIELDS:\n csv_column = mapping.get(field_name)\n if csv_column and csv_column in row:\n field_value = row[csv_column].strip()\n if field_name == \"birthdate\" or field_name == \"registration_date\":\n # Handle date conversions\n if field_value:\n try:\n # Attempt to parse common date formats\n if '/' in field_value:\n # Try MM/DD/YYYY or DD/MM/YYYY\n if len(field_value.split('/')[2]) == 2: # YY format\n dt = datetime.strptime(field_value, '%m/%d/%y').date() if len(field_value.split('/')[0]) < 3 else datetime.strptime(field_value, '%d/%m/%y').date() # noqa\n else:\n dt = datetime.strptime(field_value, '%m/%d/%Y').date() if len(field_value.split('/')[0]) < 3 else datetime.strptime(field_value, '%d/%m/%Y').date() # noqa\n elif '-' in field_value:\n # Try YYYY-MM-DD or DD-MM-YYYY or MM-DD-YYYY\n if len(field_value.split('-')[0]) == 4: # YYYY format\n dt = datetime.strptime(field_value, '%Y-%m-%d').date()\n elif len(field_value.split('-')[2]) == 4: # YYYY format\n dt = datetime.strptime(field_value, '%m-%d-%Y').date() if len(field_value.split('-')[0]) < 3 else datetime.strptime(field_value, '%d-%m-%Y').date() # noqa\n else:\n # Default to MM-DD-YY\n dt = datetime.strptime(field_value, '%m-%d-%y').date()\n else:\n dt = None\n\n if dt:\n defaults[field_name] = dt\n else:\n logger.warning(f\"Could not parse date '{field_value}' for field {field_name}. Skipping.\")\n except ValueError as ve:\n logger.warning(f\"Date parsing error for '{field_value}' in field {field_name}: {ve}\")\n except Exception as ex:\n logger.error(f\"Unexpected error parsing date '{field_value}' for field {field_name}: {ex}\")\n elif field_name == \"is_targeted\" or field_name == \"yard_sign\" or field_name == \"window_sticker\" or field_name == \"door_visit\":\n # Handle boolean fields\n if field_value.lower() == 'true' or field_value == '1':\n defaults[field_name] = True\n elif field_value.lower() == 'false' or field_value == '0':\n defaults[field_name] = False\n else:\n defaults[field_name] = None # Or sensible default/error\n elif field_name == \"phone\":\n defaults[field_name] = format_phone_number(field_value)\n elif field_name == \"email\":\n defaults[field_name] = field_value.lower() # Store emails as lowercase\n elif field_name == \"candidate_support\":\n if field_value in [choice[0] for choice in Voter.CANDIDATE_SUPPORT_CHOICES]:\n defaults[field_name] = field_value\n else:\n logger.warning(f\"Invalid candidate_support value: {field_value}. Skipping.\")\n elif field_name == \"phone_type\":\n if field_value in [choice[0] for choice in Voter.PHONE_TYPE_CHOICES]:\n defaults[field_name] = field_value\n else:\n logger.warning(f\"Invalid phone_type value: {field_value}. Skipping.\")\n elif field_name == \"secondary_phone_type\":\n if field_value in [choice[0] for choice in Voter.PHONE_TYPE_CHOICES]:\n defaults[field_name] = field_value\n else:\n logger.warning(f\"Invalid secondary_phone_type value: {field_value}. Skipping.\")\n elif field_name == \"state\" or field_name == \"prior_state\":\n # Ensure state is uppercase and valid length\n if field_value and len(field_value) <= 2:\n defaults[field_name] = field_value.upper()\n else:\n logger.warning(f\"Invalid state value: {field_value}. Skipping.\")\n else:\n defaults[field_name] = field_value\n\n # Try to get voter. If not found, create new. Update if found.\n voter, created = Voter.objects.update_or_create(\n tenant=tenant,\n voter_id=voter_id,\n defaults=defaults\n )\n if created:\n created_count += 1\n else:\n updated_count += 1\n\n # Special handling for interests - assuming a comma-separated list in CSV\n if 'interests' in mapping and row.get(mapping['interests']):\n interest_names = [name.strip() for name in row[mapping['interests']].split(',') if name.strip()]\n for interest_name in interest_names:\n interest, _ = Interest.objects.get_or_create(tenant=tenant, name=interest_name)\n voter.interests.add(interest)\n\n if (i + 1) % 100 == 0:\n print(f\"DEBUG: Voter import progress: {total_processed} processed. {created_count} created. {updated_count} updated.\")\n\n except Exception as e:\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n logger.error(f\"Error importing row: {row}. Error: {e}\")\n\n # Clean up the temporary file\n if os.path.exists(file_path):\n os.remove(file_path)\n\n if temp_error_file_path and os.path.exists(temp_error_file_path):\n os.remove(temp_error_file_path)\n\n self.message_user(request, f\"Import complete: {created_count + updated_count} voters created/updated. ({created_count} new, {updated_count} updated, {skipped_no_change} skipped with no changes, {skipped_no_id} skipped missing ID, {errors} errors)\")\n # Store failed rows in session for download, limit to avoid session overflow\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n\n if errors > 0:\n error_url = reverse(\"admin:voter-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = VoterImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n \"title\": \"Map Voter Fields\",\n \"headers\": headers,\n \"model_fields\": VOTER_MAPPABLE_FIELDS,\n \"tenant_id\": tenant.id,\n \"file_path\": file_path,\n \"action_url\": request.path,\n \"opts\": self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = VoterImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Voters\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)\n\n\n@admin.register(Event)\nclass EventAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('name', 'date', 'event_type', 'tenant', 'location_name', 'address', 'city', 'state', 'zip_code')\n list_filter = ('tenant', 'event_type')\n search_fields = ('name', 'location_name', 'address', 'city', 'state', 'zip_code')\n inlines = [VolunteerEventInline]\n change_list_template = \"admin/event_change_list.html\"\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='event-download-errors'),\n path('import-events/', self.admin_site.admin_view(self.import_events), name='import-events'),\n ]\n return my_urls + urls\n\n def import_events(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n mapping = {}\n for field_name, _ in EVENT_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n total_count = 0\n create_count = 0\n update_count = 0\n preview_data = []\n for row in reader:\n total_count += 1\n event_name = row.get(mapping.get('name'))\n event_date = row.get(mapping.get('date'))\n\n exists = False\n if event_name and event_date:\n try:\n # Assuming name and date uniquely identify an event\n # This might need refinement based on actual data uniqueness requirements\n if '/' in event_date:\n dt = datetime.strptime(event_date, '%m/%d/%Y').date()\n elif '-' in event_date:\n dt = datetime.strptime(event_date, '%Y-%m-%d').date()\n else:\n dt = None\n\n if dt:\n exists = Event.objects.filter(tenant=tenant, name=event_name, date=dt).exists()\n\n except ValueError:\n # Handle cases where date parsing fails\n pass\n\n if exists:\n update_count += 1\n action = 'update'\n else:\n create_count += 1\n action = 'create'\n\n if len(preview_data) < 10:\n preview_data.append({\n 'action': action,\n 'identifier': f\"Event: {event_name} (Date: {event_date})\",\n 'details': f\"Location: {row.get(mapping.get('location_name', '')) or ''}\"\n })\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Import Preview\",\n 'total_count': total_count,\n 'create_count': create_count,\n 'update_count': update_count,\n 'preview_data': preview_data,\n 'mapping': mapping,\n 'file_path': file_path,\n 'tenant_id': tenant_id,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n\n elif \"_import\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in EVENT_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n count = 0\n errors = 0\n failed_rows = []\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n for row in reader:\n try:\n event_name = row.get(mapping.get('name'))\n event_date = row.get(mapping.get('date'))\n event_type_name = row.get(mapping.get('event_type'))\n\n if not event_name or not event_date or not event_type_name:\n row[\"Import Error\"] = \"Missing event name, date, or type\"\n failed_rows.append(row)\n errors += 1\n continue\n\n # Date parsing for event_date\n try:\n if '/' in event_date:\n parsed_date = datetime.strptime(event_date, '%m/%d/%Y').date()\n elif '-' in event_date:\n parsed_date = datetime.strptime(event_date, '%Y-%m-%d').date()\n else:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n except ValueError:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n\n event_type_obj, _ = EventType.objects.get_or_create(tenant=tenant, name=event_type_name)\n\n defaults = {\n 'date': parsed_date,\n 'event_type': event_type_obj,\n 'description': row.get(mapping.get('description')) or '',\n 'location_name': row.get(mapping.get('location_name')) or '',\n 'address': row.get(mapping.get('address')) or '',\n 'city': row.get(mapping.get('city')) or '',\n 'state': row.get(mapping.get('state')) or '',\n 'zip_code': row.get(mapping.get('zip_code')) or '',\n 'latitude': row.get(mapping.get('latitude')) or None,\n 'longitude': row.get(mapping.get('longitude')) or None,\n }\n\n # Handle start_time and end_time\n start_time_str = row.get(mapping.get('start_time'))\n if start_time_str:\n try:\n defaults['start_time'] = datetime.strptime(start_time_str, '%H:%M').time()\n except ValueError:\n logger.warning(f\"Invalid start_time format: {start_time_str}. Skipping.\")\n end_time_str = row.get(mapping.get('end_time'))\n if end_time_str:\n try:\n defaults['end_time'] = datetime.strptime(end_time_str, '%H:%M').time()\n except ValueError:\n logger.warning(f\"Invalid end_time format: {end_time_str}. Skipping.\")\n\n Event.objects.update_or_create(\n tenant=tenant,\n name=event_name,\n defaults=defaults\n )\n count += 1\n except Exception as e:\n logger.error(f\"Error importing: {e}\")\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n\n if os.path.exists(file_path):\n os.remove(file_path)\n self.message_user(request, f\"Successfully imported {count} events.\")\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n if errors > 0:\n error_url = reverse(\"admin:event-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = EventImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Map Event Fields\",\n 'headers': headers,\n 'model_fields': EVENT_MAPPABLE_FIELDS,\n 'tenant_id': tenant.id,\n 'file_path': file_path,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = EventImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Events\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)\n\n\n@admin.register(Volunteer)\nclass VolunteerAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('first_name', 'last_name', 'email', 'phone', 'tenant')\n list_filter = ('tenant',)\n search_fields = ('first_name', 'last_name', 'email', 'phone')\n change_list_template = \"admin/volunteer_change_list.html\"\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='volunteer-download-errors'),\n path('import-volunteers/', self.admin_site.admin_view(self.import_volunteers), name='import-volunteers'),\n ]\n return my_urls + urls\n\n def import_volunteers(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n mapping = {}\n for field_name, _ in VOLUNTEER_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n total_count = 0\n create_count = 0\n update_count = 0\n preview_data = []\n for row in reader:\n total_count += 1\n email = row.get(mapping.get('email'))\n\n exists = False\n if email:\n exists = Volunteer.objects.filter(tenant=tenant, email=email).exists()\n\n if exists:\n update_count += 1\n action = 'update'\n else:\n create_count += 1\n action = 'create'\n\n if len(preview_data) < 10:\n preview_data.append({\n 'action': action,\n 'identifier': f\"Volunteer: {email}\",\n 'details': f\"Name: {row.get(mapping.get('first_name', '')) or ''} {row.get(mapping.get('last_name', '')) or ''}\"\n })\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Import Preview\",\n 'total_count': total_count,\n 'create_count': create_count,\n 'update_count': update_count,\n 'preview_data': preview_data,\n 'mapping': mapping,\n 'file_path': file_path,\n 'tenant_id': tenant_id,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n\n elif \"_import\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in VOLUNTEER_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n count = 0\n errors = 0\n failed_rows = []\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n for row in reader:\n try:\n email = row.get(mapping.get('email'))\n if not email:\n row[\"Import Error\"] = \"Missing email\"\n failed_rows.append(row)\n errors += 1\n continue\n\n defaults = {\n 'first_name': row.get(mapping.get('first_name')) or '',\n 'last_name': row.get(mapping.get('last_name')) or '',\n 'phone': format_phone_number(row.get(mapping.get('phone')) or ''),\n 'notes': row.get(mapping.get('notes')) or '',\n }\n\n Volunteer.objects.update_or_create(\n tenant=tenant,\n email=email,\n defaults=defaults\n )\n count += 1\n except Exception as e:\n logger.error(f\"Error importing volunteer: {e}\")\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n\n if os.path.exists(file_path):\n os.remove(file_path)\n self.message_user(request, f\"Successfully imported {count} volunteers.\")\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n if errors > 0:\n error_url = reverse(\"admin:volunteer-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = VolunteerImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Map Volunteer Fields\",\n 'headers': headers,\n 'model_fields': VOLUNTEER_MAPPABLE_FIELDS,\n 'tenant_id': tenant.id,\n 'file_path': file_path,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = VolunteerImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Volunteers\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)\n\n\n@admin.register(EventParticipation)\nclass EventParticipationAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('event', 'voter', 'participation_status')\n list_filter = ('event', 'participation_status', 'voter__tenant')\n search_fields = ('event__name', 'voter__first_name', 'voter__last_name', 'voter__voter_id')\n change_list_template = 'admin/eventparticipation_change_list.html'\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_list(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='eventparticipation-download-errors'),\n path('import-event-participations/', self.admin_site.admin_view(self.import_event_participations), name='import-event-participations'),\n ]\n return my_urls + urls\n\n def import_event_participations(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n mapping = {}\n for field_name, _ in EVENT_PARTICIPATION_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n total_count = 0\n create_count = 0\n update_count = 0\n preview_data = []\n for row in reader:\n total_count += 1\n voter_id = row.get(mapping.get('voter_id'))\n\n # Extract first_name and last_name from CSV based on mapping\n csv_first_name = row.get(mapping.get('first_name'), '')\n csv_last_name = row.get(mapping.get('last_name'), '')\n csv_full_name = f\"{csv_first_name} {csv_last_name}\".strip()\n\n exists = False\n voter_full_name = \"N/A\" # Initialize voter_full_name\n if voter_id:\n try:\n voter = Voter.objects.get(tenant=tenant, voter_id=voter_id)\n voter_full_name = f\"{voter.first_name} {voter.last_name}\" # Get voter's full name\n if event_name:\n exists = EventParticipation.objects.filter(voter=voter, event__name=event_name).exists()\n except Voter.DoesNotExist:\n pass\n\n if exists:\n update_count += 1\n action = 'update'\n else:\n create_count += 1\n action = 'create'\n\n if len(preview_data) < 10:\n preview_data.append({\n 'action': action,\n 'csv_full_name': csv_full_name, # Add CSV name\n 'identifier': f\"Voter: {voter_full_name} (ID: {voter_id})\" if voter_id else \"N/A\", # Include full name\n 'details': f\"Participation: {row.get(mapping.get('participation_status', '')) or ''}\"\n })\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Import Preview\",\n 'total_count': total_count,\n 'create_count': create_count,\n 'update_count': update_count,\n 'preview_data': preview_data,\n 'mapping': mapping,\n 'file_path': file_path,\n 'tenant_id': tenant_id,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n\n elif \"_import\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in EVENT_PARTICIPATION_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n count = 0\n errors = 0\n failed_rows = []\n for row in reader:\n try:\n voter_id = row.get(mapping.get('voter_id')) if mapping.get('voter_id') else None\n participation_status_val = row.get(mapping.get('participation_status')) if mapping.get('participation_status') else None\n\n if voter_id: # Only strip if voter_id is not None\n voter_id = voter_id.strip()\n\n if not voter_id:\n row[\"Import Error\"] = \"Missing voter ID\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n voter = Voter.objects.get(tenant=tenant, voter_id=voter_id)\n except Voter.DoesNotExist:\n error_msg = f\"Voter with ID {voter_id} not found\"\n logger.error(error_msg)\n row[\"Import Error\"] = error_msg\n failed_rows.append(row)\n errors += 1\n continue\n\n event = None\n event_name = row.get(mapping.get('event_name')) if mapping.get('event_name') else None\n if event_name:\n try:\n event = Event.objects.get(tenant=tenant, name=event_name)\n except Event.DoesNotExist:\n pass\n\n if not event:\n error_msg = \"Event not found (check Event Name)\"\n logger.error(error_msg)\n row[\"Import Error\"] = error_msg\n failed_rows.append(row)\n errors += 1\n continue\n\n defaults = {}\n if participation_status_val and participation_status_val.strip():\n status_obj, _ = ParticipationStatus.objects.get_or_create(tenant=tenant, name=participation_status_val.strip())\n defaults['participation_status'] = status_obj\n else:\n # Default to 'Invited' if not specified\n status_obj, _ = ParticipationStatus.objects.get_or_create(tenant=tenant, name='Invited')\n defaults['participation_status'] = status_obj\n EventParticipation.objects.update_or_create(\n event=event,\n voter=voter,\n defaults=defaults\n )\n count += 1\n except Exception as e:\n logger.error(f\"Error importing: {e}\")\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n\n if os.path.exists(file_path):\n os.remove(file_path)\n self.message_user(request, f\"Successfully imported {count} participations.\")\n # Optimization: Limit error log size in session to avoid overflow\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n logger.info(f\"Stored {len(failed_rows)} failed rows in session for {self.model._meta.model_name}\")\n if errors > 0:\n error_url = reverse(\"admin:eventparticipation-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = EventParticipationImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Map Participation Fields\",\n 'headers': headers,\n 'model_fields': EVENT_PARTICIPATION_MAPPABLE_FIELDS,\n 'tenant_id': tenant.id,\n 'file_path': file_path,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = EventParticipationImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Participations\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)\n\n@admin.register(Donation)\nclass DonationAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('voter', 'date', 'amount', 'method')\n list_filter = ('voter__tenant', 'method')\n search_fields = ('voter__first_name', 'voter__last_name', 'voter__voter_id', 'method__name')\n change_list_template = 'admin/donation_change_list.html'\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='donation-download-errors'),\n path('import-donations/', self.admin_site.admin_view(self.import_donations), name='import-donations'),\n ]\n return my_urls + urls\n\n def import_donations(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n mapping = {}\n for field_name, _ in DONATION_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n total_count = 0\n create_count = 0\n update_count = 0\n preview_data = []\n for row in reader:\n total_count += 1\n voter_id = row.get(mapping.get('voter_id'))\n\n exists = False\n if voter_id:\n exists = Voter.objects.filter(tenant=tenant, voter_id=voter_id).exists()\n\n if exists:\n update_count += 1\n action = 'update'\n else:\n create_count += 1\n action = 'create'\n\n if len(preview_data) < 10:\n preview_data.append({\n 'action': action,\n 'identifier': f\"Voter ID: {voter_id}\",\n 'details': f\"Amount: {row.get(mapping.get('amount', '')) or ''}, Method: {row.get(mapping.get('method', '')) or ''}\"\n })\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Import Preview\",\n 'total_count': total_count,\n 'create_count': create_count,\n 'update_count': update_count,\n 'preview_data': preview_data,\n 'mapping': mapping,\n 'file_path': file_path,\n 'tenant_id': tenant_id,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n\n elif \"_import\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in DONATION_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n count = 0\n errors = 0\n failed_rows = []\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n for row in reader:\n try:\n voter_id = row.get(mapping.get('voter_id'))\n date_str = row.get(mapping.get('date'))\n amount_str = row.get(mapping.get('amount'))\n method_name = row.get(mapping.get('method'))\n\n if voter_id: # Only strip if voter_id is not None\n voter_id = voter_id.strip()\n\n if not voter_id:\n row[\"Import Error\"] = \"Missing voter ID\"\n failed_rows.append(row)\n errors += 1\n continue\n\n if not date_str or not amount_str:\n row[\"Import Error\"] = \"Missing date or amount\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n voter = Voter.objects.get(tenant=tenant, voter_id=voter_id)\n except Voter.DoesNotExist:\n row[\"Import Error\"] = f\"Voter {voter_id} not found\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n if '/' in date_str:\n parsed_date = datetime.strptime(date_str, '%m/%d/%Y').date()\n elif '-' in date_str:\n parsed_date = datetime.strptime(date_str, '%Y-%m-%d').date()\n else:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n except ValueError:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n amount = Decimal(amount_str)\n except InvalidOperation:\n row[\"Import Error\"] = \"Invalid amount format\"\n failed_rows.append(row)\n errors += 1\n continue\n\n donation_method, _ = DonationMethod.objects.get_or_create(tenant=tenant, name=method_name)\n\n Donation.objects.create(\n voter=voter,\n date=parsed_date,\n amount=amount,\n method=donation_method\n )\n count += 1\n except Exception as e:\n logger.error(f\"Error importing: {e}\")\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n\n if os.path.exists(file_path):\n os.remove(file_path)\n self.message_user(request, f\"Successfully imported {count} donations.\")\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n if errors > 0:\n error_url = reverse(\"admin:donation-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = DonationImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Map Donation Fields\",\n 'headers': headers,\n 'model_fields': DONATION_MAPPABLE_FIELDS,\n 'tenant_id': tenant.id,\n 'file_path': file_path,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = DonationImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Donations\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)\n\n@admin.register(Interaction)\nclass InteractionAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('voter', 'date', 'type', 'description', 'volunteer')\n list_filter = ('voter__tenant', 'type', 'volunteer')\n search_fields = ('voter__first_name', 'voter__last_name', 'voter__voter_id', 'description', 'volunteer__first_name', 'volunteer__last_name')\n autocomplete_fields = ['voter', 'volunteer']\n change_list_template = 'admin/interaction_change_list.html'\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='interaction-download-errors'),\n path('import-interactions/', self.admin_site.admin_view(self.import_interactions), name='import-interactions'),\n ]\n return my_urls + urls\n\n def import_interactions(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n mapping = {}\n for field_name, _ in INTERACTION_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n total_count = 0\n create_count = 0\n update_count = 0\n preview_data = []\n for row in reader:\n total_count += 1\n voter_id = row.get(mapping.get('voter_id'))\n volunteer_email = row.get(mapping.get('volunteer_email'))\n\n exists = False\n if voter_id:\n exists = Voter.objects.filter(tenant=tenant, voter_id=voter_id).exists()\n\n if exists:\n update_count += 1\n action = 'update'\n else:\n create_count += 1\n action = 'create'\n\n if len(preview_data) < 10:\n preview_data.append({\n 'action': action,\n 'identifier': f\"Voter ID: {voter_id}\",\n 'details': f\"Type: {row.get(mapping.get('type', '')) or ''}, Volunteer: {volunteer_email or ''}\"\n })\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Import Preview\",\n 'total_count': total_count,\n 'create_count': create_count,\n 'update_count': update_count,\n 'preview_data': preview_data,\n 'mapping': mapping,\n 'file_path': file_path,\n 'tenant_id': tenant_id,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n\n elif \"_import\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in INTERACTION_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n count = 0\n errors = 0\n failed_rows = []\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n for row in reader:\n try:\n voter_id = row.get(mapping.get('voter_id'))\n volunteer_email = row.get(mapping.get('volunteer_email'))\n date_str = row.get(mapping.get('date'))\n type_name = row.get(mapping.get('type'))\n\n if voter_id: # Only strip if voter_id is not None\n voter_id = voter_id.strip()\n\n if not voter_id:\n row[\"Import Error\"] = \"Missing voter ID\"\n failed_rows.append(row)\n errors += 1\n continue\n\n if not date_str or not type_name:\n row[\"Import Error\"] = \"Missing date or description\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n voter = Voter.objects.get(tenant=tenant, voter_id=voter_id)\n except Voter.DoesNotExist:\n row[\"Import Error\"] = f\"Voter {voter_id} not found\"\n failed_rows.append(row)\n errors += 1\n continue\n\n volunteer = None\n if volunteer_email:\n try:\n volunteer = Volunteer.objects.get(tenant=tenant, email=volunteer_email)\n except Volunteer.DoesNotExist:\n pass # Volunteer is optional\n\n try:\n if '/' in date_str:\n parsed_date = datetime.strptime(date_str, '%m/%d/%Y').date()\n elif '-' in date_str:\n parsed_date = datetime.strptime(date_str, '%Y-%m-%d').date()\n else:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n except ValueError:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n\n interaction_type, _ = InteractionType.objects.get_or_create(tenant=tenant, name=type_name)\n\n Interaction.objects.create(\n voter=voter,\n volunteer=volunteer,\n date=parsed_date,\n type=interaction_type,\n description=row.get(mapping.get('description')) or '',\n notes=row.get(mapping.get('notes')) or ''\n )\n count += 1\n except Exception as e:\n logger.error(f\"Error importing: {e}\")\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n\n if os.path.exists(file_path):\n os.remove(file_path)\n self.message_user(request, f\"Successfully imported {count} interactions.\")\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n if errors > 0:\n error_url = reverse(\"admin:interaction-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = InteractionImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Map Interaction Fields\",\n 'headers': headers,\n 'model_fields': INTERACTION_MAPPABLE_FIELDS,\n 'tenant_id': tenant.id,\n 'file_path': file_path,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = InteractionImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Interactions\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)\n\n@admin.register(VoterLikelihood)\nclass VoterLikelihoodAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('voter', 'election_type', 'likelihood')\n list_filter = ('voter__tenant', 'election_type', 'likelihood')\n search_fields = ('voter__first_name', 'voter__last_name', 'voter__voter_id', 'election_type__name')\n change_list_template = 'admin/voterlikelihood_change_list.html'\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='voterlikelihood-download-errors'),\n path('import-likelihoods/', self.admin_site.admin_view(self.import_likelihoods), name='import-likelihoods'),\n ]\n return my_urls + urls\n\n def import_likelihoods(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n mapping = {}\n for field_name, _ in VOTER_LIKELIHOOD_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n total_count = 0\n create_count = 0\n update_count = 0\n preview_data = []\n for row in reader:\n total_count += 1\n voter_id = row.get(mapping.get('voter_id'))\n\n exists = False\n if voter_id:\n exists = Voter.objects.filter(tenant=tenant, voter_id=voter_id).exists()\n\n if exists:\n update_count += 1\n action = 'update'\n else:\n create_count += 1\n action = 'create'\n\n if len(preview_data) < 10:\n preview_data.append({\n 'action': action,\n 'identifier': f\"Voter ID: {voter_id}\",\n 'details': f\"Likelihood: {row.get(mapping.get('likelihood', '')) or ''}\"\n })\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Import Preview\",\n 'total_count': total_count,\n 'create_count': create_count,\n 'update_count': update_count,\n 'preview_data': preview_data,\n 'mapping': mapping,\n 'file_path': file_path,\n 'tenant_id': tenant_id,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n\n elif \"_import\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in VOTER_LIKELIHOOD_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n count = 0\n errors = 0\n failed_rows = []\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n for row in reader:\n try:\n voter_id = row.get(mapping.get('voter_id'))\n election_type_name = row.get(mapping.get('election_type'))\n likelihood_val = row.get(mapping.get('likelihood'))\n\n if voter_id: # Only strip if voter_id is not None\n voter_id = voter_id.strip()\n\n if not voter_id:\n row[\"Import Error\"] = \"Missing voter ID\"\n failed_rows.append(row)\n errors += 1\n continue\n\n if not election_type_name or not likelihood_val:\n row[\"Import Error\"] = \"Missing election type or likelihood\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n voter = Voter.objects.get(tenant=tenant, voter_id=voter_id)\n except Voter.DoesNotExist:\n row[\"Import Error\"] = f\"Voter {voter_id} not found\"\n failed_rows.append(row)\n errors += 1\n continue\n\n election_type, _ = ElectionType.objects.get_or_create(tenant=tenant, name=election_type_name)\n\n VoterLikelihood.objects.update_or_create(\n voter=voter,\n election_type=election_type,\n defaults={'likelihood': likelihood_val}\n )\n count += 1\n except Exception as e:\n print(f\"DEBUG: Likelihood import failed: {e}\")\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n\n if os.path.exists(file_path):\n os.remove(file_path)\n self.message_user(request, f\"Import complete: {count} likelihoods created/updated.\")\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n if errors > 0:\n error_url = reverse(\"admin:voterlikelihood-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = VoterLikelihoodImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Map Likelihood Fields\",\n 'headers': headers,\n 'model_fields': VOTER_LIKELIHOOD_MAPPABLE_FIELDS,\n 'tenant_id': tenant.id,\n 'file_path': file_path,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = VoterLikelihoodImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Likelihoods\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)\n\n\n@admin.register(VotingRecord)\nclass VotingRecordAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('voter', 'election_date', 'election_description', 'primary_party')\n list_filter = ('voter__tenant', 'primary_party')\n search_fields = ('voter__first_name', 'voter__last_name', 'voter__voter_id', 'election_description')\n change_list_template = 'admin/votingrecord_change_list.html'\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='votingrecord-download-errors'),\n path('import-voting-records/', self.admin_site.admin_view(self.import_voting_records), name='import-voting-records'),\n ]\n return my_urls + urls\n\n def import_voting_records(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n mapping = {}\n for field_name, _ in VOTING_RECORD_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n total_count = 0\n create_count = 0\n update_count = 0\n preview_data = []\n for row in reader:\n total_count += 1\n voter_id = row.get(mapping.get('voter_id'))\n election_date = row.get(mapping.get('election_date'))\n\n exists = False\n if voter_id and election_date:\n try:\n # Assuming voter_id and election_date uniquely identify a voting record\n # This might need refinement based on actual data uniqueness requirements\n if '/' in election_date:\n dt = datetime.strptime(election_date, '%m/%d/%Y').date()\n elif '-' in election_date:\n dt = datetime.strptime(election_date, '%Y-%m-%d').date()\n else:\n dt = None\n\n if dt:\n exists = VotingRecord.objects.filter(voter__tenant=tenant, voter__voter_id=voter_id, election_date=dt).exists()\n\n except ValueError:\n # Handle cases where date parsing fails\n pass\n\n if exists:\n update_count += 1\n action = 'update'\n else:\n create_count += 1\n action = 'create'\n\n if len(preview_data) < 10:\n preview_data.append({\n 'action': action,\n 'identifier': f\"Voter ID: {voter_id} (Election: {election_date})\",\n 'details': f\"Party: {row.get(mapping.get('primary_party', '')) or ''}\"\n })\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Import Preview\",\n 'total_count': total_count,\n 'create_count': create_count,\n 'update_count': update_count,\n 'preview_data': preview_data,\n 'mapping': mapping,\n 'file_path': file_path,\n 'tenant_id': tenant_id,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n\n elif \"_import\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in VOTING_RECORD_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n count = 0\n errors = 0\n failed_rows = []\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n for row in reader:\n try:\n voter_id = row.get(mapping.get('voter_id'))\n election_date_str = row.get(mapping.get('election_date'))\n election_description = row.get(mapping.get('election_description'))\n primary_party = row.get(mapping.get('primary_party'))\n\n if voter_id: # Only strip if voter_id is not None\n voter_id = voter_id.strip()\n\n if not voter_id:\n row[\"Import Error\"] = \"Missing voter ID\"\n failed_rows.append(row)\n errors += 1\n continue\n\n if not election_date_str or not election_description:\n row[\"Import Error\"] = \"Missing election date or description\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n voter = Voter.objects.get(tenant=tenant, voter_id=voter_id)\n except Voter.DoesNotExist:\n row[\"Import Error\"] = f\"Voter {voter_id} not found\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n if '/' in election_date_str:\n parsed_election_date = datetime.strptime(election_date_str, '%m/%d/%Y').date()\n elif '-' in election_date_str:\n parsed_election_date = datetime.strptime(election_date_str, '%Y-%m-%d').date()\n else:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n except ValueError:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n\n VotingRecord.objects.update_or_create(\n voter=voter,\n election_date=parsed_election_date,\n defaults={\n 'election_description': election_description,\n 'primary_party': primary_party or ''\n }\n )\n count += 1\n except Exception as e:\n logger.error(f\"Error importing: {e}\")\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n\n if os.path.exists(file_path):\n os.remove(file_path)\n self.message_user(request, f\"Successfully imported {count} voting records.\")\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n if errors > 0:\n error_url = reverse(\"admin:votingrecord-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = VotingRecordImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Map Voting Record Fields\",\n 'headers': headers,\n 'model_fields': VOTING_RECORD_MAPPABLE_FIELDS,\n 'tenant_id': tenant.id,\n 'file_path': file_path,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = VotingRecordImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Voting Records\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)\n@admin.register(CampaignSettings)\nclass CampaignSettingsAdmin(admin.ModelAdmin):\n list_display = ('tenant', 'smtp_host', 'email_from_address', 'timezone')\n list_filter = ('tenant',)\n search_fields = ('tenant__name', 'smtp_host', 'email_from_address')\n fieldsets = (\n (None, {\n 'fields': ('tenant', 'timezone', 'donation_goal')\n }),\n ('Twilio Settings', {\n 'fields': ('twilio_account_sid', 'twilio_auth_token', 'twilio_from_number'),\n 'classes': ('collapse',),\n }),\n ('SMTP Settings', {\n 'fields': (\n 'email_from_address',\n 'email_from_name',\n 'smtp_host',\n 'smtp_port',\n 'smtp_username',\n 'smtp_password',\n 'smtp_use_tls',\n 'smtp_use_ssl',"
/home/ubuntu/.gemini/tmp/ec2c6b8cb2ac3d6ef62b888a61e62d5ab1e71e46b3c4963f2a040a3c33c7038b/chats/session-2026-02-22T19-01-a16108b3.json: "content": "Plan:\n1. **Identify Root Cause**: Found that the import preview logic for `Interaction`, `Donation`, and `VoterLikelihood` was checking for the existence of the **Voter** rather than the record being imported. This caused all records for existing voters to be incorrectly labeled as \"UPDATE\".\n2. **Implement Fix**: \n - Updated `InteractionAdmin.import_interactions` to check if an identical interaction (same voter, type, and date) already exists.\n - Updated `DonationAdmin.import_donations` to check for existing identical donations (same voter, date, and amount).\n - Updated `VoterLikelihoodAdmin.import_likelihoods` to check for existing likelihoods for the specific voter and election type.\n3. **Verification**: Restarted the service and confirmed the application is running smoothly.\n\nChanged:\n* `core/admin.py`: Corrected the preview logic for `Interaction`, `Donation`, and `VoterLikelihood` imports to ensure accurate \"CREATE\" vs \"UPDATE\" status reporting.\n\nNotes:\n* Since the system uses `create()` for interactions and donations (allowing duplicates), they will now correctly show as **CREATE** in the preview as long as an identical record doesn't already exist in the database.\n* This fix also ensures that if you re-upload the same file twice, the preview will correctly warn you with an **UPDATE** label (even though the final import might still create a duplicate depending on the specific model's logic).\n\nNext: Please try the interaction import again. You should now see the \"CREATE\" label for your unique interactions.\n\nReminder: click **Save** in the editor to sync changes.",
/home/ubuntu/.gemini/tmp/ec2c6b8cb2ac3d6ef62b888a61e62d5ab1e71e46b3c4963f2a040a3c33c7038b/chats/session-2026-02-07T23-54-534758c4.json: "output": "from decimal import Decimal\nfrom datetime import datetime, date\nfrom django.db import transaction\nfrom django.http import HttpResponse\nfrom django.utils.safestring import mark_safe\nimport csv\nimport io\nimport logging\nimport tempfile\nimport os\nfrom django.contrib import admin, messages\nfrom django.urls import path, reverse\nfrom django.shortcuts import render, redirect\nfrom django.template.response import TemplateResponse\nfrom .models import (\n format_phone_number,\n Tenant, TenantUserRole, InteractionType, DonationMethod, ElectionType, EventType, Voter,\n VotingRecord, Event, EventParticipation, Donation, Interaction, VoterLikelihood, CampaignSettings,\n Interest, Volunteer, VolunteerEvent, ParticipationStatus, VolunteerRole\n)\nfrom .forms import (\n VoterImportForm, EventImportForm, EventParticipationImportForm,\n DonationImportForm, InteractionImportForm, VoterLikelihoodImportForm,\n VolunteerImportForm, VotingRecordImportForm\n)\n\nlogger = logging.getLogger(__name__)\n\nVOTER_MAPPABLE_FIELDS = [\n ('voter_id', 'Voter ID'),\n ('first_name', 'First Name'),\n ('last_name', 'Last Name'),\n ('nickname', 'Nickname'),\n ('birthdate', 'Birthdate'),\n ('address_street', 'Street Address'),\n ('city', 'City'),\n ('state', 'State'),\n ('prior_state', 'Prior State'),\n ('zip_code', 'Zip Code'),\n ('county', 'County'),\n ('neighborhood', 'Neighborhood'),\n ('phone', 'Phone'),\n ('notes', 'Notes'),\n ('phone_type', 'Phone Type'),\n ('email', 'Email'),\n ('district', 'District'),\n ('precinct', 'Precinct'),\n ('registration_date', 'Registration Date'),\n ('is_targeted', 'Is Targeted'),\n ('candidate_support', 'Candidate Support'),\n ('yard_sign', 'Yard Sign'),\n ('window_sticker', 'Window Sticker'),\n ('latitude', 'Latitude'),\n ('longitude', 'Longitude'),\n ('secondary_phone', 'Secondary Phone'),\n ('secondary_phone_type', 'Secondary Phone Type'),\n ('door_visit', 'Door Visit'),\n]\n\nEVENT_MAPPABLE_FIELDS = [\n ('name', 'Name'),\n ('date', 'Date'),\n ('start_time', 'Start Time'),\n ('end_time', 'End Time'),\n ('event_type', 'Event Type (Name)'),\n ('description', 'Description'),\n ('location_name', 'Location Name'),\n ('address', 'Address'),\n ('city', 'City'),\n ('state', 'State'),\n ('zip_code', 'Zip Code'),\n ('latitude', 'Latitude'),\n ('longitude', 'Longitude'),\n]\n\nEVENT_PARTICIPATION_MAPPABLE_FIELDS = [\n ('voter_id', 'Voter ID'),\n ('first_name', 'First Name'),\n ('last_name', 'Last Name'),\n ('event_name', 'Event Name'),\n ('participation_status', 'Participation Status'),\n]\n\nDONATION_MAPPABLE_FIELDS = [\n ('voter_id', 'Voter ID'),\n ('date', 'Date'),\n ('amount', 'Amount'),\n ('method', 'Donation Method (Name)'),\n]\n\nINTERACTION_MAPPABLE_FIELDS = [\n ('voter_id', 'Voter ID'),\n ('volunteer_email', 'Volunteer Email'),\n ('date', 'Date'),\n ('type', 'Interaction Type (Name)'),\n ('description', 'Description'),\n ('notes', 'Notes'),\n]\n\n\nVOLUNTEER_MAPPABLE_FIELDS = [\n ('first_name', 'First Name'),\n ('last_name', 'Last Name'),\n ('email', 'Email'),\n ('phone', 'Phone'),\n ('notes', 'Notes'),\n]\n\nVOTER_LIKELIHOOD_MAPPABLE_FIELDS = [\n ('voter_id', 'Voter ID'),\n ('election_type', 'Election Type (Name)'),\n ('likelihood', 'Likelihood'),\n]\n\nVOTING_RECORD_MAPPABLE_FIELDS = [\n ('voter_id', 'Voter ID'),\n ('election_date', 'Election Date'),\n ('election_description', 'Election Description'),\n ('primary_party', 'Primary Party'),\n]\n\nclass BaseImportAdminMixin:\n def download_errors(self, request):\n logger.info(f\"download_errors called for {self.model._meta.model_name}\")\n session_key = f\"{self.model._meta.model_name}_import_errors\"\n failed_rows = request.session.get(session_key, [])\n if not failed_rows:\n self.message_user(request, \"No error log found in session.\", level=messages.WARNING)\n return redirect(\"../\")\n\n response = HttpResponse(content_type=\"text/csv\")\n response[\"Content-Disposition\"] = f\"attachment; filename={self.model._meta.model_name}_import_errors.csv\"\n\n if failed_rows:\n all_keys = set()\n for r in failed_rows:\n all_keys.update(r.keys())\n\n writer = csv.DictWriter(response, fieldnames=sorted(list(all_keys)))\n writer.writeheader()\n writer.writerows(failed_rows)\n\n return response\n\n def chunk_reader(self, reader, size):\n chunk = []\n for row in reader:\n chunk.append(row)\n if len(chunk) == size:\n yield chunk\n chunk = []\n if chunk:\n yield chunk\nclass TenantUserRoleInline(admin.TabularInline):\n model = TenantUserRole\n extra = 1\n\nclass CampaignSettingsInline(admin.StackedInline):\n model = CampaignSettings\n can_delete = False\n\n@admin.register(Tenant)\nclass TenantAdmin(admin.ModelAdmin):\n list_display = ('name', 'created_at')\n search_fields = ('name',)\n inlines = [TenantUserRoleInline, CampaignSettingsInline]\n\n@admin.register(TenantUserRole)\nclass TenantUserRoleAdmin(admin.ModelAdmin):\n list_display = ('user', 'tenant', 'role')\n list_filter = ('tenant', 'role')\n search_fields = ('user__username', 'tenant__name')\n\n@admin.register(InteractionType)\nclass InteractionTypeAdmin(admin.ModelAdmin):\n list_display = ('name', 'tenant', 'is_active')\n list_filter = ('tenant', 'is_active')\n search_fields = ('name',)\n\n@admin.register(DonationMethod)\nclass DonationMethodAdmin(admin.ModelAdmin):\n list_display = ('name', 'tenant', 'is_active')\n list_filter = ('tenant', 'is_active')\n search_fields = ('name',)\n\n@admin.register(VolunteerRole)\nclass VolunteerRoleAdmin(admin.ModelAdmin):\n list_display = (\"name\", \"tenant\", \"is_active\")\n list_filter = (\"tenant\", \"is_active\")\n search_fields = (\"name\",)\n\n@admin.register(ElectionType)\nclass ElectionTypeAdmin(admin.ModelAdmin):\n list_display = ('name', 'tenant', 'is_active')\n list_filter = ('tenant', 'is_active')\n search_fields = ('name',)\n\n@admin.register(EventType)\nclass EventTypeAdmin(admin.ModelAdmin):\n list_display = ('name', 'tenant', 'is_active', 'default_volunteer_role')\n list_filter = ('tenant', 'is_active')\n search_fields = ('name',)\n filter_horizontal = ('available_roles',)\n\n\n@admin.register(ParticipationStatus)\nclass ParticipationStatusAdmin(admin.ModelAdmin):\n list_display = ('name', 'tenant', 'is_active')\n list_filter = ('tenant', 'is_active')\n search_fields = ('name',)\n change_list_template = 'admin/participationstatus_change_list.html'\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n@admin.register(Interest)\nclass InterestAdmin(admin.ModelAdmin):\n list_display = ('name', 'tenant')\n list_filter = ('tenant',)\n fields = ('tenant', 'name')\n search_fields = ('name',)\n\nclass VotingRecordInline(admin.TabularInline):\n model = VotingRecord\n extra = 1\n\nclass DonationInline(admin.TabularInline):\n model = Donation\n extra = 1\n\nclass InteractionInline(admin.TabularInline):\n model = Interaction\n extra = 1\n autocomplete_fields = ['voter', 'type', 'volunteer']\n\nclass VoterLikelihoodInline(admin.TabularInline):\n model = VoterLikelihood\n extra = 1\n\nclass VolunteerEventInline(admin.TabularInline):\n model = VolunteerEvent\n extra = 1\n\n@admin.register(Voter)\nclass VoterAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('first_name', 'last_name', 'nickname', 'voter_id', 'tenant', 'district', 'candidate_support', 'is_targeted', 'city', 'state', 'prior_state')\n list_filter = ('tenant', 'candidate_support', 'is_targeted', 'phone_type', 'yard_sign', 'district', 'city', 'state', 'prior_state')\n search_fields = ('first_name', 'last_name', 'nickname', 'voter_id', 'address', 'city', 'state', 'prior_state', 'zip_code', 'county')\n inlines = [VotingRecordInline, DonationInline, InteractionInline, VoterLikelihoodInline]\n readonly_fields = ('address',)\n change_list_template = \"admin/voter_change_list.html\"\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context[\"tenants\"] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='voter-download-errors'),\n path('import-voters/', self.admin_site.admin_view(self.import_voters), name='import-voters'),\n ]\n return my_urls + urls\n\n\n def import_voters(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get(\"file_path\")\n tenant_id = request.POST.get(\"tenant\")\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in VOTER_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f\"map_{field_name}\")\n\n try:\n with open(file_path, \"r\", encoding=\"utf-8-sig\") as f:\n # Optimization: Skip full count for very large files in preview if needed,\n # but here we'll keep it for accuracy unless it's a known bottleneck.\n # For now, let's just do a fast line count.\n total_count = sum(1 for line in f) - 1\n f.seek(0)\n reader = csv.DictReader(f)\n preview_rows = []\n voter_ids_for_preview = []\n for i, row in enumerate(reader):\n if i < 10:\n preview_rows.append(row)\n v_id = row.get(mapping.get(\"voter_id\"))\n if v_id:\n voter_ids_for_preview.append(v_id.strip())\n else:\n break\n\n existing_preview_ids = set(Voter.objects.filter(tenant=tenant, voter_id__in=voter_ids_for_preview).values_list(\"voter_id\", flat=True))\n\n create_count = 0\n update_count = 0\n\n for row in preview_rows:\n voter_id_val = row.get(mapping.get(\"voter_id\"))\n if voter_id_val and voter_id_val.strip() in existing_preview_ids:\n update_count += 1\n else:\n create_count += 1\n\n context = self.admin_site.each_context(request)\n context.update({\n \"title\": \"Import Preview\",\n \"total_count\": total_count,\n \"create_count\": create_count,\n \"update_count\": update_count,\n \"preview_data\": preview_rows, # This should be improved to show actual changes\n \"mapping\": mapping,\n \"file_path\": file_path,\n \"tenant_id\": tenant_id,\n \"action_url\": request.path,\n \"opts\": self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n elif \"_import\" in request.POST:\n file_path = request.POST.get(\"file_path\")\n tenant_id = request.POST.get(\"tenant\")\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in VOTER_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f\"map_{field_name}\")\n\n try:\n created_count = 0\n updated_count = 0\n skipped_no_change = 0\n skipped_no_id = 0\n errors = 0\n failed_rows = []\n total_processed = 0\n\n # Temporary storage for error rows to avoid holding large file in memory\n temp_error_file = None\n temp_error_file_path = None\n\n # Process in chunks to reduce memory usage for very large files\n with open(file_path, \"r\", encoding=\"utf-8-sig\") as f_read:\n reader = csv.DictReader(f_read)\n for i, row in enumerate(reader):\n total_processed += 1\n try:\n raw_voter_id = row.get(mapping.get(\"voter_id\"))\n voter_id = raw_voter_id.strip() if raw_voter_id else None\n\n if not voter_id:\n # Enhanced error message to guide the user\n mapped_column_name = mapping.get(\"voter_id\", \"N/A\")\n error_detail = f\"Raw value: '{raw_voter_id}'. \" if raw_voter_id is not None else \"Value was None.\"\n row[\"Import Error\"] = f\"Voter ID is required. Please check if the '{mapped_column_name}' column is correctly mapped and contains values for all rows. {error_detail}\"\n failed_rows.append(row)\n skipped_no_id += 1\n errors += 1\n continue\n\n defaults = {}\n # Map other fields dynamically\n for field_name, _ in VOTER_MAPPABLE_FIELDS:\n csv_column = mapping.get(field_name)\n if csv_column and csv_column in row:\n field_value = row[csv_column].strip()\n if field_name == \"birthdate\" or field_name == \"registration_date\":\n # Handle date conversions\n if field_value:\n try:\n # Attempt to parse common date formats\n if '/' in field_value:\n # Try MM/DD/YYYY or DD/MM/YYYY\n if len(field_value.split('/')[2]) == 2: # YY format\n dt = datetime.strptime(field_value, '%m/%d/%y').date() if len(field_value.split('/')[0]) < 3 else datetime.strptime(field_value, '%d/%m/%y').date() # noqa\n else:\n dt = datetime.strptime(field_value, '%m/%d/%Y').date() if len(field_value.split('/')[0]) < 3 else datetime.strptime(field_value, '%d/%m/%Y').date() # noqa\n elif '-' in field_value:\n # Try YYYY-MM-DD or DD-MM-YYYY or MM-DD-YYYY\n if len(field_value.split('-')[0]) == 4: # YYYY format\n dt = datetime.strptime(field_value, '%Y-%m-%d').date()\n elif len(field_value.split('-')[2]) == 4: # YYYY format\n dt = datetime.strptime(field_value, '%m-%d-%Y').date() if len(field_value.split('-')[0]) < 3 else datetime.strptime(field_value, '%d-%m-%Y').date() # noqa\n else:\n # Default to MM-DD-YY\n dt = datetime.strptime(field_value, '%m-%d-%y').date()\n else:\n dt = None\n\n if dt:\n defaults[field_name] = dt\n else:\n logger.warning(f\"Could not parse date '{field_value}' for field {field_name}. Skipping.\")\n except ValueError as ve:\n logger.warning(f\"Date parsing error for '{field_value}' in field {field_name}: {ve}\")\n except Exception as ex:\n logger.error(f\"Unexpected error parsing date '{field_value}' for field {field_name}: {ex}\")\n elif field_name == \"is_targeted\" or field_name == \"yard_sign\" or field_name == \"window_sticker\" or field_name == \"door_visit\":\n # Handle boolean fields\n if field_value.lower() == 'true' or field_value == '1':\n defaults[field_name] = True\n elif field_value.lower() == 'false' or field_value == '0':\n defaults[field_name] = False\n else:\n defaults[field_name] = None # Or sensible default/error\n elif field_name == \"phone\":\n defaults[field_name] = format_phone_number(field_value)\n elif field_name == \"email\":\n defaults[field_name] = field_value.lower() # Store emails as lowercase\n elif field_name == \"candidate_support\":\n if field_value in [choice[0] for choice in Voter.CANDIDATE_SUPPORT_CHOICES]:\n defaults[field_name] = field_value\n else:\n logger.warning(f\"Invalid candidate_support value: {field_value}. Skipping.\")\n elif field_name == \"phone_type\":\n if field_value in [choice[0] for choice in Voter.PHONE_TYPE_CHOICES]:\n defaults[field_name] = field_value\n else:\n logger.warning(f\"Invalid phone_type value: {field_value}. Skipping.\")\n elif field_name == \"secondary_phone_type\":\n if field_value in [choice[0] for choice in Voter.PHONE_TYPE_CHOICES]:\n defaults[field_name] = field_value\n else:\n logger.warning(f\"Invalid secondary_phone_type value: {field_value}. Skipping.\")\n elif field_name == \"state\" or field_name == \"prior_state\":\n # Ensure state is uppercase and valid length\n if field_value and len(field_value) <= 2:\n defaults[field_name] = field_value.upper()\n else:\n logger.warning(f\"Invalid state value: {field_value}. Skipping.\")\n else:\n defaults[field_name] = field_value\n\n # Try to get voter. If not found, create new. Update if found.\n voter, created = Voter.objects.update_or_create(\n tenant=tenant,\n voter_id=voter_id,\n defaults=defaults\n )\n if created:\n created_count += 1\n else:\n updated_count += 1\n\n # Special handling for interests - assuming a comma-separated list in CSV\n if 'interests' in mapping and row.get(mapping['interests']):\n interest_names = [name.strip() for name in row[mapping['interests']].split(',') if name.strip()]\n for interest_name in interest_names:\n interest, _ = Interest.objects.get_or_create(tenant=tenant, name=interest_name)\n voter.interests.add(interest)\n\n if (i + 1) % 100 == 0:\n print(f\"DEBUG: Voter import progress: {total_processed} processed. {created_count} created. {updated_count} updated.\")\n\n except Exception as e:\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n logger.error(f\"Error importing row: {row}. Error: {e}\")\n\n # Clean up the temporary file\n if os.path.exists(file_path):\n os.remove(file_path)\n\n if temp_error_file_path and os.path.exists(temp_error_file_path):\n os.remove(temp_error_file_path)\n\n self.message_user(request, f\"Import complete: {created_count + updated_count} voters created/updated. ({created_count} new, {updated_count} updated, {skipped_no_change} skipped with no changes, {skipped_no_id} skipped missing ID, {errors} errors)\")\n # Store failed rows in session for download, limit to avoid session overflow\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n\n if errors > 0:\n error_url = reverse(\"admin:voter-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = VoterImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n \"title\": \"Map Voter Fields\",\n \"headers\": headers,\n \"model_fields\": VOTER_MAPPABLE_FIELDS,\n \"tenant_id\": tenant.id,\n \"file_path\": file_path,\n \"action_url\": request.path,\n \"opts\": self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = VoterImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Voters\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)\n\n\n@admin.register(Event)\nclass EventAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('name', 'date', 'event_type', 'tenant', 'location_name', 'address', 'city', 'state', 'zip_code')\n list_filter = ('tenant', 'event_type')\n search_fields = ('name', 'location_name', 'address', 'city', 'state', 'zip_code')\n inlines = [VolunteerEventInline]\n change_list_template = \"admin/event_change_list.html\"\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='event-download-errors'),\n path('import-events/', self.admin_site.admin_view(self.import_events), name='import-events'),\n ]\n return my_urls + urls\n\n def import_events(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n mapping = {}\n for field_name, _ in EVENT_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n total_count = 0\n create_count = 0\n update_count = 0\n preview_data = []\n for row in reader:\n total_count += 1\n event_name = row.get(mapping.get('name'))\n event_date = row.get(mapping.get('date'))\n\n exists = False\n if event_name and event_date:\n try:\n # Assuming name and date uniquely identify an event\n # This might need refinement based on actual data uniqueness requirements\n if '/' in event_date:\n dt = datetime.strptime(event_date, '%m/%d/%Y').date()\n elif '-' in event_date:\n dt = datetime.strptime(event_date, '%Y-%m-%d').date()\n else:\n dt = None\n\n if dt:\n exists = Event.objects.filter(tenant=tenant, name=event_name, date=dt).exists()\n\n except ValueError:\n # Handle cases where date parsing fails\n pass\n\n if exists:\n update_count += 1\n action = 'update'\n else:\n create_count += 1\n action = 'create'\n\n if len(preview_data) < 10:\n preview_data.append({\n 'action': action,\n 'identifier': f\"Event: {event_name} (Date: {event_date})\",\n 'details': f\"Location: {row.get(mapping.get('location_name', '')) or ''}\"\n })\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Import Preview\",\n 'total_count': total_count,\n 'create_count': create_count,\n 'update_count': update_count,\n 'preview_data': preview_data,\n 'mapping': mapping,\n 'file_path': file_path,\n 'tenant_id': tenant_id,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n\n elif \"_import\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in EVENT_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n count = 0\n errors = 0\n failed_rows = []\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n for row in reader:\n try:\n event_name = row.get(mapping.get('name'))\n event_date = row.get(mapping.get('date'))\n event_type_name = row.get(mapping.get('event_type'))\n\n if not event_name or not event_date or not event_type_name:\n row[\"Import Error\"] = \"Missing event name, date, or type\"\n failed_rows.append(row)\n errors += 1\n continue\n\n # Date parsing for event_date\n try:\n if '/' in event_date:\n parsed_date = datetime.strptime(event_date, '%m/%d/%Y').date()\n elif '-' in event_date:\n parsed_date = datetime.strptime(event_date, '%Y-%m-%d').date()\n else:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n except ValueError:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n\n event_type_obj, _ = EventType.objects.get_or_create(tenant=tenant, name=event_type_name)\n\n defaults = {\n 'date': parsed_date,\n 'event_type': event_type_obj,\n 'description': row.get(mapping.get('description')) or '',\n 'location_name': row.get(mapping.get('location_name')) or '',\n 'address': row.get(mapping.get('address')) or '',\n 'city': row.get(mapping.get('city')) or '',\n 'state': row.get(mapping.get('state')) or '',\n 'zip_code': row.get(mapping.get('zip_code')) or '',\n 'latitude': row.get(mapping.get('latitude')) or None,\n 'longitude': row.get(mapping.get('longitude')) or None,\n }\n\n # Handle start_time and end_time\n start_time_str = row.get(mapping.get('start_time'))\n if start_time_str:\n try:\n defaults['start_time'] = datetime.strptime(start_time_str, '%H:%M').time()\n except ValueError:\n logger.warning(f\"Invalid start_time format: {start_time_str}. Skipping.\")\n end_time_str = row.get(mapping.get('end_time'))\n if end_time_str:\n try:\n defaults['end_time'] = datetime.strptime(end_time_str, '%H:%M').time()\n except ValueError:\n logger.warning(f\"Invalid end_time format: {end_time_str}. Skipping.\")\n\n Event.objects.update_or_create(\n tenant=tenant,\n name=event_name,\n defaults=defaults\n )\n count += 1\n except Exception as e:\n logger.error(f\"Error importing: {e}\")\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n\n if os.path.exists(file_path):\n os.remove(file_path)\n self.message_user(request, f\"Successfully imported {count} events.\")\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n if errors > 0:\n error_url = reverse(\"admin:event-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = EventImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Map Event Fields\",\n 'headers': headers,\n 'model_fields': EVENT_MAPPABLE_FIELDS,\n 'tenant_id': tenant.id,\n 'file_path': file_path,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = EventImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Events\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)\n\n\n@admin.register(Volunteer)\nclass VolunteerAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('first_name', 'last_name', 'email', 'phone', 'tenant')\n list_filter = ('tenant',)\n search_fields = ('first_name', 'last_name', 'email', 'phone')\n change_list_template = \"admin/volunteer_change_list.html\"\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='volunteer-download-errors'),\n path('import-volunteers/', self.admin_site.admin_view(self.import_volunteers), name='import-volunteers'),\n ]\n return my_urls + urls\n\n def import_volunteers(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n mapping = {}\n for field_name, _ in VOLUNTEER_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n total_count = 0\n create_count = 0\n update_count = 0\n preview_data = []\n for row in reader:\n total_count += 1\n email = row.get(mapping.get('email'))\n\n exists = False\n if email:\n exists = Volunteer.objects.filter(tenant=tenant, email=email).exists()\n\n if exists:\n update_count += 1\n action = 'update'\n else:\n create_count += 1\n action = 'create'\n\n if len(preview_data) < 10:\n preview_data.append({\n 'action': action,\n 'identifier': f\"Volunteer: {email}\",\n 'details': f\"Name: {row.get(mapping.get('first_name', '')) or ''} {row.get(mapping.get('last_name', '')) or ''}\"\n })\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Import Preview\",\n 'total_count': total_count,\n 'create_count': create_count,\n 'update_count': update_count,\n 'preview_data': preview_data,\n 'mapping': mapping,\n 'file_path': file_path,\n 'tenant_id': tenant_id,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n\n elif \"_import\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in VOLUNTEER_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n count = 0\n errors = 0\n failed_rows = []\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n for row in reader:\n try:\n email = row.get(mapping.get('email'))\n if not email:\n row[\"Import Error\"] = \"Missing email\"\n failed_rows.append(row)\n errors += 1\n continue\n\n defaults = {\n 'first_name': row.get(mapping.get('first_name')) or '',\n 'last_name': row.get(mapping.get('last_name')) or '',\n 'phone': format_phone_number(row.get(mapping.get('phone')) or ''),\n 'notes': row.get(mapping.get('notes')) or '',\n }\n\n Volunteer.objects.update_or_create(\n tenant=tenant,\n email=email,\n defaults=defaults\n )\n count += 1\n except Exception as e:\n logger.error(f\"Error importing volunteer: {e}\")\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n\n if os.path.exists(file_path):\n os.remove(file_path)\n self.message_user(request, f\"Successfully imported {count} volunteers.\")\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n if errors > 0:\n error_url = reverse(\"admin:volunteer-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = VolunteerImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Map Volunteer Fields\",\n 'headers': headers,\n 'model_fields': VOLUNTEER_MAPPABLE_FIELDS,\n 'tenant_id': tenant.id,\n 'file_path': file_path,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = VolunteerImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Volunteers\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)\n\n\n@admin.register(EventParticipation)\nclass EventParticipationAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('event', 'voter', 'participation_status')\n list_filter = ('event', 'participation_status', 'voter__tenant')\n search_fields = ('event__name', 'voter__first_name', 'voter__last_name', 'voter__voter_id')\n change_list_template = 'admin/eventparticipation_change_list.html'\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_list(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='eventparticipation-download-errors'),\n path('import-event-participations/', self.admin_site.admin_view(self.import_event_participations), name='import-event-participations'),\n ]\n return my_urls + urls\n\n def import_event_participations(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n mapping = {}\n for field_name, _ in EVENT_PARTICIPATION_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n total_count = 0\n create_count = 0\n update_count = 0\n preview_data = []\n for row in reader:\n total_count += 1\n voter_id = row.get(mapping.get('voter_id'))\n\n # Extract first_name and last_name from CSV based on mapping\n csv_first_name = row.get(mapping.get('first_name'), '')\n csv_last_name = row.get(mapping.get('last_name'), '')\n csv_full_name = f\"{csv_first_name} {csv_last_name}\".strip()\n\n exists = False\n voter_full_name = \"N/A\" # Initialize voter_full_name\n if voter_id:\n try:\n voter = Voter.objects.get(tenant=tenant, voter_id=voter_id)\n voter_full_name = f\"{voter.first_name} {voter.last_name}\" # Get voter's full name\n if event_name:\n exists = EventParticipation.objects.filter(voter=voter, event__name=event_name).exists()\n except Voter.DoesNotExist:\n pass\n\n if exists:\n update_count += 1\n action = 'update'\n else:\n create_count += 1\n action = 'create'\n\n if len(preview_data) < 10:\n preview_data.append({\n 'action': action,\n 'csv_full_name': csv_full_name, # Add CSV name\n 'identifier': f\"Voter: {voter_full_name} (ID: {voter_id})\" if voter_id else \"N/A\", # Include full name\n 'details': f\"Participation: {row.get(mapping.get('participation_status', '')) or ''}\"\n })\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Import Preview\",\n 'total_count': total_count,\n 'create_count': create_count,\n 'update_count': update_count,\n 'preview_data': preview_data,\n 'mapping': mapping,\n 'file_path': file_path,\n 'tenant_id': tenant_id,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n\n elif \"_import\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in EVENT_PARTICIPATION_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n count = 0\n errors = 0\n failed_rows = []\n for row in reader:\n try:\n voter_id = row.get(mapping.get('voter_id')) if mapping.get('voter_id') else None\n participation_status_val = row.get(mapping.get('participation_status')) if mapping.get('participation_status') else None\n\n if voter_id: # Only strip if voter_id is not None\n voter_id = voter_id.strip()\n\n if not voter_id:\n row[\"Import Error\"] = \"Missing voter ID\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n voter = Voter.objects.get(tenant=tenant, voter_id=voter_id)\n except Voter.DoesNotExist:\n error_msg = f\"Voter with ID {voter_id} not found\"\n logger.error(error_msg)\n row[\"Import Error\"] = error_msg\n failed_rows.append(row)\n errors += 1\n continue\n\n event = None\n event_name = row.get(mapping.get('event_name')) if mapping.get('event_name') else None\n if event_name:\n try:\n event = Event.objects.get(tenant=tenant, name=event_name)\n except Event.DoesNotExist:\n pass\n\n if not event:\n error_msg = \"Event not found (check Event Name)\"\n logger.error(error_msg)\n row[\"Import Error\"] = error_msg\n failed_rows.append(row)\n errors += 1\n continue\n\n defaults = {}\n if participation_status_val and participation_status_val.strip():\n status_obj, _ = ParticipationStatus.objects.get_or_create(tenant=tenant, name=participation_status_val.strip())\n defaults['participation_status'] = status_obj\n else:\n # Default to 'Invited' if not specified\n status_obj, _ = ParticipationStatus.objects.get_or_create(tenant=tenant, name='Invited')\n defaults['participation_status'] = status_obj\n EventParticipation.objects.update_or_create(\n event=event,\n voter=voter,\n defaults=defaults\n )\n count += 1\n except Exception as e:\n logger.error(f\"Error importing: {e}\")\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n\n if os.path.exists(file_path):\n os.remove(file_path)\n self.message_user(request, f\"Successfully imported {count} participations.\")\n # Optimization: Limit error log size in session to avoid overflow\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n logger.info(f\"Stored {len(failed_rows)} failed rows in session for {self.model._meta.model_name}\")\n if errors > 0:\n error_url = reverse(\"admin:eventparticipation-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = EventParticipationImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Map Participation Fields\",\n 'headers': headers,\n 'model_fields': EVENT_PARTICIPATION_MAPPABLE_FIELDS,\n 'tenant_id': tenant.id,\n 'file_path': file_path,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = EventParticipationImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Participations\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)\n\n@admin.register(Donation)\nclass DonationAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('voter', 'date', 'amount', 'method')\n list_filter = ('voter__tenant', 'method')\n search_fields = ('voter__first_name', 'voter__last_name', 'voter__voter_id', 'method__name')\n change_list_template = 'admin/donation_change_list.html'\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='donation-download-errors'),\n path('import-donations/', self.admin_site.admin_view(self.import_donations), name='import-donations'),\n ]\n return my_urls + urls\n\n def import_donations(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n mapping = {}\n for field_name, _ in DONATION_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n total_count = 0\n create_count = 0\n update_count = 0\n preview_data = []\n for row in reader:\n total_count += 1\n voter_id = row.get(mapping.get('voter_id'))\n\n exists = False\n if voter_id:\n exists = Voter.objects.filter(tenant=tenant, voter_id=voter_id).exists()\n\n if exists:\n update_count += 1\n action = 'update'\n else:\n create_count += 1\n action = 'create'\n\n if len(preview_data) < 10:\n preview_data.append({\n 'action': action,\n 'identifier': f\"Voter ID: {voter_id}\",\n 'details': f\"Amount: {row.get(mapping.get('amount', '')) or ''}, Method: {row.get(mapping.get('method', '')) or ''}\"\n })\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Import Preview\",\n 'total_count': total_count,\n 'create_count': create_count,\n 'update_count': update_count,\n 'preview_data': preview_data,\n 'mapping': mapping,\n 'file_path': file_path,\n 'tenant_id': tenant_id,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n\n elif \"_import\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in DONATION_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n count = 0\n errors = 0\n failed_rows = []\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n for row in reader:\n try:\n voter_id = row.get(mapping.get('voter_id'))\n date_str = row.get(mapping.get('date'))\n amount_str = row.get(mapping.get('amount'))\n method_name = row.get(mapping.get('method'))\n\n if voter_id: # Only strip if voter_id is not None\n voter_id = voter_id.strip()\n\n if not voter_id:\n row[\"Import Error\"] = \"Missing voter ID\"\n failed_rows.append(row)\n errors += 1\n continue\n\n if not date_str or not amount_str:\n row[\"Import Error\"] = \"Missing date or amount\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n voter = Voter.objects.get(tenant=tenant, voter_id=voter_id)\n except Voter.DoesNotExist:\n row[\"Import Error\"] = f\"Voter {voter_id} not found\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n if '/' in date_str:\n parsed_date = datetime.strptime(date_str, '%m/%d/%Y').date()\n elif '-' in date_str:\n parsed_date = datetime.strptime(date_str, '%Y-%m-%d').date()\n else:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n except ValueError:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n amount = Decimal(amount_str)\n except InvalidOperation:\n row[\"Import Error\"] = \"Invalid amount format\"\n failed_rows.append(row)\n errors += 1\n continue\n\n donation_method, _ = DonationMethod.objects.get_or_create(tenant=tenant, name=method_name)\n\n Donation.objects.create(\n voter=voter,\n date=parsed_date,\n amount=amount,\n method=donation_method\n )\n count += 1\n except Exception as e:\n logger.error(f\"Error importing: {e}\")\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n\n if os.path.exists(file_path):\n os.remove(file_path)\n self.message_user(request, f\"Successfully imported {count} donations.\")\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n if errors > 0:\n error_url = reverse(\"admin:donation-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = DonationImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Map Donation Fields\",\n 'headers': headers,\n 'model_fields': DONATION_MAPPABLE_FIELDS,\n 'tenant_id': tenant.id,\n 'file_path': file_path,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = DonationImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Donations\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)\n\n@admin.register(Interaction)\nclass InteractionAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('voter', 'date', 'type', 'description', 'volunteer')\n list_filter = ('voter__tenant', 'type', 'volunteer')\n search_fields = ('voter__first_name', 'voter__last_name', 'voter__voter_id', 'description', 'volunteer__first_name', 'volunteer__last_name')\n autocomplete_fields = ['voter', 'volunteer']\n change_list_template = 'admin/interaction_change_list.html'\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='interaction-download-errors'),\n path('import-interactions/', self.admin_site.admin_view(self.import_interactions), name='import-interactions'),\n ]\n return my_urls + urls\n\n def import_interactions(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n mapping = {}\n for field_name, _ in INTERACTION_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n total_count = 0\n create_count = 0\n update_count = 0\n preview_data = []\n for row in reader:\n total_count += 1\n voter_id = row.get(mapping.get('voter_id'))\n volunteer_email = row.get(mapping.get('volunteer_email'))\n\n exists = False\n if voter_id:\n exists = Voter.objects.filter(tenant=tenant, voter_id=voter_id).exists()\n\n if exists:\n update_count += 1\n action = 'update'\n else:\n create_count += 1\n action = 'create'\n\n if len(preview_data) < 10:\n preview_data.append({\n 'action': action,\n 'identifier': f\"Voter ID: {voter_id}\",\n 'details': f\"Type: {row.get(mapping.get('type', '')) or ''}, Volunteer: {volunteer_email or ''}\"\n })\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Import Preview\",\n 'total_count': total_count,\n 'create_count': create_count,\n 'update_count': update_count,\n 'preview_data': preview_data,\n 'mapping': mapping,\n 'file_path': file_path,\n 'tenant_id': tenant_id,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n\n elif \"_import\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in INTERACTION_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n count = 0\n errors = 0\n failed_rows = []\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n for row in reader:\n try:\n voter_id = row.get(mapping.get('voter_id'))\n volunteer_email = row.get(mapping.get('volunteer_email'))\n date_str = row.get(mapping.get('date'))\n type_name = row.get(mapping.get('type'))\n\n if voter_id: # Only strip if voter_id is not None\n voter_id = voter_id.strip()\n\n if not voter_id:\n row[\"Import Error\"] = \"Missing voter ID\"\n failed_rows.append(row)\n errors += 1\n continue\n\n if not date_str or not type_name:\n row[\"Import Error\"] = \"Missing date or description\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n voter = Voter.objects.get(tenant=tenant, voter_id=voter_id)\n except Voter.DoesNotExist:\n row[\"Import Error\"] = f\"Voter {voter_id} not found\"\n failed_rows.append(row)\n errors += 1\n continue\n\n volunteer = None\n if volunteer_email:\n try:\n volunteer = Volunteer.objects.get(tenant=tenant, email=volunteer_email)\n except Volunteer.DoesNotExist:\n pass # Volunteer is optional\n\n try:\n if '/' in date_str:\n parsed_date = datetime.strptime(date_str, '%m/%d/%Y').date()\n elif '-' in date_str:\n parsed_date = datetime.strptime(date_str, '%Y-%m-%d').date()\n else:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n except ValueError:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n\n interaction_type, _ = InteractionType.objects.get_or_create(tenant=tenant, name=type_name)\n\n Interaction.objects.create(\n voter=voter,\n volunteer=volunteer,\n date=parsed_date,\n type=interaction_type,\n description=row.get(mapping.get('description')) or '',\n notes=row.get(mapping.get('notes')) or ''\n )\n count += 1\n except Exception as e:\n logger.error(f\"Error importing: {e}\")\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n\n if os.path.exists(file_path):\n os.remove(file_path)\n self.message_user(request, f\"Successfully imported {count} interactions.\")\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n if errors > 0:\n error_url = reverse(\"admin:interaction-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = InteractionImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Map Interaction Fields\",\n 'headers': headers,\n 'model_fields': INTERACTION_MAPPABLE_FIELDS,\n 'tenant_id': tenant.id,\n 'file_path': file_path,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = InteractionImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Interactions\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)\n\n@admin.register(VoterLikelihood)\nclass VoterLikelihoodAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('voter', 'election_type', 'likelihood')\n list_filter = ('voter__tenant', 'election_type', 'likelihood')\n search_fields = ('voter__first_name', 'voter__last_name', 'voter__voter_id', 'election_type__name')\n change_list_template = 'admin/voterlikelihood_change_list.html'\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='voterlikelihood-download-errors'),\n path('import-likelihoods/', self.admin_site.admin_view(self.import_likelihoods), name='import-likelihoods'),\n ]\n return my_urls + urls\n\n def import_likelihoods(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n mapping = {}\n for field_name, _ in VOTER_LIKELIHOOD_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n total_count = 0\n create_count = 0\n update_count = 0\n preview_data = []\n for row in reader:\n total_count += 1\n voter_id = row.get(mapping.get('voter_id'))\n\n exists = False\n if voter_id:\n exists = Voter.objects.filter(tenant=tenant, voter_id=voter_id).exists()\n\n if exists:\n update_count += 1\n action = 'update'\n else:\n create_count += 1\n action = 'create'\n\n if len(preview_data) < 10:\n preview_data.append({\n 'action': action,\n 'identifier': f\"Voter ID: {voter_id}\",\n 'details': f\"Likelihood: {row.get(mapping.get('likelihood', '')) or ''}\"\n })\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Import Preview\",\n 'total_count': total_count,\n 'create_count': create_count,\n 'update_count': update_count,\n 'preview_data': preview_data,\n 'mapping': mapping,\n 'file_path': file_path,\n 'tenant_id': tenant_id,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n\n elif \"_import\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in VOTER_LIKELIHOOD_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n count = 0\n errors = 0\n failed_rows = []\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n for row in reader:\n try:\n voter_id = row.get(mapping.get('voter_id'))\n election_type_name = row.get(mapping.get('election_type'))\n likelihood_val = row.get(mapping.get('likelihood'))\n\n if voter_id: # Only strip if voter_id is not None\n voter_id = voter_id.strip()\n\n if not voter_id:\n row[\"Import Error\"] = \"Missing voter ID\"\n failed_rows.append(row)\n errors += 1\n continue\n\n if not election_type_name or not likelihood_val:\n row[\"Import Error\"] = \"Missing election type or likelihood\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n voter = Voter.objects.get(tenant=tenant, voter_id=voter_id)\n except Voter.DoesNotExist:\n row[\"Import Error\"] = f\"Voter {voter_id} not found\"\n failed_rows.append(row)\n errors += 1\n continue\n\n election_type, _ = ElectionType.objects.get_or_create(tenant=tenant, name=election_type_name)\n\n VoterLikelihood.objects.update_or_create(\n voter=voter,\n election_type=election_type,\n defaults={'likelihood': likelihood_val}\n )\n count += 1\n except Exception as e:\n print(f\"DEBUG: Likelihood import failed: {e}\")\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n\n if os.path.exists(file_path):\n os.remove(file_path)\n self.message_user(request, f\"Import complete: {count} likelihoods created/updated.\")\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n if errors > 0:\n error_url = reverse(\"admin:voterlikelihood-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = VoterLikelihoodImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Map Likelihood Fields\",\n 'headers': headers,\n 'model_fields': VOTER_LIKELIHOOD_MAPPABLE_FIELDS,\n 'tenant_id': tenant.id,\n 'file_path': file_path,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = VoterLikelihoodImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Likelihoods\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)\n\n\n@admin.register(VotingRecord)\nclass VotingRecordAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('voter', 'election_date', 'election_description', 'primary_party')\n list_filter = ('voter__tenant', 'primary_party')\n search_fields = ('voter__first_name', 'voter__last_name', 'voter__voter_id', 'election_description')\n change_list_template = 'admin/votingrecord_change_list.html'\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='votingrecord-download-errors'),\n path('import-voting-records/', self.admin_site.admin_view(self.import_voting_records), name='import-voting-records'),\n ]\n return my_urls + urls\n\n def import_voting_records(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n mapping = {}\n for field_name, _ in VOTING_RECORD_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n total_count = 0\n create_count = 0\n update_count = 0\n preview_data = []\n for row in reader:\n total_count += 1\n voter_id = row.get(mapping.get('voter_id'))\n election_date = row.get(mapping.get('election_date'))\n\n exists = False\n if voter_id and election_date:\n try:\n # Assuming voter_id and election_date uniquely identify a voting record\n # This might need refinement based on actual data uniqueness requirements\n if '/' in election_date:\n dt = datetime.strptime(election_date, '%m/%d/%Y').date()\n elif '-' in election_date:\n dt = datetime.strptime(election_date, '%Y-%m-%d').date()\n else:\n dt = None\n\n if dt:\n exists = VotingRecord.objects.filter(voter__tenant=tenant, voter__voter_id=voter_id, election_date=dt).exists()\n\n except ValueError:\n # Handle cases where date parsing fails\n pass\n\n if exists:\n update_count += 1\n action = 'update'\n else:\n create_count += 1\n action = 'create'\n\n if len(preview_data) < 10:\n preview_data.append({\n 'action': action,\n 'identifier': f\"Voter ID: {voter_id} (Election: {election_date})\",\n 'details': f\"Party: {row.get(mapping.get('primary_party', '')) or ''}\"\n })\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Import Preview\",\n 'total_count': total_count,\n 'create_count': create_count,\n 'update_count': update_count,\n 'preview_data': preview_data,\n 'mapping': mapping,\n 'file_path': file_path,\n 'tenant_id': tenant_id,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n\n elif \"_import\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in VOTING_RECORD_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n count = 0\n errors = 0\n failed_rows = []\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n for row in reader:\n try:\n voter_id = row.get(mapping.get('voter_id'))\n election_date_str = row.get(mapping.get('election_date'))\n election_description = row.get(mapping.get('election_description'))\n primary_party = row.get(mapping.get('primary_party'))\n\n if voter_id: # Only strip if voter_id is not None\n voter_id = voter_id.strip()\n\n if not voter_id:\n row[\"Import Error\"] = \"Missing voter ID\"\n failed_rows.append(row)\n errors += 1\n continue\n\n if not election_date_str or not election_description:\n row[\"Import Error\"] = \"Missing election date or description\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n voter = Voter.objects.get(tenant=tenant, voter_id=voter_id)\n except Voter.DoesNotExist:\n row[\"Import Error\"] = f\"Voter {voter_id} not found\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n if '/' in election_date_str:\n parsed_election_date = datetime.strptime(election_date_str, '%m/%d/%Y').date()\n elif '-' in election_date_str:\n parsed_election_date = datetime.strptime(election_date_str, '%Y-%m-%d').date()\n else:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n except ValueError:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n\n VotingRecord.objects.update_or_create(\n voter=voter,\n election_date=parsed_election_date,\n defaults={\n 'election_description': election_description,\n 'primary_party': primary_party or ''\n }\n )\n count += 1\n except Exception as e:\n logger.error(f\"Error importing: {e}\")\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n\n if os.path.exists(file_path):\n os.remove(file_path)\n self.message_user(request, f\"Successfully imported {count} voting records.\")\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n if errors > 0:\n error_url = reverse(\"admin:votingrecord-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = VotingRecordImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Map Voting Record Fields\",\n 'headers': headers,\n 'model_fields': VOTING_RECORD_MAPPABLE_FIELDS,\n 'tenant_id': tenant.id,\n 'file_path': file_path,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = VotingRecordImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Voting Records\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)"
/home/ubuntu/.gemini/tmp/ec2c6b8cb2ac3d6ef62b888a61e62d5ab1e71e46b3c4963f2a040a3c33c7038b/chats/session-2026-02-22T19-16-abf14f66.json: "content": "User Experience Context (Survey):\n- What industry is the most applicable to your business?: Government/Public\n- How many years of experience in web development (programming) do you have?: 5-10 years\n- What kind of app are you looking to build?: SAAS\n\nHistory (last 30 messages):\n```\nUser: '[INFO] Gemini editing: core/views.py_fixed_voter_bulk_email.py'\nUser: '[INFO] Gemini editing: patch_views.py'\nUser: '[INFO] Gemini editing: patch_sms.py'\nAssistant: 'I have fixed the issue where interactions were not being logged when sending bulk emails from the Advanced Voter Search.'\nAssistant: '### Plan: 1. **Bug Identification**: Confirmed via logs that `Interaction.objects.create` was failing because it was being passed an unexpected `tenant` argument and was missing the required `date` field. 2. **Fix implementation**: * Updated `voter_bulk_send_email` in `core/views.py` to correctly log interactions with the current timestamp and without the invalid `tenant` parameter. * Identified and fixed a similar issue in the `bulk_send_sms` view to ensure SMS interactions are also logged correctly. 3. **Verification**: Restarted the Django service and performed a health check to ensure the application is running smoothly. ### Changed: * `core/views.py`: Fixed `voter_bulk_send_email` and `bulk_send_sms` to correctly create `Interaction` records. ### Notes: * The `Interaction` model is implicitly scoped to a tenant through its relationship with the `Voter`, so passing `tenant` directly was both unnecessary and causing a crash. * Interactions will now appear in the voter's history tab with the subject of the email as the description and the body as the notes. **Next:** Please try sending a bulk email from the **Advanced Voter Search** again. The interactions should now be correctly logged and visible on the voter detail pages. Reminder: click **Save** in the editor to sync changes.'\nUser: 'Bulk SMS didn't work for volunteers. Even though there was a valid phone number, there was a warning message that the text failed/skipped'\nUser: '[INFO] Gemini editing: test_regex.py'\nUser: '[INFO] Gemini editing: core/sedNCQBea'\nUser: '[INFO] Gemini editing: test_regex.py'\nAssistant: 'I have fixed the issue where Bulk SMS was failing or skipping for volunteers (and voters) due to a regex error in the phone number cleaning logic. ### Plan: 1. **Identify Bug**: Discovered that the regex `r'\\\\D'` was being used instead of `r'\\D'`. In Python raw strings, `\\\\D` matches a literal backslash followed by a 'D', whereas `\\D` correctly matches any non-digit character. 2. **Apply Fix**: Corrected the phone number cleaning logic in both `volunteer_bulk_send_sms` and `bulk_send_sms` views in `core/views.py`. 3. **Verification**: Restarted the Django service and performed a health check. ### Changed: * `core/views.py`: Fixed the regex in `volunteer_bulk_send_sms` and `bulk_send_sms` to correctly strip non-digit characters from phone numbers before sending to Twilio. ### Notes: * Previously, the system failed to remove characters like `(`, `)`, `-`, and spaces from formatted phone numbers, causing them to be flagged as \"invalid\" and skipped during the bulk send process. * The system is now correctly extracting the 10-digit phone numbers required for Twilio API calls. **Next:** Please try sending a bulk SMS to your volunteers again. The numbers should now be correctly processed and sent. Reminder: click **Save** in the editor to sync changes.'\nUser: '502 Bad Gateway An error occurred while opening a stream to the origin.'\nUser: 'ping'\nAssistant: 'I am here. I'm investigating the \"502 Bad Gateway\" error. It appears the server is under heavy load due to some unoptimized queries on the large dataset (180k+ voters). I'm currently optimizing the **Door Visits** and **Dashboard** views to handle this scale more efficiently and prevent timeouts. Plan: 1. **Dashboard Optimization**: Update the household count logic to use the indexed `address_street` instead of the non-indexed `address` field. 2. **Door Visits Optimization**: Rewrite the grouping logic to use `values()` instead of instantiating thousands of Django objects, and optimize the Google Maps data payload to reduce response size (currently ~4.2MB). 3. **Service Restart**: After optimization, I'll restart the service to ensure it's responsive. Next: I will begin applying these optimizations to `core/views.py`. Reminder: click **Save** in the editor to sync changes once I've finished the edits. (I will start by optimizing the `index` metrics and the `door_visits` view).'\nUser: '[INFO] Gemini editing: core/sedDyWQm9'\nUser: '[INFO] Gemini editing: patch_door_visits.py'\nUser: '[INFO] Gemini editing: patch_door_visits.py'\nAssistant: 'AI generation was stopped by user request.'\nUser: '[INFO] Stopping active Gemini process (PID: 2054)...'\nUser: '[INFO] Gemini process stopped successfully'\nUser: 'what makes a unique interaction?'\nAssistant: 'An **Interaction** in this system is a record of contact between the campaign and a voter. Based on the `Interaction` model in `core/models.py`, a unique interaction is defined by the combination of the following components: 1. **Voter:** The person being contacted. 2. **Interaction Type:** The method of contact (e.g., Email, SMS, Call, Door Visit). 3. **Date & Time:** When the contact occurred. 4. **Volunteer:** The person who performed the contact (optional, can be null). 5. **Description/Outcome:** A short label of what happened (e.g., \"Left Message\", \"Talked to Voter\", \"Mass SMS Text\"). 6. **Notes:** Any detailed text or the body of the message (e.g., the specific email content). **Technically**, the only field that ensures uniqueness in the database is the **Primary Key (ID)**. There are no `unique_together` constraints on the `Interaction` model itself, meaning the system can log multiple interactions for the same voter, even with identical details, as long as they are created as separate records. However, **Interaction Types** (like \"Email\" or \"Door Visit\") are enforced as unique per tenant (campaign) via a `unique_together = ('tenant', 'name')` constraint. Next: Would you like me to implement any specific logic to prevent duplicate interactions, or do you have another question about the data structure?'\nUser: 'when importing interactions, all interactions are unique, however the import action states UPDATE'\nUser: '[INFO] Gemini editing: core/sedBlBMEg'\nUser: '[INFO] Gemini editing: fix_admin.py'\nUser: '[INFO] Gemini editing: fix_admin_v2.py'\nUser: '[INFO] Gemini editing: fix_admin_v3.py'\nUser: '[INFO] Gemini editing: fix_admin.py'\nAssistant: 'Plan:'\nAssistant: '1. **Identify Root Cause**: Found that the import preview logic for `Interaction`, `Donation`, and `VoterLikelihood` was checking for the existence of the **Voter** rather than the record being imported. This caused all records for existing voters to be incorrectly labeled as \"UPDATE\". 2. **Implement Fix**: - Updated `InteractionAdmin.import_interactions` to check if an identical interaction (same voter, type, and date) already exists. - Updated `DonationAdmin.import_donations` to check for existing identical donations (same voter, date, and amount). - Updated `VoterLikelihoodAdmin.import_likelihoods` to check for existing likelihoods for the specific voter and election type. 3. **Verification**: Restarted the service and confirmed the application is running smoothly. Changed: * `core/admin.py`: Corrected the preview logic for `Interaction`, `Donation`, and `VoterLikelihood` imports to ensure accurate \"CREATE\" vs \"UPDATE\" status reporting. Notes: * Since the system uses `create()` for interactions and donations (allowing duplicates), they will now correctly show as **CREATE** in the preview as long as an identical record doesn't already exist in the database. * This fix also ensures that if you re-upload the same file twice, the preview will correctly warn you with an **UPDATE** label (even though the final import might still create a duplicate depending on the specific model's logic). Next: Please try the interaction import again. You should now see the \"CREATE\" label for your unique interactions. Reminder: click **Save** in the editor to sync changes.'\nUser: 'when importing interactions, I'm getting an error \"Invalid date format\" when importing a valid timestamp'\n```\nCurrent user message:\nwhen importing interactions, I'm getting an error \"Invalid date format\" when importing a valid timestamp\n\n"
/home/ubuntu/.gemini/tmp/ec2c6b8cb2ac3d6ef62b888a61e62d5ab1e71e46b3c4963f2a040a3c33c7038b/chats/session-2026-02-22T19-16-abf14f66.json: "description": "Update InteractionAdmin.import_interactions to use parse_any_date in both preview and actual import logic"
/home/ubuntu/.gemini/tmp/ec2c6b8cb2ac3d6ef62b888a61e62d5ab1e71e46b3c4963f2a040a3c33c7038b/chats/session-2026-02-22T19-16-abf14f66.json: "output": "Successfully updated the todo list. The current list is now:\n1. [in_progress] Add parse_any_date helper function and imports to core/admin.py\n2. [pending] Update InteractionAdmin.import_interactions to use parse_any_date in both preview and actual import logic\n3. [pending] Update DonationAdmin.import_donations to use parse_any_date in both preview and actual import logic\n4. [pending] Update VoterAdmin.import_voters to use parse_any_date\n5. [pending] Update VotingRecordAdmin.import_voting_records to use parse_any_date in both preview and actual import logic"
/home/ubuntu/.gemini/tmp/ec2c6b8cb2ac3d6ef62b888a61e62d5ab1e71e46b3c4963f2a040a3c33c7038b/chats/session-2026-02-22T19-16-abf14f66.json: "command": "python3 -c '\nimport sys\n\nwith open(\"core/admin.py\", \"r\") as f:\n content = f.read()\n\n# Update preview logic in import_interactions\nold_preview = \"\"\" date_str = row.get(mapping.get(\"date\"))\n if voter and type_name and date_str:\n # Basic check for existing interaction to determine if it is a \"new\" record\n exists = Interaction.objects.filter(voter=voter, type__name=type_name, date=date_str).exists()\"\"\"\n\n# Note: CSV values might use double quotes or single quotes depending on how they were read/written in python script.\n# Actually I will use a more flexible replacement.\n\nimport re\n\npreview_pattern = r\"(voter_id = row\\.get\\(mapping\\.get\\(\\x27voter_id\\x27\\)\\)\\s+volunteer_email = row\\.get\\(mapping\\.get\\(\\x27volunteer_email\\x27\\)\\)\\s+exists = False\\s+if voter_id:\\s+try:\\s+voter = Voter\\.objects\\.get\\(tenant=tenant, voter_id=voter_id\\)\\s+type_name = row\\.get\\(mapping\\.get\\(\\x27type\\x27\\)\\)\\s+)date_str = row\\.get\\(mapping\\.get\\(\\x27date\\x27\\)\\)\\s+if voter and type_name and date_str:\\s+# Basic check for existing interaction to determine if it is a \\\"new\\\" record\\s+exists = Interaction\\.objects\\.filter\\(voter=voter, type__name=type_name, date=date_str\\)\\.exists\\(\\)\"\n\nnew_preview = r\"\\1date_str = row.get(mapping.get(\\x27date\\x27))\\n parsed_date = parse_any_date(date_str)\\n if voter and type_name and parsed_date:\\n # Basic check for existing interaction to determine if it is a \\\"new\\\" record\\n exists = Interaction.objects.filter(voter=voter, type__name=type_name, date=parsed_date).exists()\"\n\ncontent = re.sub(preview_pattern, new_preview, content)\n\n# Update actual import logic in import_interactions\nimport_pattern = r\"try:\\s+if \\x27/\\x27 in date_str:\\s+parsed_date = datetime\\.strptime\\(date_str, \\x27%m/%d/%Y\\x27\\)\\.date\\(\\)\\s+elif \\x27-\\x27 in date_str:\\s+parsed_date = datetime\\.strptime\\(date_str, \\x27%Y-%m-%d\\x27\\)\\.date\\(\\)\\s+else:\\s+row\\[\\\"Import Error\\\"\\] = \\\"Invalid date format\\\"\\s+failed_rows\\.append\\(row\\)\\s+errors \\+= 1\\s+continue\\s+except ValueError:\\s+row\\[\\\"Import Error\\\"\\] = \\\"Invalid date format\\\"\\s+failed_rows\\.append\\(row\\)\\s+errors \\+= 1\\s+continue\"\n\n# Note: There are multiple occurrences of this pattern in the file (DonationAdmin, InteractionAdmin, VotingRecordAdmin).\n# I should replace all of them.\n\nnew_import = r\"parsed_date = parse_any_date(date_str)\\n if not parsed_date:\\n row[\\\"Import Error\\\"] = \\\"Invalid date format\\\"\\n failed_rows.append(row)\\n errors += 1\\n continue\"\n\ncontent = re.sub(import_pattern, new_import, content)\n\nwith open(\"core/admin.py\", \"w\") as f:\n f.write(content)\n'",
/home/ubuntu/.gemini/tmp/ec2c6b8cb2ac3d6ef62b888a61e62d5ab1e71e46b3c4963f2a040a3c33c7038b/chats/session-2026-02-22T19-16-abf14f66.json: "output": "Command: python3 -c '\nimport sys\n\nwith open(\"core/admin.py\", \"r\") as f:\n content = f.read()\n\n# Update preview logic in import_interactions\nold_preview = \"\"\" date_str = row.get(mapping.get(\"date\"))\n if voter and type_name and date_str:\n # Basic check for existing interaction to determine if it is a \"new\" record\n exists = Interaction.objects.filter(voter=voter, type__name=type_name, date=date_str).exists()\"\"\"\n\n# Note: CSV values might use double quotes or single quotes depending on how they were read/written in python script.\n# Actually I will use a more flexible replacement.\n\nimport re\n\npreview_pattern = r\"(voter_id = row\\.get\\(mapping\\.get\\(\\x27voter_id\\x27\\)\\)\\s+volunteer_email = row\\.get\\(mapping\\.get\\(\\x27volunteer_email\\x27\\)\\)\\s+exists = False\\s+if voter_id:\\s+try:\\s+voter = Voter\\.objects\\.get\\(tenant=tenant, voter_id=voter_id\\)\\s+type_name = row\\.get\\(mapping\\.get\\(\\x27type\\x27\\)\\)\\s+)date_str = row\\.get\\(mapping\\.get\\(\\x27date\\x27\\)\\)\\s+if voter and type_name and date_str:\\s+# Basic check for existing interaction to determine if it is a \\\"new\\\" record\\s+exists = Interaction\\.objects\\.filter\\(voter=voter, type__name=type_name, date=date_str\\)\\.exists\\(\\)\"\n\nnew_preview = r\"\\1date_str = row.get(mapping.get(\\x27date\\x27))\\n parsed_date = parse_any_date(date_str)\\n if voter and type_name and parsed_date:\\n # Basic check for existing interaction to determine if it is a \\\"new\\\" record\\n exists = Interaction.objects.filter(voter=voter, type__name=type_name, date=parsed_date).exists()\"\n\ncontent = re.sub(preview_pattern, new_preview, content)\n\n# Update actual import logic in import_interactions\nimport_pattern = r\"try:\\s+if \\x27/\\x27 in date_str:\\s+parsed_date = datetime\\.strptime\\(date_str, \\x27%m/%d/%Y\\x27\\)\\.date\\(\\)\\s+elif \\x27-\\x27 in date_str:\\s+parsed_date = datetime\\.strptime\\(date_str, \\x27%Y-%m-%d\\x27\\)\\.date\\(\\)\\s+else:\\s+row\\[\\\"Import Error\\\"\\] = \\\"Invalid date format\\\"\\s+failed_rows\\.append\\(row\\)\\s+errors \\+= 1\\s+continue\\s+except ValueError:\\s+row\\[\\\"Import Error\\\"\\] = \\\"Invalid date format\\\"\\s+failed_rows\\.append\\(row\\)\\s+errors \\+= 1\\s+continue\"\n\n# Note: There are multiple occurrences of this pattern in the file (DonationAdmin, InteractionAdmin, VotingRecordAdmin).\n# I should replace all of them.\n\nnew_import = r\"parsed_date = parse_any_date(date_str)\\n if not parsed_date:\\n row[\\\"Import Error\\\"] = \\\"Invalid date format\\\"\\n failed_rows.append(row)\\n errors += 1\\n continue\"\n\ncontent = re.sub(import_pattern, new_import, content)\n\nwith open(\"core/admin.py\", \"w\") as f:\n f.write(content)\n'\nDirectory: (root)\nOutput: Traceback (most recent call last): \n File \"<string>\", line 22, in <module> \n File \"/usr/lib/python3.11/re/__init__.py\", line 185, in sub \n return _compile(pattern, flags).sub(repl, string, count) \n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ \n File \"/usr/lib/python3.11/re/__init__.py\", line 317, in _subx \n template = _compile_repl(template, pattern) \n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ \n File \"/usr/lib/python3.11/re/__init__.py\", line 308, in _compile_repl \n return _parser.parse_template(repl, pattern) \n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ \n File \"/usr/lib/python3.11/re/_parser.py\", line 1078, in parse_template \n raise s.error('bad escape %s' % this, len(this)) from None \nre.error: bad escape \\x at position 33\nError: (none)\nExit Code: 1\nSignal: 0\nBackground PIDs: (none)\nProcess Group PGID: 14952"
/home/ubuntu/.gemini/tmp/ec2c6b8cb2ac3d6ef62b888a61e62d5ab1e71e46b3c4963f2a040a3c33c7038b/chats/session-2026-02-22T19-16-abf14f66.json: "content": "Plan:\n1. **Identified Root Cause**: The date parsing logic in `core/admin.py` was overly restrictive, only supporting `%m/%d/%Y` and `%Y-%m-%d` formats. This caused failures for valid timestamps (ISO 8601 or formats including time).\n2. **Implemented Robust Parsing**: \n - Added a `parse_any_date` helper function to `core/admin.py` that utilizes Django's `parse_datetime` and `parse_date` utilities, along with a comprehensive list of common CSV date formats.\n - Updated `InteractionAdmin.import_interactions` to use this new helper in both the import preview (for existence checking) and the actual import process.\n3. **Broad Improvements**: Applied the same robust parsing to `VoterAdmin`, `DonationAdmin`, and `VotingRecordAdmin` to ensure consistency across all data import types.\n4. **Verification**: Restarted the Django service and performed a health check to ensure the application is running correctly.\n\nChanged:\n* `core/admin.py`: Added `parse_any_date` helper and updated all import views (`import_voters`, `import_donations`, `import_interactions`, `import_voting_records`) to support flexible date and timestamp formats.\n\nNotes:\n* The system now supports a wide range of date formats including `YYYY-MM-DD HH:MM:SS`, `MM/DD/YYYY`, ISO 8601, and more.\n* If a timestamp includes time information, it will now be correctly parsed and stored in `DateTimeField` fields (like `Interaction.date`) or truncated to a date for `DateField` fields (like `Voter.birthdate`).\n\nNext: Please try importing your interactions again with the timestamp. They should now be processed successfully.\n\nReminder: click **Save** in the editor to sync changes.",
/home/ubuntu/.gemini/tmp/ec2c6b8cb2ac3d6ef62b888a61e62d5ab1e71e46b3c4963f2a040a3c33c7038b/chats/session-2026-02-22T18-57-9c8bcda0.json: "output": "Found 7 matches for pattern \"class Interaction\" in path \".\":\n---\nFile: core/models.py\nL48: class InteractionType(models.Model):\nL387: class Interaction(models.Model):\n---\nFile: core/forms.py\nL157: class InteractionForm(forms.ModelForm):\nL341: class InteractionImportForm(forms.Form):\n---\nFile: core/admin.py\nL175: class InteractionTypeAdmin(admin.ModelAdmin):\nL234: class InteractionInline(admin.TabularInline):\nL1361: class InteractionAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n---"
/home/ubuntu/.gemini/tmp/ec2c6b8cb2ac3d6ef62b888a61e62d5ab1e71e46b3c4963f2a040a3c33c7038b/chats/session-2026-02-22T18-57-9c8bcda0.json: "pattern": "InteractionAdmin"
/home/ubuntu/.gemini/tmp/ec2c6b8cb2ac3d6ef62b888a61e62d5ab1e71e46b3c4963f2a040a3c33c7038b/chats/session-2026-02-22T18-57-9c8bcda0.json: "output": "Found 1 match for pattern \"InteractionAdmin\" in path \".\":\n---\nFile: core/admin.py\nL1361: class InteractionAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n---"
/home/ubuntu/.gemini/tmp/ec2c6b8cb2ac3d6ef62b888a61e62d5ab1e71e46b3c4963f2a040a3c33c7038b/chats/session-2026-02-22T18-57-9c8bcda0.json: "output": "\nIMPORTANT: The file content has been truncated.\nStatus: Showing lines 1361-1460 of 2004 total lines.\nAction: To read more of the file, you can use the 'offset' and 'limit' parameters in a subsequent 'read_file' call. For example, to read the next section of the file, use offset: 1460.\n\n--- FILE CONTENT (truncated) ---\nclass InteractionAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('voter', 'date', 'type', 'description', 'volunteer')\n list_filter = ('voter__tenant', 'type', 'volunteer')\n search_fields = ('voter__first_name', 'voter__last_name', 'voter__voter_id', 'description', 'volunteer__first_name', 'volunteer__last_name')\n autocomplete_fields = ['voter', 'volunteer']\n change_list_template = 'admin/interaction_change_list.html'\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='interaction-download-errors'),\n path('import-interactions/', self.admin_site.admin_view(self.import_interactions), name='import-interactions'),\n ]\n return my_urls + urls\n\n def import_interactions(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n mapping = {}\n for field_name, _ in INTERACTION_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n total_count = 0\n create_count = 0\n update_count = 0\n preview_data = []\n for row in reader:\n total_count += 1\n voter_id = row.get(mapping.get('voter_id'))\n volunteer_email = row.get(mapping.get('volunteer_email'))\n\n exists = False\n if voter_id:\n exists = Voter.objects.filter(tenant=tenant, voter_id=voter_id).exists()\n\n if exists:\n update_count += 1\n action = 'update'\n else:\n create_count += 1\n action = 'create'\n\n if len(preview_data) < 10:\n preview_data.append({\n 'action': action,\n 'identifier': f\"Voter ID: {voter_id}\",\n 'details': f\"Type: {row.get(mapping.get('type', '')) or ''}, Volunteer: {volunteer_email or ''}\"\n })\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Import Preview\",\n 'total_count': total_count,\n 'create_count': create_count,\n 'update_count': update_count,\n 'preview_data': preview_data,\n 'mapping': mapping,\n 'file_path': file_path,\n 'tenant_id': tenant_id,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n\n elif \"_import\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in INTERACTION_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n count = 0\n errors = 0\n failed_rows = []\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n for row in reader:\n try:\n voter_id = row.get(mapping.get('voter_id'))\n volunteer_email = row.get(mapping.get('volunteer_email'))\n date_str = row.get(mapping.get('date'))\n type_name = row.get(mapping.get('type'))\n\n if voter_id: # Only strip if voter_id is not None"
/home/ubuntu/.gemini/tmp/ec2c6b8cb2ac3d6ef62b888a61e62d5ab1e71e46b3c4963f2a040a3c33c7038b/chats/session-2026-02-13T02-25-4951d6db.json: "output": "\nIMPORTANT: The file content has been truncated.\nStatus: Showing lines 1-2000 of 2003 total lines.\nAction: To read more of the file, you can use the 'offset' and 'limit' parameters in a subsequent 'read_file' call. For example, to read the next section of the file, use offset: 2000.\n\n--- FILE CONTENT (truncated) ---\nfrom decimal import Decimal\nfrom datetime import datetime, date\nfrom django.db import transaction\nfrom django.http import HttpResponse\nfrom django.utils.safestring import mark_safe\nimport csv\nimport io\nimport logging\nimport tempfile\nimport os\nfrom django.contrib import admin, messages\nfrom django.urls import path, reverse\nfrom django.shortcuts import render, redirect\nfrom django.template.response import TemplateResponse\nfrom .models import (\n format_phone_number,\n Tenant, TenantUserRole, InteractionType, DonationMethod, ElectionType, EventType, Voter,\n VotingRecord, Event, EventParticipation, Donation, Interaction, VoterLikelihood, CampaignSettings,\n Interest, Volunteer, VolunteerEvent, ParticipationStatus, VolunteerRole\n)\nfrom .forms import (\n VoterImportForm, EventImportForm, EventParticipationImportForm,\n DonationImportForm, InteractionImportForm, VoterLikelihoodImportForm,\n VolunteerImportForm, VotingRecordImportForm\n)\n\nlogger = logging.getLogger(__name__)\n\nVOTER_MAPPABLE_FIELDS = [\n ('voter_id', 'Voter ID'),\n ('first_name', 'First Name'),\n ('last_name', 'Last Name'),\n ('nickname', 'Nickname'),\n ('birthdate', 'Birthdate'),\n ('address_street', 'Street Address'),\n ('city', 'City'),\n ('state', 'State'),\n ('prior_state', 'Prior State'),\n ('zip_code', 'Zip Code'),\n ('county', 'County'),\n ('neighborhood', 'Neighborhood'),\n ('phone', 'Phone'),\n ('notes', 'Notes'),\n ('phone_type', 'Phone Type'),\n ('email', 'Email'),\n ('district', 'District'),\n ('precinct', 'Precinct'),\n ('registration_date', 'Registration Date'),\n ('is_targeted', 'Is Targeted'),\n ('candidate_support', 'Candidate Support'),\n ('yard_sign', 'Yard Sign'),\n ('window_sticker', 'Window Sticker'),\n ('latitude', 'Latitude'),\n ('longitude', 'Longitude'),\n ('secondary_phone', 'Secondary Phone'),\n ('secondary_phone_type', 'Secondary Phone Type'),\n ('door_visit', 'Door Visit'),\n]\n\nEVENT_MAPPABLE_FIELDS = [\n ('name', 'Name'),\n ('date', 'Date'),\n ('start_time', 'Start Time'),\n ('end_time', 'End Time'),\n ('event_type', 'Event Type (Name)'),\n ('description', 'Description'),\n ('location_name', 'Location Name'),\n ('address', 'Address'),\n ('city', 'City'),\n ('state', 'State'),\n ('zip_code', 'Zip Code'),\n ('latitude', 'Latitude'),\n ('longitude', 'Longitude'),\n]\n\nEVENT_PARTICIPATION_MAPPABLE_FIELDS = [\n ('voter_id', 'Voter ID'),\n ('first_name', 'First Name'),\n ('last_name', 'Last Name'),\n ('event_name', 'Event Name'),\n ('participation_status', 'Participation Status'),\n]\n\nDONATION_MAPPABLE_FIELDS = [\n ('voter_id', 'Voter ID'),\n ('date', 'Date'),\n ('amount', 'Amount'),\n ('method', 'Donation Method (Name)'),\n]\n\nINTERACTION_MAPPABLE_FIELDS = [\n ('voter_id', 'Voter ID'),\n ('volunteer_email', 'Volunteer Email'),\n ('date', 'Date'),\n ('type', 'Interaction Type (Name)'),\n ('description', 'Description'),\n ('notes', 'Notes'),\n]\n\n\nVOLUNTEER_MAPPABLE_FIELDS = [\n ('first_name', 'First Name'),\n ('last_name', 'Last Name'),\n ('email', 'Email'),\n ('phone', 'Phone'),\n ('notes', 'Notes'),\n]\n\nVOTER_LIKELIHOOD_MAPPABLE_FIELDS = [\n ('voter_id', 'Voter ID'),\n ('election_type', 'Election Type (Name)'),\n ('likelihood', 'Likelihood'),\n]\n\nVOTING_RECORD_MAPPABLE_FIELDS = [\n ('voter_id', 'Voter ID'),\n ('election_date', 'Election Date'),\n ('election_description', 'Election Description'),\n ('primary_party', 'Primary Party'),\n]\n\nclass BaseImportAdminMixin:\n def download_errors(self, request):\n logger.info(f\"download_errors called for {self.model._meta.model_name}\")\n session_key = f\"{self.model._meta.model_name}_import_errors\"\n failed_rows = request.session.get(session_key, [])\n if not failed_rows:\n self.message_user(request, \"No error log found in session.\", level=messages.WARNING)\n return redirect(\"../\")\n\n response = HttpResponse(content_type=\"text/csv\")\n response[\"Content-Disposition\"] = f\"attachment; filename={self.model._meta.model_name}_import_errors.csv\"\n\n if failed_rows:\n all_keys = set()\n for r in failed_rows:\n all_keys.update(r.keys())\n\n writer = csv.DictWriter(response, fieldnames=sorted(list(all_keys)))\n writer.writeheader()\n writer.writerows(failed_rows)\n\n return response\n\n def chunk_reader(self, reader, size):\n chunk = []\n for row in reader:\n chunk.append(row)\n if len(chunk) == size:\n yield chunk\n chunk = []\n if chunk:\n yield chunk\nclass TenantUserRoleInline(admin.TabularInline):\n model = TenantUserRole\n extra = 1\n\nclass CampaignSettingsInline(admin.StackedInline):\n model = CampaignSettings\n can_delete = False\n\n@admin.register(Tenant)\nclass TenantAdmin(admin.ModelAdmin):\n list_display = ('name', 'created_at')\n search_fields = ('name',)\n inlines = [TenantUserRoleInline, CampaignSettingsInline]\n\n@admin.register(TenantUserRole)\nclass TenantUserRoleAdmin(admin.ModelAdmin):\n list_display = ('user', 'tenant', 'role')\n list_filter = ('tenant', 'role')\n search_fields = ('user__username', 'tenant__name')\n\n@admin.register(InteractionType)\nclass InteractionTypeAdmin(admin.ModelAdmin):\n list_display = ('name', 'tenant', 'is_active')\n list_filter = ('tenant', 'is_active')\n search_fields = ('name',)\n\n@admin.register(DonationMethod)\nclass DonationMethodAdmin(admin.ModelAdmin):\n list_display = ('name', 'tenant', 'is_active')\n list_filter = ('tenant', 'is_active')\n search_fields = ('name',)\n\n@admin.register(VolunteerRole)\nclass VolunteerRoleAdmin(admin.ModelAdmin):\n list_display = (\"name\", \"tenant\", \"is_active\")\n list_filter = (\"tenant\", \"is_active\")\n search_fields = (\"name\",)\n\n@admin.register(ElectionType)\nclass ElectionTypeAdmin(admin.ModelAdmin):\n list_display = ('name', 'tenant', 'is_active')\n list_filter = ('tenant', 'is_active')\n search_fields = ('name',)\n\n@admin.register(EventType)\nclass EventTypeAdmin(admin.ModelAdmin):\n list_display = ('name', 'tenant', 'is_active', 'default_volunteer_role')\n list_filter = ('tenant', 'is_active')\n search_fields = ('name',)\n filter_horizontal = ('available_roles',)\n\n\n@admin.register(ParticipationStatus)\nclass ParticipationStatusAdmin(admin.ModelAdmin):\n list_display = ('name', 'tenant', 'is_active')\n list_filter = ('tenant', 'is_active')\n search_fields = ('name',)\n change_list_template = 'admin/participationstatus_change_list.html'\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n@admin.register(Interest)\nclass InterestAdmin(admin.ModelAdmin):\n list_display = ('name', 'tenant')\n list_filter = ('tenant',)\n fields = ('tenant', 'name')\n search_fields = ('name',)\n\nclass VotingRecordInline(admin.TabularInline):\n model = VotingRecord\n extra = 1\n\nclass DonationInline(admin.TabularInline):\n model = Donation\n extra = 1\n\nclass InteractionInline(admin.TabularInline):\n model = Interaction\n extra = 1\n autocomplete_fields = ['voter', 'type', 'volunteer']\n\nclass VoterLikelihoodInline(admin.TabularInline):\n model = VoterLikelihood\n extra = 1\n\nclass VolunteerEventInline(admin.TabularInline):\n model = VolunteerEvent\n extra = 1\n\n@admin.register(Voter)\nclass VoterAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('first_name', 'last_name', 'nickname', 'voter_id', 'tenant', 'district', 'candidate_support', 'is_targeted', 'city', 'state', 'prior_state')\n list_filter = ('tenant', 'candidate_support', 'is_targeted', 'phone_type', 'yard_sign', 'district', 'city', 'state', 'prior_state')\n search_fields = ('first_name', 'last_name', 'nickname', 'voter_id', 'address', 'city', 'state', 'prior_state', 'zip_code', 'county')\n inlines = [VotingRecordInline, DonationInline, InteractionInline, VoterLikelihoodInline]\n readonly_fields = ('address',)\n change_list_template = \"admin/voter_change_list.html\"\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context[\"tenants\"] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='voter-download-errors'),\n path('import-voters/', self.admin_site.admin_view(self.import_voters), name='import-voters'),\n ]\n return my_urls + urls\n\n\n def import_voters(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get(\"file_path\")\n tenant_id = request.POST.get(\"tenant\")\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in VOTER_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f\"map_{field_name}\")\n\n try:\n with open(file_path, \"r\", encoding=\"utf-8-sig\") as f:\n # Optimization: Skip full count for very large files in preview if needed,\n # but here we'll keep it for accuracy unless it's a known bottleneck.\n # For now, let's just do a fast line count.\n total_count = sum(1 for line in f) - 1\n f.seek(0)\n reader = csv.DictReader(f)\n preview_rows = []\n voter_ids_for_preview = []\n for i, row in enumerate(reader):\n if i < 10:\n preview_rows.append(row)\n v_id = row.get(mapping.get(\"voter_id\"))\n if v_id:\n voter_ids_for_preview.append(v_id.strip())\n else:\n break\n\n existing_preview_ids = set(Voter.objects.filter(tenant=tenant, voter_id__in=voter_ids_for_preview).values_list(\"voter_id\", flat=True))\n\n create_count = 0\n update_count = 0\n\n for row in preview_rows:\n voter_id_val = row.get(mapping.get(\"voter_id\"))\n if voter_id_val and voter_id_val.strip() in existing_preview_ids:\n update_count += 1\n else:\n create_count += 1\n\n context = self.admin_site.each_context(request)\n context.update({\n \"title\": \"Import Preview\",\n \"total_count\": total_count,\n \"create_count\": create_count,\n \"update_count\": update_count,\n \"preview_data\": preview_rows, # This should be improved to show actual changes\n \"mapping\": mapping,\n \"file_path\": file_path,\n \"tenant_id\": tenant_id,\n \"action_url\": request.path,\n \"opts\": self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n elif \"_import\" in request.POST:\n file_path = request.POST.get(\"file_path\")\n tenant_id = request.POST.get(\"tenant\")\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in VOTER_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f\"map_{field_name}\")\n\n try:\n created_count = 0\n updated_count = 0\n skipped_no_change = 0\n skipped_no_id = 0\n errors = 0\n failed_rows = []\n total_processed = 0\n\n # Temporary storage for error rows to avoid holding large file in memory\n temp_error_file = None\n temp_error_file_path = None\n\n # Process in chunks to reduce memory usage for very large files\n with open(file_path, \"r\", encoding=\"utf-8-sig\") as f_read:\n reader = csv.DictReader(f_read)\n for i, row in enumerate(reader):\n total_processed += 1\n try:\n raw_voter_id = row.get(mapping.get(\"voter_id\"))\n voter_id = raw_voter_id.strip() if raw_voter_id else None\n\n if not voter_id:\n # Enhanced error message to guide the user\n mapped_column_name = mapping.get(\"voter_id\", \"N/A\")\n error_detail = f\"Raw value: '{raw_voter_id}'. \" if raw_voter_id is not None else \"Value was None.\"\n row[\"Import Error\"] = f\"Voter ID is required. Please check if the '{mapped_column_name}' column is correctly mapped and contains values for all rows. {error_detail}\"\n failed_rows.append(row)\n skipped_no_id += 1\n errors += 1\n continue\n\n defaults = {}\n # Map other fields dynamically\n for field_name, _ in VOTER_MAPPABLE_FIELDS:\n csv_column = mapping.get(field_name)\n if csv_column and csv_column in row:\n field_value = row[csv_column].strip()\n if field_name == \"birthdate\" or field_name == \"registration_date\":\n # Handle date conversions\n if field_value:\n try:\n # Attempt to parse common date formats\n if '/' in field_value:\n # Try MM/DD/YYYY or DD/MM/YYYY\n if len(field_value.split('/')[2]) == 2: # YY format\n dt = datetime.strptime(field_value, '%m/%d/%y').date() if len(field_value.split('/')[0]) < 3 else datetime.strptime(field_value, '%d/%m/%y').date() # noqa\n else:\n dt = datetime.strptime(field_value, '%m/%d/%Y').date() if len(field_value.split('/')[0]) < 3 else datetime.strptime(field_value, '%d/%m/%Y').date() # noqa\n elif '-' in field_value:\n # Try YYYY-MM-DD or DD-MM-YYYY or MM-DD-YYYY\n if len(field_value.split('-')[0]) == 4: # YYYY format\n dt = datetime.strptime(field_value, '%Y-%m-%d').date()\n elif len(field_value.split('-')[2]) == 4: # YYYY format\n dt = datetime.strptime(field_value, '%m-%d-%Y').date() if len(field_value.split('-')[0]) < 3 else datetime.strptime(field_value, '%d-%m-%Y').date() # noqa\n else:\n # Default to MM-DD-YY\n dt = datetime.strptime(field_value, '%m-%d-%y').date()\n else:\n dt = None\n\n if dt:\n defaults[field_name] = dt\n else:\n logger.warning(f\"Could not parse date '{field_value}' for field {field_name}. Skipping.\")\n except ValueError as ve:\n logger.warning(f\"Date parsing error for '{field_value}' in field {field_name}: {ve}\")\n except Exception as ex:\n logger.error(f\"Unexpected error parsing date '{field_value}' for field {field_name}: {ex}\")\n elif field_name == \"is_targeted\" or field_name == \"yard_sign\" or field_name == \"window_sticker\" or field_name == \"door_visit\":\n # Handle boolean fields\n if field_value.lower() == 'true' or field_value == '1':\n defaults[field_name] = True\n elif field_value.lower() == 'false' or field_value == '0':\n defaults[field_name] = False\n else:\n defaults[field_name] = None # Or sensible default/error\n elif field_name == \"phone\":\n defaults[field_name] = format_phone_number(field_value)\n elif field_name == \"email\":\n defaults[field_name] = field_value.lower() # Store emails as lowercase\n elif field_name == \"candidate_support\":\n if field_value in [choice[0] for choice in Voter.CANDIDATE_SUPPORT_CHOICES]:\n defaults[field_name] = field_value\n else:\n logger.warning(f\"Invalid candidate_support value: {field_value}. Skipping.\")\n elif field_name == \"phone_type\":\n if field_value in [choice[0] for choice in Voter.PHONE_TYPE_CHOICES]:\n defaults[field_name] = field_value\n else:\n logger.warning(f\"Invalid phone_type value: {field_value}. Skipping.\")\n elif field_name == \"secondary_phone_type\":\n if field_value in [choice[0] for choice in Voter.PHONE_TYPE_CHOICES]:\n defaults[field_name] = field_value\n else:\n logger.warning(f\"Invalid secondary_phone_type value: {field_value}. Skipping.\")\n elif field_name == \"state\" or field_name == \"prior_state\":\n # Ensure state is uppercase and valid length\n if field_value and len(field_value) <= 2:\n defaults[field_name] = field_value.upper()\n else:\n logger.warning(f\"Invalid state value: {field_value}. Skipping.\")\n else:\n defaults[field_name] = field_value\n\n # Try to get voter. If not found, create new. Update if found.\n voter, created = Voter.objects.update_or_create(\n tenant=tenant,\n voter_id=voter_id,\n defaults=defaults\n )\n if created:\n created_count += 1\n else:\n updated_count += 1\n\n # Special handling for interests - assuming a comma-separated list in CSV\n if 'interests' in mapping and row.get(mapping['interests']):\n interest_names = [name.strip() for name in row[mapping['interests']].split(',') if name.strip()]\n for interest_name in interest_names:\n interest, _ = Interest.objects.get_or_create(tenant=tenant, name=interest_name)\n voter.interests.add(interest)\n\n if (i + 1) % 100 == 0:\n print(f\"DEBUG: Voter import progress: {total_processed} processed. {created_count} created. {updated_count} updated.\")\n\n except Exception as e:\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n logger.error(f\"Error importing row: {row}. Error: {e}\")\n\n # Clean up the temporary file\n if os.path.exists(file_path):\n os.remove(file_path)\n\n if temp_error_file_path and os.path.exists(temp_error_file_path):\n os.remove(temp_error_file_path)\n\n self.message_user(request, f\"Import complete: {created_count + updated_count} voters created/updated. ({created_count} new, {updated_count} updated, {skipped_no_change} skipped with no changes, {skipped_no_id} skipped missing ID, {errors} errors)\")\n # Store failed rows in session for download, limit to avoid session overflow\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n\n if errors > 0:\n error_url = reverse(\"admin:voter-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = VoterImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n \"title\": \"Map Voter Fields\",\n \"headers\": headers,\n \"model_fields\": VOTER_MAPPABLE_FIELDS,\n \"tenant_id\": tenant.id,\n \"file_path\": file_path,\n \"action_url\": request.path,\n \"opts\": self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = VoterImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Voters\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)\n\n\n@admin.register(Event)\nclass EventAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('name', 'date', 'event_type', 'tenant', 'location_name', 'address', 'city', 'state', 'zip_code')\n list_filter = ('tenant', 'event_type')\n search_fields = ('name', 'location_name', 'address', 'city', 'state', 'zip_code')\n inlines = [VolunteerEventInline]\n change_list_template = \"admin/event_change_list.html\"\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='event-download-errors'),\n path('import-events/', self.admin_site.admin_view(self.import_events), name='import-events'),\n ]\n return my_urls + urls\n\n def import_events(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n mapping = {}\n for field_name, _ in EVENT_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n total_count = 0\n create_count = 0\n update_count = 0\n preview_data = []\n for row in reader:\n total_count += 1\n event_name = row.get(mapping.get('name'))\n event_date = row.get(mapping.get('date'))\n\n exists = False\n if event_name and event_date:\n try:\n # Assuming name and date uniquely identify an event\n # This might need refinement based on actual data uniqueness requirements\n if '/' in event_date:\n dt = datetime.strptime(event_date, '%m/%d/%Y').date()\n elif '-' in event_date:\n dt = datetime.strptime(event_date, '%Y-%m-%d').date()\n else:\n dt = None\n\n if dt:\n exists = Event.objects.filter(tenant=tenant, name=event_name, date=dt).exists()\n\n except ValueError:\n # Handle cases where date parsing fails\n pass\n\n if exists:\n update_count += 1\n action = 'update'\n else:\n create_count += 1\n action = 'create'\n\n if len(preview_data) < 10:\n preview_data.append({\n 'action': action,\n 'identifier': f\"Event: {event_name} (Date: {event_date})\",\n 'details': f\"Location: {row.get(mapping.get('location_name', '')) or ''}\"\n })\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Import Preview\",\n 'total_count': total_count,\n 'create_count': create_count,\n 'update_count': update_count,\n 'preview_data': preview_data,\n 'mapping': mapping,\n 'file_path': file_path,\n 'tenant_id': tenant_id,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n\n elif \"_import\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in EVENT_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n count = 0\n errors = 0\n failed_rows = []\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n for row in reader:\n try:\n event_name = row.get(mapping.get('name'))\n event_date = row.get(mapping.get('date'))\n event_type_name = row.get(mapping.get('event_type'))\n\n if not event_name or not event_date or not event_type_name:\n row[\"Import Error\"] = \"Missing event name, date, or type\"\n failed_rows.append(row)\n errors += 1\n continue\n\n # Date parsing for event_date\n try:\n if '/' in event_date:\n parsed_date = datetime.strptime(event_date, '%m/%d/%Y').date()\n elif '-' in event_date:\n parsed_date = datetime.strptime(event_date, '%Y-%m-%d').date()\n else:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n except ValueError:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n\n event_type_obj, _ = EventType.objects.get_or_create(tenant=tenant, name=event_type_name)\n\n defaults = {\n 'date': parsed_date,\n 'event_type': event_type_obj,\n 'description': row.get(mapping.get('description')) or '',\n 'location_name': row.get(mapping.get('location_name')) or '',\n 'address': row.get(mapping.get('address')) or '',\n 'city': row.get(mapping.get('city')) or '',\n 'state': row.get(mapping.get('state')) or '',\n 'zip_code': row.get(mapping.get('zip_code')) or '',\n 'latitude': row.get(mapping.get('latitude')) or None,\n 'longitude': row.get(mapping.get('longitude')) or None,\n }\n\n # Handle start_time and end_time\n start_time_str = row.get(mapping.get('start_time'))\n if start_time_str:\n try:\n defaults['start_time'] = datetime.strptime(start_time_str, '%H:%M').time()\n except ValueError:\n logger.warning(f\"Invalid start_time format: {start_time_str}. Skipping.\")\n end_time_str = row.get(mapping.get('end_time'))\n if end_time_str:\n try:\n defaults['end_time'] = datetime.strptime(end_time_str, '%H:%M').time()\n except ValueError:\n logger.warning(f\"Invalid end_time format: {end_time_str}. Skipping.\")\n\n Event.objects.update_or_create(\n tenant=tenant,\n name=event_name,\n defaults=defaults\n )\n count += 1\n except Exception as e:\n logger.error(f\"Error importing: {e}\")\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n\n if os.path.exists(file_path):\n os.remove(file_path)\n self.message_user(request, f\"Successfully imported {count} events.\")\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n if errors > 0:\n error_url = reverse(\"admin:event-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = EventImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Map Event Fields\",\n 'headers': headers,\n 'model_fields': EVENT_MAPPABLE_FIELDS,\n 'tenant_id': tenant.id,\n 'file_path': file_path,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = EventImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Events\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)\n\n\n@admin.register(Volunteer)\nclass VolunteerAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('first_name', 'last_name', 'email', 'phone', 'tenant')\n list_filter = ('tenant',)\n search_fields = ('first_name', 'last_name', 'email', 'phone')\n change_list_template = \"admin/volunteer_change_list.html\"\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='volunteer-download-errors'),\n path('import-volunteers/', self.admin_site.admin_view(self.import_volunteers), name='import-volunteers'),\n ]\n return my_urls + urls\n\n def import_volunteers(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n mapping = {}\n for field_name, _ in VOLUNTEER_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n total_count = 0\n create_count = 0\n update_count = 0\n preview_data = []\n for row in reader:\n total_count += 1\n email = row.get(mapping.get('email'))\n\n exists = False\n if email:\n exists = Volunteer.objects.filter(tenant=tenant, email=email).exists()\n\n if exists:\n update_count += 1\n action = 'update'\n else:\n create_count += 1\n action = 'create'\n\n if len(preview_data) < 10:\n preview_data.append({\n 'action': action,\n 'identifier': f\"Volunteer: {email}\",\n 'details': f\"Name: {row.get(mapping.get('first_name', '')) or ''} {row.get(mapping.get('last_name', '')) or ''}\"\n })\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Import Preview\",\n 'total_count': total_count,\n 'create_count': create_count,\n 'update_count': update_count,\n 'preview_data': preview_data,\n 'mapping': mapping,\n 'file_path': file_path,\n 'tenant_id': tenant_id,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n\n elif \"_import\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in VOLUNTEER_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n count = 0\n errors = 0\n failed_rows = []\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n for row in reader:\n try:\n email = row.get(mapping.get('email'))\n if not email:\n row[\"Import Error\"] = \"Missing email\"\n failed_rows.append(row)\n errors += 1\n continue\n\n defaults = {\n 'first_name': row.get(mapping.get('first_name')) or '',\n 'last_name': row.get(mapping.get('last_name')) or '',\n 'phone': format_phone_number(row.get(mapping.get('phone')) or ''),\n 'notes': row.get(mapping.get('notes')) or '',\n }\n\n Volunteer.objects.update_or_create(\n tenant=tenant,\n email=email,\n defaults=defaults\n )\n count += 1\n except Exception as e:\n logger.error(f\"Error importing volunteer: {e}\")\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n\n if os.path.exists(file_path):\n os.remove(file_path)\n self.message_user(request, f\"Successfully imported {count} volunteers.\")\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n if errors > 0:\n error_url = reverse(\"admin:volunteer-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = VolunteerImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Map Volunteer Fields\",\n 'headers': headers,\n 'model_fields': VOLUNTEER_MAPPABLE_FIELDS,\n 'tenant_id': tenant.id,\n 'file_path': file_path,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = VolunteerImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Volunteers\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)\n\n\n@admin.register(EventParticipation)\nclass EventParticipationAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('event', 'voter', 'participation_status')\n list_filter = ('event', 'participation_status', 'voter__tenant')\n search_fields = ('event__name', 'voter__first_name', 'voter__last_name', 'voter__voter_id')\n change_list_template = 'admin/eventparticipation_change_list.html'\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_list(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='eventparticipation-download-errors'),\n path('import-event-participations/', self.admin_site.admin_view(self.import_event_participations), name='import-event-participations'),\n ]\n return my_urls + urls\n\n def import_event_participations(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n mapping = {}\n for field_name, _ in EVENT_PARTICIPATION_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n total_count = 0\n create_count = 0\n update_count = 0\n preview_data = []\n for row in reader:\n total_count += 1\n voter_id = row.get(mapping.get('voter_id'))\n\n # Extract first_name and last_name from CSV based on mapping\n csv_first_name = row.get(mapping.get('first_name'), '')\n csv_last_name = row.get(mapping.get('last_name'), '')\n csv_full_name = f\"{csv_first_name} {csv_last_name}\".strip()\n\n exists = False\n voter_full_name = \"N/A\" # Initialize voter_full_name\n if voter_id:\n try:\n voter = Voter.objects.get(tenant=tenant, voter_id=voter_id)\n voter_full_name = f\"{voter.first_name} {voter.last_name}\" # Get voter's full name\n if event_name:\n exists = EventParticipation.objects.filter(voter=voter, event__name=event_name).exists()\n except Voter.DoesNotExist:\n pass\n\n if exists:\n update_count += 1\n action = 'update'\n else:\n create_count += 1\n action = 'create'\n\n if len(preview_data) < 10:\n preview_data.append({\n 'action': action,\n 'csv_full_name': csv_full_name, # Add CSV name\n 'identifier': f\"Voter: {voter_full_name} (ID: {voter_id})\" if voter_id else \"N/A\", # Include full name\n 'details': f\"Participation: {row.get(mapping.get('participation_status', '')) or ''}\"\n })\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Import Preview\",\n 'total_count': total_count,\n 'create_count': create_count,\n 'update_count': update_count,\n 'preview_data': preview_data,\n 'mapping': mapping,\n 'file_path': file_path,\n 'tenant_id': tenant_id,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n\n elif \"_import\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in EVENT_PARTICIPATION_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n count = 0\n errors = 0\n failed_rows = []\n for row in reader:\n try:\n voter_id = row.get(mapping.get('voter_id')) if mapping.get('voter_id') else None\n participation_status_val = row.get(mapping.get('participation_status')) if mapping.get('participation_status') else None\n\n if voter_id: # Only strip if voter_id is not None\n voter_id = voter_id.strip()\n\n if not voter_id:\n row[\"Import Error\"] = \"Missing voter ID\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n voter = Voter.objects.get(tenant=tenant, voter_id=voter_id)\n except Voter.DoesNotExist:\n error_msg = f\"Voter with ID {voter_id} not found\"\n logger.error(error_msg)\n row[\"Import Error\"] = error_msg\n failed_rows.append(row)\n errors += 1\n continue\n\n event = None\n event_name = row.get(mapping.get('event_name')) if mapping.get('event_name') else None\n if event_name:\n try:\n event = Event.objects.get(tenant=tenant, name=event_name)\n except Event.DoesNotExist:\n pass\n\n if not event:\n error_msg = \"Event not found (check Event Name)\"\n logger.error(error_msg)\n row[\"Import Error\"] = error_msg\n failed_rows.append(row)\n errors += 1\n continue\n\n defaults = {}\n if participation_status_val and participation_status_val.strip():\n status_obj, _ = ParticipationStatus.objects.get_or_create(tenant=tenant, name=participation_status_val.strip())\n defaults['participation_status'] = status_obj\n else:\n # Default to 'Invited' if not specified\n status_obj, _ = ParticipationStatus.objects.get_or_create(tenant=tenant, name='Invited')\n defaults['participation_status'] = status_obj\n EventParticipation.objects.update_or_create(\n event=event,\n voter=voter,\n defaults=defaults\n )\n count += 1\n except Exception as e:\n logger.error(f\"Error importing: {e}\")\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n\n if os.path.exists(file_path):\n os.remove(file_path)\n self.message_user(request, f\"Successfully imported {count} participations.\")\n # Optimization: Limit error log size in session to avoid overflow\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n logger.info(f\"Stored {len(failed_rows)} failed rows in session for {self.model._meta.model_name}\")\n if errors > 0:\n error_url = reverse(\"admin:eventparticipation-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = EventParticipationImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Map Participation Fields\",\n 'headers': headers,\n 'model_fields': EVENT_PARTICIPATION_MAPPABLE_FIELDS,\n 'tenant_id': tenant.id,\n 'file_path': file_path,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = EventParticipationImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Participations\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)\n\n@admin.register(Donation)\nclass DonationAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('voter', 'date', 'amount', 'method')\n list_filter = ('voter__tenant', 'method')\n search_fields = ('voter__first_name', 'voter__last_name', 'voter__voter_id', 'method__name')\n change_list_template = 'admin/donation_change_list.html'\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='donation-download-errors'),\n path('import-donations/', self.admin_site.admin_view(self.import_donations), name='import-donations'),\n ]\n return my_urls + urls\n\n def import_donations(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n mapping = {}\n for field_name, _ in DONATION_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n total_count = 0\n create_count = 0\n update_count = 0\n preview_data = []\n for row in reader:\n total_count += 1\n voter_id = row.get(mapping.get('voter_id'))\n\n exists = False\n if voter_id:\n exists = Voter.objects.filter(tenant=tenant, voter_id=voter_id).exists()\n\n if exists:\n update_count += 1\n action = 'update'\n else:\n create_count += 1\n action = 'create'\n\n if len(preview_data) < 10:\n preview_data.append({\n 'action': action,\n 'identifier': f\"Voter ID: {voter_id}\",\n 'details': f\"Amount: {row.get(mapping.get('amount', '')) or ''}, Method: {row.get(mapping.get('method', '')) or ''}\"\n })\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Import Preview\",\n 'total_count': total_count,\n 'create_count': create_count,\n 'update_count': update_count,\n 'preview_data': preview_data,\n 'mapping': mapping,\n 'file_path': file_path,\n 'tenant_id': tenant_id,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n\n elif \"_import\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in DONATION_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n count = 0\n errors = 0\n failed_rows = []\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n for row in reader:\n try:\n voter_id = row.get(mapping.get('voter_id'))\n date_str = row.get(mapping.get('date'))\n amount_str = row.get(mapping.get('amount'))\n method_name = row.get(mapping.get('method'))\n\n if voter_id: # Only strip if voter_id is not None\n voter_id = voter_id.strip()\n\n if not voter_id:\n row[\"Import Error\"] = \"Missing voter ID\"\n failed_rows.append(row)\n errors += 1\n continue\n\n if not date_str or not amount_str:\n row[\"Import Error\"] = \"Missing date or amount\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n voter = Voter.objects.get(tenant=tenant, voter_id=voter_id)\n except Voter.DoesNotExist:\n row[\"Import Error\"] = f\"Voter {voter_id} not found\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n if '/' in date_str:\n parsed_date = datetime.strptime(date_str, '%m/%d/%Y').date()\n elif '-' in date_str:\n parsed_date = datetime.strptime(date_str, '%Y-%m-%d').date()\n else:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n except ValueError:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n amount = Decimal(amount_str)\n except InvalidOperation:\n row[\"Import Error\"] = \"Invalid amount format\"\n failed_rows.append(row)\n errors += 1\n continue\n\n donation_method, _ = DonationMethod.objects.get_or_create(tenant=tenant, name=method_name)\n\n Donation.objects.create(\n voter=voter,\n date=parsed_date,\n amount=amount,\n method=donation_method\n )\n count += 1\n except Exception as e:\n logger.error(f\"Error importing: {e}\")\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n\n if os.path.exists(file_path):\n os.remove(file_path)\n self.message_user(request, f\"Successfully imported {count} donations.\")\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n if errors > 0:\n error_url = reverse(\"admin:donation-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = DonationImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Map Donation Fields\",\n 'headers': headers,\n 'model_fields': DONATION_MAPPABLE_FIELDS,\n 'tenant_id': tenant.id,\n 'file_path': file_path,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = DonationImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Donations\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)\n\n@admin.register(Interaction)\nclass InteractionAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('voter', 'date', 'type', 'description', 'volunteer')\n list_filter = ('voter__tenant', 'type', 'volunteer')\n search_fields = ('voter__first_name', 'voter__last_name', 'voter__voter_id', 'description', 'volunteer__first_name', 'volunteer__last_name')\n autocomplete_fields = ['voter', 'volunteer']\n change_list_template = 'admin/interaction_change_list.html'\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='interaction-download-errors'),\n path('import-interactions/', self.admin_site.admin_view(self.import_interactions), name='import-interactions'),\n ]\n return my_urls + urls\n\n def import_interactions(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n mapping = {}\n for field_name, _ in INTERACTION_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n total_count = 0\n create_count = 0\n update_count = 0\n preview_data = []\n for row in reader:\n total_count += 1\n voter_id = row.get(mapping.get('voter_id'))\n volunteer_email = row.get(mapping.get('volunteer_email'))\n\n exists = False\n if voter_id:\n exists = Voter.objects.filter(tenant=tenant, voter_id=voter_id).exists()\n\n if exists:\n update_count += 1\n action = 'update'\n else:\n create_count += 1\n action = 'create'\n\n if len(preview_data) < 10:\n preview_data.append({\n 'action': action,\n 'identifier': f\"Voter ID: {voter_id}\",\n 'details': f\"Type: {row.get(mapping.get('type', '')) or ''}, Volunteer: {volunteer_email or ''}\"\n })\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Import Preview\",\n 'total_count': total_count,\n 'create_count': create_count,\n 'update_count': update_count,\n 'preview_data': preview_data,\n 'mapping': mapping,\n 'file_path': file_path,\n 'tenant_id': tenant_id,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n\n elif \"_import\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in INTERACTION_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n count = 0\n errors = 0\n failed_rows = []\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n for row in reader:\n try:\n voter_id = row.get(mapping.get('voter_id'))\n volunteer_email = row.get(mapping.get('volunteer_email'))\n date_str = row.get(mapping.get('date'))\n type_name = row.get(mapping.get('type'))\n\n if voter_id: # Only strip if voter_id is not None\n voter_id = voter_id.strip()\n\n if not voter_id:\n row[\"Import Error\"] = \"Missing voter ID\"\n failed_rows.append(row)\n errors += 1\n continue\n\n if not date_str or not type_name:\n row[\"Import Error\"] = \"Missing date or description\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n voter = Voter.objects.get(tenant=tenant, voter_id=voter_id)\n except Voter.DoesNotExist:\n row[\"Import Error\"] = f\"Voter {voter_id} not found\"\n failed_rows.append(row)\n errors += 1\n continue\n\n volunteer = None\n if volunteer_email:\n try:\n volunteer = Volunteer.objects.get(tenant=tenant, email=volunteer_email)\n except Volunteer.DoesNotExist:\n pass # Volunteer is optional\n\n try:\n if '/' in date_str:\n parsed_date = datetime.strptime(date_str, '%m/%d/%Y').date()\n elif '-' in date_str:\n parsed_date = datetime.strptime(date_str, '%Y-%m-%d').date()\n else:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n except ValueError:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n\n interaction_type, _ = InteractionType.objects.get_or_create(tenant=tenant, name=type_name)\n\n Interaction.objects.create(\n voter=voter,\n volunteer=volunteer,\n date=parsed_date,\n type=interaction_type,\n description=row.get(mapping.get('description')) or '',\n notes=row.get(mapping.get('notes')) or ''\n )\n count += 1\n except Exception as e:\n logger.error(f\"Error importing: {e}\")\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n\n if os.path.exists(file_path):\n os.remove(file_path)\n self.message_user(request, f\"Successfully imported {count} interactions.\")\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n if errors > 0:\n error_url = reverse(\"admin:interaction-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = InteractionImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Map Interaction Fields\",\n 'headers': headers,\n 'model_fields': INTERACTION_MAPPABLE_FIELDS,\n 'tenant_id': tenant.id,\n 'file_path': file_path,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = InteractionImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Interactions\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)\n\n@admin.register(VoterLikelihood)\nclass VoterLikelihoodAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('voter', 'election_type', 'likelihood')\n list_filter = ('voter__tenant', 'election_type', 'likelihood')\n search_fields = ('voter__first_name', 'voter__last_name', 'voter__voter_id', 'election_type__name')\n change_list_template = 'admin/voterlikelihood_change_list.html'\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='voterlikelihood-download-errors'),\n path('import-likelihoods/', self.admin_site.admin_view(self.import_likelihoods), name='import-likelihoods'),\n ]\n return my_urls + urls\n\n def import_likelihoods(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n mapping = {}\n for field_name, _ in VOTER_LIKELIHOOD_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n total_count = 0\n create_count = 0\n update_count = 0\n preview_data = []\n for row in reader:\n total_count += 1\n voter_id = row.get(mapping.get('voter_id'))\n\n exists = False\n if voter_id:\n exists = Voter.objects.filter(tenant=tenant, voter_id=voter_id).exists()\n\n if exists:\n update_count += 1\n action = 'update'\n else:\n create_count += 1\n action = 'create'\n\n if len(preview_data) < 10:\n preview_data.append({\n 'action': action,\n 'identifier': f\"Voter ID: {voter_id}\",\n 'details': f\"Likelihood: {row.get(mapping.get('likelihood', '')) or ''}\"\n })\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Import Preview\",\n 'total_count': total_count,\n 'create_count': create_count,\n 'update_count': update_count,\n 'preview_data': preview_data,\n 'mapping': mapping,\n 'file_path': file_path,\n 'tenant_id': tenant_id,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n\n elif \"_import\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in VOTER_LIKELIHOOD_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n count = 0\n errors = 0\n failed_rows = []\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n for row in reader:\n try:\n voter_id = row.get(mapping.get('voter_id'))\n election_type_name = row.get(mapping.get('election_type'))\n likelihood_val = row.get(mapping.get('likelihood'))\n\n if voter_id: # Only strip if voter_id is not None\n voter_id = voter_id.strip()\n\n if not voter_id:\n row[\"Import Error\"] = \"Missing voter ID\"\n failed_rows.append(row)\n errors += 1\n continue\n\n if not election_type_name or not likelihood_val:\n row[\"Import Error\"] = \"Missing election type or likelihood\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n voter = Voter.objects.get(tenant=tenant, voter_id=voter_id)\n except Voter.DoesNotExist:\n row[\"Import Error\"] = f\"Voter {voter_id} not found\"\n failed_rows.append(row)\n errors += 1\n continue\n\n election_type, _ = ElectionType.objects.get_or_create(tenant=tenant, name=election_type_name)\n\n VoterLikelihood.objects.update_or_create(\n voter=voter,\n election_type=election_type,\n defaults={'likelihood': likelihood_val}\n )\n count += 1\n except Exception as e:\n print(f\"DEBUG: Likelihood import failed: {e}\")\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n\n if os.path.exists(file_path):\n os.remove(file_path)\n self.message_user(request, f\"Import complete: {count} likelihoods created/updated.\")\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n if errors > 0:\n error_url = reverse(\"admin:voterlikelihood-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = VoterLikelihoodImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Map Likelihood Fields\",\n 'headers': headers,\n 'model_fields': VOTER_LIKELIHOOD_MAPPABLE_FIELDS,\n 'tenant_id': tenant.id,\n 'file_path': file_path,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = VoterLikelihoodImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Likelihoods\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)\n\n\n@admin.register(VotingRecord)\nclass VotingRecordAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('voter', 'election_date', 'election_description', 'primary_party')\n list_filter = ('voter__tenant', 'primary_party')\n search_fields = ('voter__first_name', 'voter__last_name', 'voter__voter_id', 'election_description')\n change_list_template = 'admin/votingrecord_change_list.html'\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='votingrecord-download-errors'),\n path('import-voting-records/', self.admin_site.admin_view(self.import_voting_records), name='import-voting-records'),\n ]\n return my_urls + urls\n\n def import_voting_records(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n mapping = {}\n for field_name, _ in VOTING_RECORD_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n total_count = 0\n create_count = 0\n update_count = 0\n preview_data = []\n for row in reader:\n total_count += 1\n voter_id = row.get(mapping.get('voter_id'))\n election_date = row.get(mapping.get('election_date'))\n\n exists = False\n if voter_id and election_date:\n try:\n # Assuming voter_id and election_date uniquely identify a voting record\n # This might need refinement based on actual data uniqueness requirements\n if '/' in election_date:\n dt = datetime.strptime(election_date, '%m/%d/%Y').date()\n elif '-' in election_date:\n dt = datetime.strptime(election_date, '%Y-%m-%d').date()\n else:\n dt = None\n\n if dt:\n exists = VotingRecord.objects.filter(voter__tenant=tenant, voter__voter_id=voter_id, election_date=dt).exists()\n\n except ValueError:\n # Handle cases where date parsing fails\n pass\n\n if exists:\n update_count += 1\n action = 'update'\n else:\n create_count += 1\n action = 'create'\n\n if len(preview_data) < 10:\n preview_data.append({\n 'action': action,\n 'identifier': f\"Voter ID: {voter_id} (Election: {election_date})\",\n 'details': f\"Party: {row.get(mapping.get('primary_party', '')) or ''}\"\n })\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Import Preview\",\n 'total_count': total_count,\n 'create_count': create_count,\n 'update_count': update_count,\n 'preview_data': preview_data,\n 'mapping': mapping,\n 'file_path': file_path,\n 'tenant_id': tenant_id,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n\n elif \"_import\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in VOTING_RECORD_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n count = 0\n errors = 0\n failed_rows = []\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n for row in reader:\n try:\n voter_id = row.get(mapping.get('voter_id'))\n election_date_str = row.get(mapping.get('election_date'))\n election_description = row.get(mapping.get('election_description'))\n primary_party = row.get(mapping.get('primary_party'))\n\n if voter_id: # Only strip if voter_id is not None\n voter_id = voter_id.strip()\n\n if not voter_id:\n row[\"Import Error\"] = \"Missing voter ID\"\n failed_rows.append(row)\n errors += 1\n continue\n\n if not election_date_str or not election_description:\n row[\"Import Error\"] = \"Missing election date or description\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n voter = Voter.objects.get(tenant=tenant, voter_id=voter_id)\n except Voter.DoesNotExist:\n row[\"Import Error\"] = f\"Voter {voter_id} not found\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n if '/' in election_date_str:\n parsed_election_date = datetime.strptime(election_date_str, '%m/%d/%Y').date()\n elif '-' in election_date_str:\n parsed_election_date = datetime.strptime(election_date_str, '%Y-%m-%d').date()\n else:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n except ValueError:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n\n VotingRecord.objects.update_or_create(\n voter=voter,\n election_date=parsed_election_date,\n defaults={\n 'election_description': election_description,\n 'primary_party': primary_party or ''\n }\n )\n count += 1\n except Exception as e:\n logger.error(f\"Error importing: {e}\")\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n\n if os.path.exists(file_path):\n os.remove(file_path)\n self.message_user(request, f\"Successfully imported {count} voting records.\")\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n if errors > 0:\n error_url = reverse(\"admin:votingrecord-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = VotingRecordImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Map Voting Record Fields\",\n 'headers': headers,\n 'model_fields': VOTING_RECORD_MAPPABLE_FIELDS,\n 'tenant_id': tenant.id,\n 'file_path': file_path,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = VotingRecordImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Voting Records\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)\n@admin.register(CampaignSettings)\nclass CampaignSettingsAdmin(admin.ModelAdmin):\n list_display = ('tenant', 'smtp_host', 'email_from_address', 'timezone')\n list_filter = ('tenant',)\n search_fields = ('tenant__name', 'smtp_host', 'email_from_address')\n fieldsets = (\n (None, {\n 'fields': ('tenant', 'timezone', 'donation_goal')\n }),\n ('Twilio Settings', {\n 'fields': ('twilio_account_sid', 'twilio_auth_token', 'twilio_from_number'),\n 'classes': ('collapse',),\n }),\n ('SMTP Settings', {\n 'fields': (\n 'email_from_address',\n 'smtp_host',\n 'smtp_port',\n 'smtp_username',\n 'smtp_password',\n 'smtp_use_tls',\n 'smtp_use_ssl',\n ),"
/home/ubuntu/.gemini/tmp/ec2c6b8cb2ac3d6ef62b888a61e62d5ab1e71e46b3c4963f2a040a3c33c7038b/chats/session-2026-02-06T14-13-3fce0361.json: "output": "from decimal import Decimal\nfrom datetime import datetime, date\nfrom django.db import transaction\nfrom django.http import HttpResponse\nfrom django.utils.safestring import mark_safe\nimport csv\nimport io\nimport logging\nimport tempfile\nimport os\nfrom django.contrib import admin, messages\nfrom django.urls import path, reverse\nfrom django.shortcuts import render, redirect\nfrom django.template.response import TemplateResponse\nfrom .models import (\n format_phone_number,\n Tenant, TenantUserRole, InteractionType, DonationMethod, ElectionType, EventType, Voter,\n VotingRecord, Event, EventParticipation, Donation, Interaction, VoterLikelihood, CampaignSettings,\n Interest, Volunteer, VolunteerEvent, ParticipationStatus, VolunteerRole\n)\nfrom .forms import (\n VoterImportForm, EventImportForm, EventParticipationImportForm,\n DonationImportForm, InteractionImportForm, VoterLikelihoodImportForm,\n VolunteerImportForm, VotingRecordImportForm\n)\n\nlogger = logging.getLogger(__name__)\n\nVOTER_MAPPABLE_FIELDS = [\n ('voter_id', 'Voter ID'),\n ('first_name', 'First Name'),\n ('last_name', 'Last Name'),\n ('nickname', 'Nickname'),\n ('birthdate', 'Birthdate'),\n ('address_street', 'Street Address'),\n ('city', 'City'),\n ('state', 'State'),\n ('prior_state', 'Prior State'),\n ('zip_code', 'Zip Code'),\n ('county', 'County'),\n ('neighborhood', 'Neighborhood'),\n ('phone', 'Phone'),\n ('notes', 'Notes'),\n ('phone_type', 'Phone Type'),\n ('email', 'Email'),\n ('district', 'District'),\n ('precinct', 'Precinct'),\n ('registration_date', 'Registration Date'),\n ('is_targeted', 'Is Targeted'),\n ('candidate_support', 'Candidate Support'),\n ('yard_sign', 'Yard Sign'),\n ('window_sticker', 'Window Sticker'),\n ('latitude', 'Latitude'),\n ('longitude', 'Longitude'),\n ('secondary_phone', 'Secondary Phone'),\n ('secondary_phone_type', 'Secondary Phone Type'),\n ('door_visit', 'Door Visit'),\n]\n\nEVENT_MAPPABLE_FIELDS = [\n ('name', 'Name'),\n ('date', 'Date'),\n ('start_time', 'Start Time'),\n ('end_time', 'End Time'),\n ('event_type', 'Event Type (Name)'),\n ('description', 'Description'),\n ('location_name', 'Location Name'),\n ('address', 'Address'),\n ('city', 'City'),\n ('state', 'State'),\n ('zip_code', 'Zip Code'),\n ('latitude', 'Latitude'),\n ('longitude', 'Longitude'),\n]\n\nEVENT_PARTICIPATION_MAPPABLE_FIELDS = [\n ('voter_id', 'Voter ID'),\n ('first_name', 'First Name'),\n ('last_name', 'Last Name'),\n ('event_name', 'Event Name'),\n ('participation_status', 'Participation Status'),\n]\n\nDONATION_MAPPABLE_FIELDS = [\n ('voter_id', 'Voter ID'),\n ('date', 'Date'),\n ('amount', 'Amount'),\n ('method', 'Donation Method (Name)'),\n]\n\nINTERACTION_MAPPABLE_FIELDS = [\n ('voter_id', 'Voter ID'),\n ('volunteer_email', 'Volunteer Email'),\n ('date', 'Date'),\n ('type', 'Interaction Type (Name)'),\n ('description', 'Description'),\n ('notes', 'Notes'),\n]\n\n\nVOLUNTEER_MAPPABLE_FIELDS = [\n ('first_name', 'First Name'),\n ('last_name', 'Last Name'),\n ('email', 'Email'),\n ('phone', 'Phone'),\n ('notes', 'Notes'),\n]\n\nVOTER_LIKELIHOOD_MAPPABLE_FIELDS = [\n ('voter_id', 'Voter ID'),\n ('election_type', 'Election Type (Name)'),\n ('likelihood', 'Likelihood'),\n]\n\nVOTING_RECORD_MAPPABLE_FIELDS = [\n ('voter_id', 'Voter ID'),\n ('election_date', 'Election Date'),\n ('election_description', 'Election Description'),\n ('primary_party', 'Primary Party'),\n]\n\nclass BaseImportAdminMixin:\n def download_errors(self, request):\n logger.info(f\"download_errors called for {self.model._meta.model_name}\")\n session_key = f\"{self.model._meta.model_name}_import_errors\"\n failed_rows = request.session.get(session_key, [])\n if not failed_rows:\n self.message_user(request, \"No error log found in session.\", level=messages.WARNING)\n return redirect(\"../\")\n\n response = HttpResponse(content_type=\"text/csv\")\n response[\"Content-Disposition\"] = f\"attachment; filename={self.model._meta.model_name}_import_errors.csv\"\n\n if failed_rows:\n all_keys = set()\n for r in failed_rows:\n all_keys.update(r.keys())\n\n writer = csv.DictWriter(response, fieldnames=sorted(list(all_keys)))\n writer.writeheader()\n writer.writerows(failed_rows)\n\n return response\n\n def chunk_reader(self, reader, size):\n chunk = []\n for row in reader:\n chunk.append(row)\n if len(chunk) == size:\n yield chunk\n chunk = []\n if chunk:\n yield chunk\nclass TenantUserRoleInline(admin.TabularInline):\n model = TenantUserRole\n extra = 1\n\nclass CampaignSettingsInline(admin.StackedInline):\n model = CampaignSettings\n can_delete = False\n\n@admin.register(Tenant)\nclass TenantAdmin(admin.ModelAdmin):\n list_display = ('name', 'created_at')\n search_fields = ('name',)\n inlines = [TenantUserRoleInline, CampaignSettingsInline]\n\n@admin.register(TenantUserRole)\nclass TenantUserRoleAdmin(admin.ModelAdmin):\n list_display = ('user', 'tenant', 'role')\n list_filter = ('tenant', 'role')\n search_fields = ('user__username', 'tenant__name')\n\n@admin.register(InteractionType)\nclass InteractionTypeAdmin(admin.ModelAdmin):\n list_display = ('name', 'tenant', 'is_active')\n list_filter = ('tenant', 'is_active')\n search_fields = ('name',)\n\n@admin.register(DonationMethod)\nclass DonationMethodAdmin(admin.ModelAdmin):\n list_display = ('name', 'tenant', 'is_active')\n list_filter = ('tenant', 'is_active')\n search_fields = ('name',)\n\n@admin.register(VolunteerRole)\nclass VolunteerRoleAdmin(admin.ModelAdmin):\n list_display = (\"name\", \"tenant\", \"is_active\")\n list_filter = (\"tenant\", \"is_active\")\n search_fields = (\"name\",)\n\n@admin.register(ElectionType)\nclass ElectionTypeAdmin(admin.ModelAdmin):\n list_display = ('name', 'tenant', 'is_active')\n list_filter = ('tenant', 'is_active')\n search_fields = ('name',)\n\n@admin.register(EventType)\nclass EventTypeAdmin(admin.ModelAdmin):\n list_display = ('name', 'tenant', 'is_active', 'default_volunteer_role')\n list_filter = ('tenant', 'is_active')\n search_fields = ('name',)\n filter_horizontal = ('available_roles',)\n\n\n@admin.register(ParticipationStatus)\nclass ParticipationStatusAdmin(admin.ModelAdmin):\n list_display = ('name', 'tenant', 'is_active')\n list_filter = ('tenant', 'is_active')\n search_fields = ('name',)\n change_list_template = 'admin/participationstatus_change_list.html'\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n@admin.register(Interest)\nclass InterestAdmin(admin.ModelAdmin):\n list_display = ('name', 'tenant')\n list_filter = ('tenant',)\n fields = ('tenant', 'name')\n search_fields = ('name',)\n\nclass VotingRecordInline(admin.TabularInline):\n model = VotingRecord\n extra = 1\n\nclass DonationInline(admin.TabularInline):\n model = Donation\n extra = 1\n\nclass InteractionInline(admin.TabularInline):\n model = Interaction\n extra = 1\n autocomplete_fields = ['voter', 'type', 'volunteer']\n\nclass VoterLikelihoodInline(admin.TabularInline):\n model = VoterLikelihood\n extra = 1\n\nclass VolunteerEventInline(admin.TabularInline):\n model = VolunteerEvent\n extra = 1\n\n@admin.register(Voter)\nclass VoterAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('first_name', 'last_name', 'nickname', 'voter_id', 'tenant', 'district', 'candidate_support', 'is_targeted', 'city', 'state', 'prior_state')\n list_filter = ('tenant', 'candidate_support', 'is_targeted', 'phone_type', 'yard_sign', 'district', 'city', 'state', 'prior_state')\n search_fields = ('first_name', 'last_name', 'nickname', 'voter_id', 'address', 'city', 'state', 'prior_state', 'zip_code', 'county')\n inlines = [VotingRecordInline, DonationInline, InteractionInline, VoterLikelihoodInline]\n readonly_fields = ('address',)\n change_list_template = \"admin/voter_change_list.html\"\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context[\"tenants\"] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='voter-download-errors'),\n path('import-voters/', self.admin_site.admin_view(self.import_voters), name='import-voters'),\n ]\n return my_urls + urls\n\n\n def import_voters(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get(\"file_path\")\n tenant_id = request.POST.get(\"tenant\")\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in VOTER_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f\"map_{field_name}\")\n\n try:\n with open(file_path, \"r\", encoding=\"utf-8-sig\") as f:\n # Optimization: Skip full count for very large files in preview if needed,\n # but here we'll keep it for accuracy unless it's a known bottleneck.\n # For now, let's just do a fast line count.\n total_count = sum(1 for line in f) - 1\n f.seek(0)\n reader = csv.DictReader(f)\n preview_rows = []\n voter_ids_for_preview = []\n for i, row in enumerate(reader):\n if i < 10:\n preview_rows.append(row)\n v_id = row.get(mapping.get(\"voter_id\"))\n if v_id:\n voter_ids_for_preview.append(v_id.strip())\n else:\n break\n\n existing_preview_ids = set(Voter.objects.filter(tenant=tenant, voter_id__in=voter_ids_for_preview).values_list(\"voter_id\", flat=True))\n\n create_count = 0\n update_count = 0\n\n for row in preview_rows:\n voter_id_val = row.get(mapping.get(\"voter_id\"))\n if voter_id_val and voter_id_val.strip() in existing_preview_ids:\n update_count += 1\n else:\n create_count += 1\n\n context = self.admin_site.each_context(request)\n context.update({\n \"title\": \"Import Preview\",\n \"total_count\": total_count,\n \"create_count\": create_count,\n \"update_count\": update_count,\n \"preview_data\": preview_rows, # This should be improved to show actual changes\n \"mapping\": mapping,\n \"file_path\": file_path,\n \"tenant_id\": tenant_id,\n \"action_url\": request.path,\n \"opts\": self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n elif \"_import\" in request.POST:\n file_path = request.POST.get(\"file_path\")\n tenant_id = request.POST.get(\"tenant\")\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in VOTER_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f\"map_{field_name}\")\n\n try:\n created_count = 0\n updated_count = 0\n skipped_no_change = 0\n skipped_no_id = 0\n errors = 0\n failed_rows = []\n total_processed = 0\n\n # Temporary storage for error rows to avoid holding large file in memory\n temp_error_file = None\n temp_error_file_path = None\n\n # Process in chunks to reduce memory usage for very large files\n with open(file_path, \"r\", encoding=\"utf-8-sig\") as f_read:\n reader = csv.DictReader(f_read)\n for i, row in enumerate(reader):\n total_processed += 1\n try:\n raw_voter_id = row.get(mapping.get(\"voter_id\"))\n voter_id = raw_voter_id.strip() if raw_voter_id else None\n\n if not voter_id:\n # Enhanced error message to guide the user\n mapped_column_name = mapping.get(\"voter_id\", \"N/A\")\n error_detail = f\"Raw value: '{raw_voter_id}'. \" if raw_voter_id is not None else \"Value was None.\"\n row[\"Import Error\"] = f\"Voter ID is required. Please check if the '{mapped_column_name}' column is correctly mapped and contains values for all rows. {error_detail}\"\n failed_rows.append(row)\n skipped_no_id += 1\n errors += 1\n continue\n\n defaults = {}\n # Map other fields dynamically\n for field_name, _ in VOTER_MAPPABLE_FIELDS:\n csv_column = mapping.get(field_name)\n if csv_column and csv_column in row:\n field_value = row[csv_column].strip()\n if field_name == \"birthdate\" or field_name == \"registration_date\":\n # Handle date conversions\n if field_value:\n try:\n # Attempt to parse common date formats\n if '/' in field_value:\n # Try MM/DD/YYYY or DD/MM/YYYY\n if len(field_value.split('/')[2]) == 2: # YY format\n dt = datetime.strptime(field_value, '%m/%d/%y').date() if len(field_value.split('/')[0]) < 3 else datetime.strptime(field_value, '%d/%m/%y').date() # noqa\n else:\n dt = datetime.strptime(field_value, '%m/%d/%Y').date() if len(field_value.split('/')[0]) < 3 else datetime.strptime(field_value, '%d/%m/%Y').date() # noqa\n elif '-' in field_value:\n # Try YYYY-MM-DD or DD-MM-YYYY or MM-DD-YYYY\n if len(field_value.split('-')[0]) == 4: # YYYY format\n dt = datetime.strptime(field_value, '%Y-%m-%d').date()\n elif len(field_value.split('-')[2]) == 4: # YYYY format\n dt = datetime.strptime(field_value, '%m-%d-%Y').date() if len(field_value.split('-')[0]) < 3 else datetime.strptime(field_value, '%d-%m-%Y').date() # noqa\n else:\n # Default to MM-DD-YY\n dt = datetime.strptime(field_value, '%m-%d-%y').date()\n else:\n dt = None\n\n if dt:\n defaults[field_name] = dt\n else:\n logger.warning(f\"Could not parse date '{field_value}' for field {field_name}. Skipping.\")\n except ValueError as ve:\n logger.warning(f\"Date parsing error for '{field_value}' in field {field_name}: {ve}\")\n except Exception as ex:\n logger.error(f\"Unexpected error parsing date '{field_value}' for field {field_name}: {ex}\")\n elif field_name == \"is_targeted\" or field_name == \"yard_sign\" or field_name == \"window_sticker\" or field_name == \"door_visit\":\n # Handle boolean fields\n if field_value.lower() == 'true' or field_value == '1':\n defaults[field_name] = True\n elif field_value.lower() == 'false' or field_value == '0':\n defaults[field_name] = False\n else:\n defaults[field_name] = None # Or sensible default/error\n elif field_name == \"phone\":\n defaults[field_name] = format_phone_number(field_value)\n elif field_name == \"email\":\n defaults[field_name] = field_value.lower() # Store emails as lowercase\n elif field_name == \"candidate_support\":\n if field_value in [choice[0] for choice in Voter.CANDIDATE_SUPPORT_CHOICES]:\n defaults[field_name] = field_value\n else:\n logger.warning(f\"Invalid candidate_support value: {field_value}. Skipping.\")\n elif field_name == \"phone_type\":\n if field_value in [choice[0] for choice in Voter.PHONE_TYPE_CHOICES]:\n defaults[field_name] = field_value\n else:\n logger.warning(f\"Invalid phone_type value: {field_value}. Skipping.\")\n elif field_name == \"secondary_phone_type\":\n if field_value in [choice[0] for choice in Voter.PHONE_TYPE_CHOICES]:\n defaults[field_name] = field_value\n else:\n logger.warning(f\"Invalid secondary_phone_type value: {field_value}. Skipping.\")\n elif field_name == \"state\" or field_name == \"prior_state\":\n # Ensure state is uppercase and valid length\n if field_value and len(field_value) <= 2:\n defaults[field_name] = field_value.upper()\n else:\n logger.warning(f\"Invalid state value: {field_value}. Skipping.\")\n else:\n defaults[field_name] = field_value\n\n # Try to get voter. If not found, create new. Update if found.\n voter, created = Voter.objects.update_or_create(\n tenant=tenant,\n voter_id=voter_id,\n defaults=defaults\n )\n if created:\n created_count += 1\n else:\n updated_count += 1\n\n # Special handling for interests - assuming a comma-separated list in CSV\n if 'interests' in mapping and row.get(mapping['interests']):\n interest_names = [name.strip() for name in row[mapping['interests']].split(',') if name.strip()]\n for interest_name in interest_names:\n interest, _ = Interest.objects.get_or_create(tenant=tenant, name=interest_name)\n voter.interests.add(interest)\n\n if (i + 1) % 100 == 0:\n print(f\"DEBUG: Voter import progress: {total_processed} processed. {created_count} created. {updated_count} updated.\")\n\n except Exception as e:\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n logger.error(f\"Error importing row: {row}. Error: {e}\")\n\n # Clean up the temporary file\n if os.path.exists(file_path):\n os.remove(file_path)\n\n if temp_error_file_path and os.path.exists(temp_error_file_path):\n os.remove(temp_error_file_path)\n\n self.message_user(request, f\"Import complete: {created_count + updated_count} voters created/updated. ({created_count} new, {updated_count} updated, {skipped_no_change} skipped with no changes, {skipped_no_id} skipped missing ID, {errors} errors)\")\n # Store failed rows in session for download, limit to avoid session overflow\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n\n if errors > 0:\n error_url = reverse(\"admin:voter-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = VoterImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n \"title\": \"Map Voter Fields\",\n \"headers\": headers,\n \"model_fields\": VOTER_MAPPABLE_FIELDS,\n \"tenant_id\": tenant.id,\n \"file_path\": file_path,\n \"action_url\": request.path,\n \"opts\": self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = VoterImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Voters\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)\n\n\n@admin.register(Event)\nclass EventAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('name', 'date', 'event_type', 'tenant', 'location_name', 'address', 'city', 'state', 'zip_code')\n list_filter = ('tenant', 'event_type')\n search_fields = ('name', 'location_name', 'address', 'city', 'state', 'zip_code')\n inlines = [VolunteerEventInline]\n change_list_template = \"admin/event_change_list.html\"\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='event-download-errors'),\n path('import-events/', self.admin_site.admin_view(self.import_events), name='import-events'),\n ]\n return my_urls + urls\n\n def import_events(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n mapping = {}\n for field_name, _ in EVENT_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n total_count = 0\n create_count = 0\n update_count = 0\n preview_data = []\n for row in reader:\n total_count += 1\n event_name = row.get(mapping.get('name'))\n event_date = row.get(mapping.get('date'))\n\n exists = False\n if event_name and event_date:\n try:\n # Assuming name and date uniquely identify an event\n # This might need refinement based on actual data uniqueness requirements\n if '/' in event_date:\n dt = datetime.strptime(event_date, '%m/%d/%Y').date()\n elif '-' in event_date:\n dt = datetime.strptime(event_date, '%Y-%m-%d').date()\n else:\n dt = None\n\n if dt:\n exists = Event.objects.filter(tenant=tenant, name=event_name, date=dt).exists()\n\n except ValueError:\n # Handle cases where date parsing fails\n pass\n\n if exists:\n update_count += 1\n action = 'update'\n else:\n create_count += 1\n action = 'create'\n\n if len(preview_data) < 10:\n preview_data.append({\n 'action': action,\n 'identifier': f\"Event: {event_name} (Date: {event_date})\",\n 'details': f\"Location: {row.get(mapping.get('location_name', '')) or ''}\"\n })\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Import Preview\",\n 'total_count': total_count,\n 'create_count': create_count,\n 'update_count': update_count,\n 'preview_data': preview_data,\n 'mapping': mapping,\n 'file_path': file_path,\n 'tenant_id': tenant_id,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n\n elif \"_import\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in EVENT_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n count = 0\n errors = 0\n failed_rows = []\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n for row in reader:\n try:\n event_name = row.get(mapping.get('name'))\n event_date = row.get(mapping.get('date'))\n event_type_name = row.get(mapping.get('event_type'))\n\n if not event_name or not event_date or not event_type_name:\n row[\"Import Error\"] = \"Missing event name, date, or type\"\n failed_rows.append(row)\n errors += 1\n continue\n\n # Date parsing for event_date\n try:\n if '/' in event_date:\n parsed_date = datetime.strptime(event_date, '%m/%d/%Y').date()\n elif '-' in event_date:\n parsed_date = datetime.strptime(event_date, '%Y-%m-%d').date()\n else:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n except ValueError:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n\n event_type_obj, _ = EventType.objects.get_or_create(tenant=tenant, name=event_type_name)\n\n defaults = {\n 'date': parsed_date,\n 'event_type': event_type_obj,\n 'description': row.get(mapping.get('description')) or '',\n 'location_name': row.get(mapping.get('location_name')) or '',\n 'address': row.get(mapping.get('address')) or '',\n 'city': row.get(mapping.get('city')) or '',\n 'state': row.get(mapping.get('state')) or '',\n 'zip_code': row.get(mapping.get('zip_code')) or '',\n 'latitude': row.get(mapping.get('latitude')) or None,\n 'longitude': row.get(mapping.get('longitude')) or None,\n }\n\n # Handle start_time and end_time\n start_time_str = row.get(mapping.get('start_time'))\n if start_time_str:\n try:\n defaults['start_time'] = datetime.strptime(start_time_str, '%H:%M').time()\n except ValueError:\n logger.warning(f\"Invalid start_time format: {start_time_str}. Skipping.\")\n end_time_str = row.get(mapping.get('end_time'))\n if end_time_str:\n try:\n defaults['end_time'] = datetime.strptime(end_time_str, '%H:%M').time()\n except ValueError:\n logger.warning(f\"Invalid end_time format: {end_time_str}. Skipping.\")\n\n Event.objects.update_or_create(\n tenant=tenant,\n name=event_name,\n defaults=defaults\n )\n count += 1\n except Exception as e:\n logger.error(f\"Error importing: {e}\")\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n\n if os.path.exists(file_path):\n os.remove(file_path)\n self.message_user(request, f\"Successfully imported {count} events.\")\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n if errors > 0:\n error_url = reverse(\"admin:event-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = EventImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Map Event Fields\",\n 'headers': headers,\n 'model_fields': EVENT_MAPPABLE_FIELDS,\n 'tenant_id': tenant.id,\n 'file_path': file_path,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = EventImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Events\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)\n\n\n@admin.register(Volunteer)\nclass VolunteerAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('first_name', 'last_name', 'email', 'phone', 'tenant')\n list_filter = ('tenant',)\n search_fields = ('first_name', 'last_name', 'email', 'phone')\n change_list_template = \"admin/volunteer_change_list.html\"\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='volunteer-download-errors'),\n path('import-volunteers/', self.admin_site.admin_view(self.import_volunteers), name='import-volunteers'),\n ]\n return my_urls + urls\n\n def import_volunteers(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n mapping = {}\n for field_name, _ in VOLUNTEER_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n total_count = 0\n create_count = 0\n update_count = 0\n preview_data = []\n for row in reader:\n total_count += 1\n email = row.get(mapping.get('email'))\n\n exists = False\n if email:\n exists = Volunteer.objects.filter(tenant=tenant, email=email).exists()\n\n if exists:\n update_count += 1\n action = 'update'\n else:\n create_count += 1\n action = 'create'\n\n if len(preview_data) < 10:\n preview_data.append({\n 'action': action,\n 'identifier': f\"Volunteer: {email}\",\n 'details': f\"Name: {row.get(mapping.get('first_name', '')) or ''} {row.get(mapping.get('last_name', '')) or ''}\"\n })\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Import Preview\",\n 'total_count': total_count,\n 'create_count': create_count,\n 'update_count': update_count,\n 'preview_data': preview_data,\n 'mapping': mapping,\n 'file_path': file_path,\n 'tenant_id': tenant_id,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n\n elif \"_import\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in VOLUNTEER_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n count = 0\n errors = 0\n failed_rows = []\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n for row in reader:\n try:\n email = row.get(mapping.get('email'))\n if not email:\n row[\"Import Error\"] = \"Missing email\"\n failed_rows.append(row)\n errors += 1\n continue\n\n defaults = {\n 'first_name': row.get(mapping.get('first_name')) or '',\n 'last_name': row.get(mapping.get('last_name')) or '',\n 'phone': format_phone_number(row.get(mapping.get('phone')) or ''),\n 'notes': row.get(mapping.get('notes')) or '',\n }\n\n Volunteer.objects.update_or_create(\n tenant=tenant,\n email=email,\n defaults=defaults\n )\n count += 1\n except Exception as e:\n logger.error(f\"Error importing volunteer: {e}\")\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n\n if os.path.exists(file_path):\n os.remove(file_path)\n self.message_user(request, f\"Successfully imported {count} volunteers.\")\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n if errors > 0:\n error_url = reverse(\"admin:volunteer-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = VolunteerImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Map Volunteer Fields\",\n 'headers': headers,\n 'model_fields': VOLUNTEER_MAPPABLE_FIELDS,\n 'tenant_id': tenant.id,\n 'file_path': file_path,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = VolunteerImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Volunteers\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)\n\n\n@admin.register(EventParticipation)\nclass EventParticipationAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('event', 'voter', 'participation_status')\n list_filter = ('event', 'participation_status', 'voter__tenant')\n search_fields = ('event__name', 'voter__first_name', 'voter__last_name', 'voter__voter_id')\n change_list_template = 'admin/eventparticipation_change_list.html'\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_list(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='eventparticipation-download-errors'),\n path('import-event-participations/', self.admin_site.admin_view(self.import_event_participations), name='import-event-participations'),\n ]\n return my_urls + urls\n\n def import_event_participations(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n mapping = {}\n for field_name, _ in EVENT_PARTICIPATION_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n total_count = 0\n create_count = 0\n update_count = 0\n preview_data = []\n for row in reader:\n total_count += 1\n voter_id = row.get(mapping.get('voter_id'))\n\n # Extract first_name and last_name from CSV based on mapping\n csv_first_name = row.get(mapping.get('first_name'), '')\n csv_last_name = row.get(mapping.get('last_name'), '')\n csv_full_name = f\"{csv_first_name} {csv_last_name}\".strip()\n\n exists = False\n voter_full_name = \"N/A\" # Initialize voter_full_name\n if voter_id:\n try:\n voter = Voter.objects.get(tenant=tenant, voter_id=voter_id)\n voter_full_name = f\"{voter.first_name} {voter.last_name}\" # Get voter's full name\n if event_name:\n exists = EventParticipation.objects.filter(voter=voter, event__name=event_name).exists()\n except Voter.DoesNotExist:\n pass\n\n if exists:\n update_count += 1\n action = 'update'\n else:\n create_count += 1\n action = 'create'\n\n if len(preview_data) < 10:\n preview_data.append({\n 'action': action,\n 'csv_full_name': csv_full_name, # Add CSV name\n 'identifier': f\"Voter: {voter_full_name} (ID: {voter_id})\" if voter_id else \"N/A\", # Include full name\n 'details': f\"Participation: {row.get(mapping.get('participation_status', '')) or ''}\"\n })\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Import Preview\",\n 'total_count': total_count,\n 'create_count': create_count,\n 'update_count': update_count,\n 'preview_data': preview_data,\n 'mapping': mapping,\n 'file_path': file_path,\n 'tenant_id': tenant_id,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n\n elif \"_import\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in EVENT_PARTICIPATION_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n count = 0\n errors = 0\n failed_rows = []\n for row in reader:\n try:\n voter_id = row.get(mapping.get('voter_id')) if mapping.get('voter_id') else None\n participation_status_val = row.get(mapping.get('participation_status')) if mapping.get('participation_status') else None\n\n if voter_id: # Only strip if voter_id is not None\n voter_id = voter_id.strip()\n\n if not voter_id:\n row[\"Import Error\"] = \"Missing voter ID\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n voter = Voter.objects.get(tenant=tenant, voter_id=voter_id)\n except Voter.DoesNotExist:\n error_msg = f\"Voter with ID {voter_id} not found\"\n logger.error(error_msg)\n row[\"Import Error\"] = error_msg\n failed_rows.append(row)\n errors += 1\n continue\n\n event = None\n event_name = row.get(mapping.get('event_name')) if mapping.get('event_name') else None\n if event_name:\n try:\n event = Event.objects.get(tenant=tenant, name=event_name)\n except Event.DoesNotExist:\n pass\n\n if not event:\n error_msg = \"Event not found (check Event Name)\"\n logger.error(error_msg)\n row[\"Import Error\"] = error_msg\n failed_rows.append(row)\n errors += 1\n continue\n\n defaults = {}\n if participation_status_val and participation_status_val.strip():\n status_obj, _ = ParticipationStatus.objects.get_or_create(tenant=tenant, name=participation_status_val.strip())\n defaults['participation_status'] = status_obj\n else:\n # Default to 'Invited' if not specified\n status_obj, _ = ParticipationStatus.objects.get_or_create(tenant=tenant, name='Invited')\n defaults['participation_status'] = status_obj\n EventParticipation.objects.update_or_create(\n event=event,\n voter=voter,\n defaults=defaults\n )\n count += 1\n except Exception as e:\n logger.error(f\"Error importing: {e}\")\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n\n if os.path.exists(file_path):\n os.remove(file_path)\n self.message_user(request, f\"Successfully imported {count} participations.\")\n # Optimization: Limit error log size in session to avoid overflow\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n logger.info(f\"Stored {len(failed_rows)} failed rows in session for {self.model._meta.model_name}\")\n if errors > 0:\n error_url = reverse(\"admin:eventparticipation-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = EventParticipationImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Map Participation Fields\",\n 'headers': headers,\n 'model_fields': EVENT_PARTICIPATION_MAPPABLE_FIELDS,\n 'tenant_id': tenant.id,\n 'file_path': file_path,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = EventParticipationImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Participations\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)\n\n@admin.register(Donation)\nclass DonationAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('voter', 'date', 'amount', 'method')\n list_filter = ('voter__tenant', 'method')\n search_fields = ('voter__first_name', 'voter__last_name', 'voter__voter_id', 'method__name')\n change_list_template = 'admin/donation_change_list.html'\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='donation-download-errors'),\n path('import-donations/', self.admin_site.admin_view(self.import_donations), name='import-donations'),\n ]\n return my_urls + urls\n\n def import_donations(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n mapping = {}\n for field_name, _ in DONATION_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n total_count = 0\n create_count = 0\n update_count = 0\n preview_data = []\n for row in reader:\n total_count += 1\n voter_id = row.get(mapping.get('voter_id'))\n\n exists = False\n if voter_id:\n exists = Voter.objects.filter(tenant=tenant, voter_id=voter_id).exists()\n\n if exists:\n update_count += 1\n action = 'update'\n else:\n create_count += 1\n action = 'create'\n\n if len(preview_data) < 10:\n preview_data.append({\n 'action': action,\n 'identifier': f\"Voter ID: {voter_id}\",\n 'details': f\"Amount: {row.get(mapping.get('amount', '')) or ''}, Method: {row.get(mapping.get('method', '')) or ''}\"\n })\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Import Preview\",\n 'total_count': total_count,\n 'create_count': create_count,\n 'update_count': update_count,\n 'preview_data': preview_data,\n 'mapping': mapping,\n 'file_path': file_path,\n 'tenant_id': tenant_id,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n\n elif \"_import\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in DONATION_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n count = 0\n errors = 0\n failed_rows = []\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n for row in reader:\n try:\n voter_id = row.get(mapping.get('voter_id'))\n date_str = row.get(mapping.get('date'))\n amount_str = row.get(mapping.get('amount'))\n method_name = row.get(mapping.get('method'))\n\n if voter_id: # Only strip if voter_id is not None\n voter_id = voter_id.strip()\n\n if not voter_id:\n row[\"Import Error\"] = \"Missing voter ID\"\n failed_rows.append(row)\n errors += 1\n continue\n\n if not date_str or not amount_str:\n row[\"Import Error\"] = \"Missing date or amount\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n voter = Voter.objects.get(tenant=tenant, voter_id=voter_id)\n except Voter.DoesNotExist:\n row[\"Import Error\"] = f\"Voter {voter_id} not found\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n if '/' in date_str:\n parsed_date = datetime.strptime(date_str, '%m/%d/%Y').date()\n elif '-' in date_str:\n parsed_date = datetime.strptime(date_str, '%Y-%m-%d').date()\n else:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n except ValueError:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n amount = Decimal(amount_str)\n except InvalidOperation:\n row[\"Import Error\"] = \"Invalid amount format\"\n failed_rows.append(row)\n errors += 1\n continue\n\n donation_method, _ = DonationMethod.objects.get_or_create(tenant=tenant, name=method_name)\n\n Donation.objects.create(\n voter=voter,\n date=parsed_date,\n amount=amount,\n method=donation_method\n )\n count += 1\n except Exception as e:\n logger.error(f\"Error importing: {e}\")\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n\n if os.path.exists(file_path):\n os.remove(file_path)\n self.message_user(request, f\"Successfully imported {count} donations.\")\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n if errors > 0:\n error_url = reverse(\"admin:donation-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = DonationImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Map Donation Fields\",\n 'headers': headers,\n 'model_fields': DONATION_MAPPABLE_FIELDS,\n 'tenant_id': tenant.id,\n 'file_path': file_path,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = DonationImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Donations\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)\n\n@admin.register(Interaction)\nclass InteractionAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('voter', 'date', 'type', 'description', 'volunteer')\n list_filter = ('voter__tenant', 'type', 'volunteer')\n search_fields = ('voter__first_name', 'voter__last_name', 'voter__voter_id', 'description', 'volunteer__first_name', 'volunteer__last_name')\n autocomplete_fields = ['voter', 'volunteer']\n change_list_template = 'admin/interaction_change_list.html'\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='interaction-download-errors'),\n path('import-interactions/', self.admin_site.admin_view(self.import_interactions), name='import-interactions'),\n ]\n return my_urls + urls\n\n def import_interactions(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n mapping = {}\n for field_name, _ in INTERACTION_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n total_count = 0\n create_count = 0\n update_count = 0\n preview_data = []\n for row in reader:\n total_count += 1\n voter_id = row.get(mapping.get('voter_id'))\n volunteer_email = row.get(mapping.get('volunteer_email'))\n\n exists = False\n if voter_id:\n exists = Voter.objects.filter(tenant=tenant, voter_id=voter_id).exists()\n\n if exists:\n update_count += 1\n action = 'update'\n else:\n create_count += 1\n action = 'create'\n\n if len(preview_data) < 10:\n preview_data.append({\n 'action': action,\n 'identifier': f\"Voter ID: {voter_id}\",\n 'details': f\"Type: {row.get(mapping.get('type', '')) or ''}, Volunteer: {volunteer_email or ''}\"\n })\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Import Preview\",\n 'total_count': total_count,\n 'create_count': create_count,\n 'update_count': update_count,\n 'preview_data': preview_data,\n 'mapping': mapping,\n 'file_path': file_path,\n 'tenant_id': tenant_id,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n\n elif \"_import\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in INTERACTION_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n count = 0\n errors = 0\n failed_rows = []\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n for row in reader:\n try:\n voter_id = row.get(mapping.get('voter_id'))\n volunteer_email = row.get(mapping.get('volunteer_email'))\n date_str = row.get(mapping.get('date'))\n type_name = row.get(mapping.get('type'))\n\n if voter_id: # Only strip if voter_id is not None\n voter_id = voter_id.strip()\n\n if not voter_id:\n row[\"Import Error\"] = \"Missing voter ID\"\n failed_rows.append(row)\n errors += 1\n continue\n\n if not date_str or not type_name:\n row[\"Import Error\"] = \"Missing date or description\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n voter = Voter.objects.get(tenant=tenant, voter_id=voter_id)\n except Voter.DoesNotExist:\n row[\"Import Error\"] = f\"Voter {voter_id} not found\"\n failed_rows.append(row)\n errors += 1\n continue\n\n volunteer = None\n if volunteer_email:\n try:\n volunteer = Volunteer.objects.get(tenant=tenant, email=volunteer_email)\n except Volunteer.DoesNotExist:\n pass # Volunteer is optional\n\n try:\n if '/' in date_str:\n parsed_date = datetime.strptime(date_str, '%m/%d/%Y').date()\n elif '-' in date_str:\n parsed_date = datetime.strptime(date_str, '%Y-%m-%d').date()\n else:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n except ValueError:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n\n interaction_type, _ = InteractionType.objects.get_or_create(tenant=tenant, name=type_name)\n\n Interaction.objects.create(\n voter=voter,\n volunteer=volunteer,\n date=parsed_date,\n type=interaction_type,\n description=row.get(mapping.get('description')) or '',\n notes=row.get(mapping.get('notes')) or ''\n )\n count += 1\n except Exception as e:\n logger.error(f\"Error importing: {e}\")\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n\n if os.path.exists(file_path):\n os.remove(file_path)\n self.message_user(request, f\"Successfully imported {count} interactions.\")\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n if errors > 0:\n error_url = reverse(\"admin:interaction-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = InteractionImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Map Interaction Fields\",\n 'headers': headers,\n 'model_fields': INTERACTION_MAPPABLE_FIELDS,\n 'tenant_id': tenant.id,\n 'file_path': file_path,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = InteractionImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Interactions\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)\n\n@admin.register(VoterLikelihood)\nclass VoterLikelihoodAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('voter', 'election_type', 'likelihood')\n list_filter = ('voter__tenant', 'election_type', 'likelihood')\n search_fields = ('voter__first_name', 'voter__last_name', 'voter__voter_id', 'election_type__name')\n change_list_template = 'admin/voterlikelihood_change_list.html'\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='voterlikelihood-download-errors'),\n path('import-likelihoods/', self.admin_site.admin_view(self.import_likelihoods), name='import-likelihoods'),\n ]\n return my_urls + urls\n\n def import_likelihoods(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n mapping = {}\n for field_name, _ in VOTER_LIKELIHOOD_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n total_count = 0\n create_count = 0\n update_count = 0\n preview_data = []\n for row in reader:\n total_count += 1\n voter_id = row.get(mapping.get('voter_id'))\n\n exists = False\n if voter_id:\n exists = Voter.objects.filter(tenant=tenant, voter_id=voter_id).exists()\n\n if exists:\n update_count += 1\n action = 'update'\n else:\n create_count += 1\n action = 'create'\n\n if len(preview_data) < 10:\n preview_data.append({\n 'action': action,\n 'identifier': f\"Voter ID: {voter_id}\",\n 'details': f\"Likelihood: {row.get(mapping.get('likelihood', '')) or ''}\"\n })\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Import Preview\",\n 'total_count': total_count,\n 'create_count': create_count,\n 'update_count': update_count,\n 'preview_data': preview_data,\n 'mapping': mapping,\n 'file_path': file_path,\n 'tenant_id': tenant_id,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n\n elif \"_import\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in VOTER_LIKELIHOOD_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n count = 0\n errors = 0\n failed_rows = []\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n for row in reader:\n try:\n voter_id = row.get(mapping.get('voter_id'))\n election_type_name = row.get(mapping.get('election_type'))\n likelihood_val = row.get(mapping.get('likelihood'))\n\n if voter_id: # Only strip if voter_id is not None\n voter_id = voter_id.strip()\n\n if not voter_id:\n row[\"Import Error\"] = \"Missing voter ID\"\n failed_rows.append(row)\n errors += 1\n continue\n\n if not election_type_name or not likelihood_val:\n row[\"Import Error\"] = \"Missing election type or likelihood\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n voter = Voter.objects.get(tenant=tenant, voter_id=voter_id)\n except Voter.DoesNotExist:\n row[\"Import Error\"] = f\"Voter {voter_id} not found\"\n failed_rows.append(row)\n errors += 1\n continue\n\n election_type, _ = ElectionType.objects.get_or_create(tenant=tenant, name=election_type_name)\n\n VoterLikelihood.objects.update_or_create(\n voter=voter,\n election_type=election_type,\n defaults={'likelihood': likelihood_val}\n )\n count += 1\n except Exception as e:\n print(f\"DEBUG: Likelihood import failed: {e}\")\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n\n if os.path.exists(file_path):\n os.remove(file_path)\n self.message_user(request, f\"Import complete: {count} likelihoods created/updated.\")\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n if errors > 0:\n error_url = reverse(\"admin:voterlikelihood-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = VoterLikelihoodImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Map Likelihood Fields\",\n 'headers': headers,\n 'model_fields': VOTER_LIKELIHOOD_MAPPABLE_FIELDS,\n 'tenant_id': tenant.id,\n 'file_path': file_path,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = VoterLikelihoodImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Likelihoods\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)\n\n\n@admin.register(VotingRecord)\nclass VotingRecordAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('voter', 'election_date', 'election_description', 'primary_party')\n list_filter = ('voter__tenant', 'primary_party')\n search_fields = ('voter__first_name', 'voter__last_name', 'voter__voter_id', 'election_description')\n change_list_template = 'admin/votingrecord_change_list.html'\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='votingrecord-download-errors'),\n path('import-voting-records/', self.admin_site.admin_view(self.import_voting_records), name='import-voting-records'),\n ]\n return my_urls + urls\n\n def import_voting_records(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n mapping = {}\n for field_name, _ in VOTING_RECORD_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n total_count = 0\n create_count = 0\n update_count = 0\n preview_data = []\n for row in reader:\n total_count += 1\n voter_id = row.get(mapping.get('voter_id'))\n election_date = row.get(mapping.get('election_date'))\n\n exists = False\n if voter_id and election_date:\n try:\n # Assuming voter_id and election_date uniquely identify a voting record\n # This might need refinement based on actual data uniqueness requirements\n if '/' in election_date:\n dt = datetime.strptime(election_date, '%m/%d/%Y').date()\n elif '-' in election_date:\n dt = datetime.strptime(election_date, '%Y-%m-%d').date()\n else:\n dt = None\n\n if dt:\n exists = VotingRecord.objects.filter(voter__tenant=tenant, voter__voter_id=voter_id, election_date=dt).exists()\n\n except ValueError:\n # Handle cases where date parsing fails\n pass\n\n if exists:\n update_count += 1\n action = 'update'\n else:\n create_count += 1\n action = 'create'\n\n if len(preview_data) < 10:\n preview_data.append({\n 'action': action,\n 'identifier': f\"Voter ID: {voter_id} (Election: {election_date})\",\n 'details': f\"Party: {row.get(mapping.get('primary_party', '')) or ''}\"\n })\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Import Preview\",\n 'total_count': total_count,\n 'create_count': create_count,\n 'update_count': update_count,\n 'preview_data': preview_data,\n 'mapping': mapping,\n 'file_path': file_path,\n 'tenant_id': tenant_id,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n\n elif \"_import\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in VOTING_RECORD_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n count = 0\n errors = 0\n failed_rows = []\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n for row in reader:\n try:\n voter_id = row.get(mapping.get('voter_id'))\n election_date_str = row.get(mapping.get('election_date'))\n election_description = row.get(mapping.get('election_description'))\n primary_party = row.get(mapping.get('primary_party'))\n\n if voter_id: # Only strip if voter_id is not None\n voter_id = voter_id.strip()\n\n if not voter_id:\n row[\"Import Error\"] = \"Missing voter ID\"\n failed_rows.append(row)\n errors += 1\n continue\n\n if not election_date_str or not election_description:\n row[\"Import Error\"] = \"Missing election date or description\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n voter = Voter.objects.get(tenant=tenant, voter_id=voter_id)\n except Voter.DoesNotExist:\n row[\"Import Error\"] = f\"Voter {voter_id} not found\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n if '/' in election_date_str:\n parsed_election_date = datetime.strptime(election_date_str, '%m/%d/%Y').date()\n elif '-' in election_date_str:\n parsed_election_date = datetime.strptime(election_date_str, '%Y-%m-%d').date()\n else:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n except ValueError:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n\n VotingRecord.objects.update_or_create(\n voter=voter,\n election_date=parsed_election_date,\n defaults={\n 'election_description': election_description,\n 'primary_party': primary_party or ''\n }\n )\n count += 1\n except Exception as e:\n logger.error(f\"Error importing: {e}\")\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n\n if os.path.exists(file_path):\n os.remove(file_path)\n self.message_user(request, f\"Successfully imported {count} voting records.\")\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n if errors > 0:\n error_url = reverse(\"admin:votingrecord-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = VotingRecordImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Map Voting Record Fields\",\n 'headers': headers,\n 'model_fields': VOTING_RECORD_MAPPABLE_FIELDS,\n 'tenant_id': tenant.id,\n 'file_path': file_path,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = VotingRecordImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Voting Records\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)"
/home/ubuntu/.gemini/tmp/ec2c6b8cb2ac3d6ef62b888a61e62d5ab1e71e46b3c4963f2a040a3c33c7038b/chats/session-2026-02-11T15-22-c8b8cfcf.json: "output": "from decimal import Decimal\nfrom datetime import datetime, date\nfrom django.db import transaction\nfrom django.http import HttpResponse\nfrom django.utils.safestring import mark_safe\nimport csv\nimport io\nimport logging\nimport tempfile\nimport os\nfrom django.contrib import admin, messages\nfrom django.urls import path, reverse\nfrom django.shortcuts import render, redirect\nfrom django.template.response import TemplateResponse\nfrom .models import (\n format_phone_number,\n Tenant, TenantUserRole, InteractionType, DonationMethod, ElectionType, EventType, Voter,\n VotingRecord, Event, EventParticipation, Donation, Interaction, VoterLikelihood, CampaignSettings,\n Interest, Volunteer, VolunteerEvent, ParticipationStatus, VolunteerRole\n)\nfrom .forms import (\n VoterImportForm, EventImportForm, EventParticipationImportForm,\n DonationImportForm, InteractionImportForm, VoterLikelihoodImportForm,\n VolunteerImportForm, VotingRecordImportForm\n)\n\nlogger = logging.getLogger(__name__)\n\nVOTER_MAPPABLE_FIELDS = [\n ('voter_id', 'Voter ID'),\n ('first_name', 'First Name'),\n ('last_name', 'Last Name'),\n ('nickname', 'Nickname'),\n ('birthdate', 'Birthdate'),\n ('address_street', 'Street Address'),\n ('city', 'City'),\n ('state', 'State'),\n ('prior_state', 'Prior State'),\n ('zip_code', 'Zip Code'),\n ('county', 'County'),\n ('neighborhood', 'Neighborhood'),\n ('phone', 'Phone'),\n ('notes', 'Notes'),\n ('phone_type', 'Phone Type'),\n ('email', 'Email'),\n ('district', 'District'),\n ('precinct', 'Precinct'),\n ('registration_date', 'Registration Date'),\n ('is_targeted', 'Is Targeted'),\n ('candidate_support', 'Candidate Support'),\n ('yard_sign', 'Yard Sign'),\n ('window_sticker', 'Window Sticker'),\n ('latitude', 'Latitude'),\n ('longitude', 'Longitude'),\n ('secondary_phone', 'Secondary Phone'),\n ('secondary_phone_type', 'Secondary Phone Type'),\n ('door_visit', 'Door Visit'),\n]\n\nEVENT_MAPPABLE_FIELDS = [\n ('name', 'Name'),\n ('date', 'Date'),\n ('start_time', 'Start Time'),\n ('end_time', 'End Time'),\n ('event_type', 'Event Type (Name)'),\n ('description', 'Description'),\n ('location_name', 'Location Name'),\n ('address', 'Address'),\n ('city', 'City'),\n ('state', 'State'),\n ('zip_code', 'Zip Code'),\n ('latitude', 'Latitude'),\n ('longitude', 'Longitude'),\n]\n\nEVENT_PARTICIPATION_MAPPABLE_FIELDS = [\n ('voter_id', 'Voter ID'),\n ('first_name', 'First Name'),\n ('last_name', 'Last Name'),\n ('event_name', 'Event Name'),\n ('participation_status', 'Participation Status'),\n]\n\nDONATION_MAPPABLE_FIELDS = [\n ('voter_id', 'Voter ID'),\n ('date', 'Date'),\n ('amount', 'Amount'),\n ('method', 'Donation Method (Name)'),\n]\n\nINTERACTION_MAPPABLE_FIELDS = [\n ('voter_id', 'Voter ID'),\n ('volunteer_email', 'Volunteer Email'),\n ('date', 'Date'),\n ('type', 'Interaction Type (Name)'),\n ('description', 'Description'),\n ('notes', 'Notes'),\n]\n\n\nVOLUNTEER_MAPPABLE_FIELDS = [\n ('first_name', 'First Name'),\n ('last_name', 'Last Name'),\n ('email', 'Email'),\n ('phone', 'Phone'),\n ('notes', 'Notes'),\n]\n\nVOTER_LIKELIHOOD_MAPPABLE_FIELDS = [\n ('voter_id', 'Voter ID'),\n ('election_type', 'Election Type (Name)'),\n ('likelihood', 'Likelihood'),\n]\n\nVOTING_RECORD_MAPPABLE_FIELDS = [\n ('voter_id', 'Voter ID'),\n ('election_date', 'Election Date'),\n ('election_description', 'Election Description'),\n ('primary_party', 'Primary Party'),\n]\n\nclass BaseImportAdminMixin:\n def download_errors(self, request):\n logger.info(f\"download_errors called for {self.model._meta.model_name}\")\n session_key = f\"{self.model._meta.model_name}_import_errors\"\n failed_rows = request.session.get(session_key, [])\n if not failed_rows:\n self.message_user(request, \"No error log found in session.\", level=messages.WARNING)\n return redirect(\"../\")\n\n response = HttpResponse(content_type=\"text/csv\")\n response[\"Content-Disposition\"] = f\"attachment; filename={self.model._meta.model_name}_import_errors.csv\"\n\n if failed_rows:\n all_keys = set()\n for r in failed_rows:\n all_keys.update(r.keys())\n\n writer = csv.DictWriter(response, fieldnames=sorted(list(all_keys)))\n writer.writeheader()\n writer.writerows(failed_rows)\n\n return response\n\n def chunk_reader(self, reader, size):\n chunk = []\n for row in reader:\n chunk.append(row)\n if len(chunk) == size:\n yield chunk\n chunk = []\n if chunk:\n yield chunk\nclass TenantUserRoleInline(admin.TabularInline):\n model = TenantUserRole\n extra = 1\n\nclass CampaignSettingsInline(admin.StackedInline):\n model = CampaignSettings\n can_delete = False\n\n@admin.register(Tenant)\nclass TenantAdmin(admin.ModelAdmin):\n list_display = ('name', 'created_at')\n search_fields = ('name',)\n inlines = [TenantUserRoleInline, CampaignSettingsInline]\n\n@admin.register(TenantUserRole)\nclass TenantUserRoleAdmin(admin.ModelAdmin):\n list_display = ('user', 'tenant', 'role')\n list_filter = ('tenant', 'role')\n search_fields = ('user__username', 'tenant__name')\n\n@admin.register(InteractionType)\nclass InteractionTypeAdmin(admin.ModelAdmin):\n list_display = ('name', 'tenant', 'is_active')\n list_filter = ('tenant', 'is_active')\n search_fields = ('name',)\n\n@admin.register(DonationMethod)\nclass DonationMethodAdmin(admin.ModelAdmin):\n list_display = ('name', 'tenant', 'is_active')\n list_filter = ('tenant', 'is_active')\n search_fields = ('name',)\n\n@admin.register(VolunteerRole)\nclass VolunteerRoleAdmin(admin.ModelAdmin):\n list_display = (\"name\", \"tenant\", \"is_active\")\n list_filter = (\"tenant\", \"is_active\")\n search_fields = (\"name\",)\n\n@admin.register(ElectionType)\nclass ElectionTypeAdmin(admin.ModelAdmin):\n list_display = ('name', 'tenant', 'is_active')\n list_filter = ('tenant', 'is_active')\n search_fields = ('name',)\n\n@admin.register(EventType)\nclass EventTypeAdmin(admin.ModelAdmin):\n list_display = ('name', 'tenant', 'is_active', 'default_volunteer_role')\n list_filter = ('tenant', 'is_active')\n search_fields = ('name',)\n filter_horizontal = ('available_roles',)\n\n\n@admin.register(ParticipationStatus)\nclass ParticipationStatusAdmin(admin.ModelAdmin):\n list_display = ('name', 'tenant', 'is_active')\n list_filter = ('tenant', 'is_active')\n search_fields = ('name',)\n change_list_template = 'admin/participationstatus_change_list.html'\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n@admin.register(Interest)\nclass InterestAdmin(admin.ModelAdmin):\n list_display = ('name', 'tenant')\n list_filter = ('tenant',)\n fields = ('tenant', 'name')\n search_fields = ('name',)\n\nclass VotingRecordInline(admin.TabularInline):\n model = VotingRecord\n extra = 1\n\nclass DonationInline(admin.TabularInline):\n model = Donation\n extra = 1\n\nclass InteractionInline(admin.TabularInline):\n model = Interaction\n extra = 1\n autocomplete_fields = ['voter', 'type', 'volunteer']\n\nclass VoterLikelihoodInline(admin.TabularInline):\n model = VoterLikelihood\n extra = 1\n\nclass VolunteerEventInline(admin.TabularInline):\n model = VolunteerEvent\n extra = 1\n\n@admin.register(Voter)\nclass VoterAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('first_name', 'last_name', 'nickname', 'voter_id', 'tenant', 'district', 'candidate_support', 'is_targeted', 'city', 'state', 'prior_state')\n list_filter = ('tenant', 'candidate_support', 'is_targeted', 'phone_type', 'yard_sign', 'district', 'city', 'state', 'prior_state')\n search_fields = ('first_name', 'last_name', 'nickname', 'voter_id', 'address', 'city', 'state', 'prior_state', 'zip_code', 'county')\n inlines = [VotingRecordInline, DonationInline, InteractionInline, VoterLikelihoodInline]\n readonly_fields = ('address',)\n change_list_template = \"admin/voter_change_list.html\"\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context[\"tenants\"] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='voter-download-errors'),\n path('import-voters/', self.admin_site.admin_view(self.import_voters), name='import-voters'),\n ]\n return my_urls + urls\n\n\n def import_voters(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get(\"file_path\")\n tenant_id = request.POST.get(\"tenant\")\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in VOTER_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f\"map_{field_name}\")\n\n try:\n with open(file_path, \"r\", encoding=\"utf-8-sig\") as f:\n # Optimization: Skip full count for very large files in preview if needed,\n # but here we'll keep it for accuracy unless it's a known bottleneck.\n # For now, let's just do a fast line count.\n total_count = sum(1 for line in f) - 1\n f.seek(0)\n reader = csv.DictReader(f)\n preview_rows = []\n voter_ids_for_preview = []\n for i, row in enumerate(reader):\n if i < 10:\n preview_rows.append(row)\n v_id = row.get(mapping.get(\"voter_id\"))\n if v_id:\n voter_ids_for_preview.append(v_id.strip())\n else:\n break\n\n existing_preview_ids = set(Voter.objects.filter(tenant=tenant, voter_id__in=voter_ids_for_preview).values_list(\"voter_id\", flat=True))\n\n create_count = 0\n update_count = 0\n\n for row in preview_rows:\n voter_id_val = row.get(mapping.get(\"voter_id\"))\n if voter_id_val and voter_id_val.strip() in existing_preview_ids:\n update_count += 1\n else:\n create_count += 1\n\n context = self.admin_site.each_context(request)\n context.update({\n \"title\": \"Import Preview\",\n \"total_count\": total_count,\n \"create_count\": create_count,\n \"update_count\": update_count,\n \"preview_data\": preview_rows, # This should be improved to show actual changes\n \"mapping\": mapping,\n \"file_path\": file_path,\n \"tenant_id\": tenant_id,\n \"action_url\": request.path,\n \"opts\": self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n elif \"_import\" in request.POST:\n file_path = request.POST.get(\"file_path\")\n tenant_id = request.POST.get(\"tenant\")\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in VOTER_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f\"map_{field_name}\")\n\n try:\n created_count = 0\n updated_count = 0\n skipped_no_change = 0\n skipped_no_id = 0\n errors = 0\n failed_rows = []\n total_processed = 0\n\n # Temporary storage for error rows to avoid holding large file in memory\n temp_error_file = None\n temp_error_file_path = None\n\n # Process in chunks to reduce memory usage for very large files\n with open(file_path, \"r\", encoding=\"utf-8-sig\") as f_read:\n reader = csv.DictReader(f_read)\n for i, row in enumerate(reader):\n total_processed += 1\n try:\n raw_voter_id = row.get(mapping.get(\"voter_id\"))\n voter_id = raw_voter_id.strip() if raw_voter_id else None\n\n if not voter_id:\n # Enhanced error message to guide the user\n mapped_column_name = mapping.get(\"voter_id\", \"N/A\")\n error_detail = f\"Raw value: '{raw_voter_id}'. \" if raw_voter_id is not None else \"Value was None.\"\n row[\"Import Error\"] = f\"Voter ID is required. Please check if the '{mapped_column_name}' column is correctly mapped and contains values for all rows. {error_detail}\"\n failed_rows.append(row)\n skipped_no_id += 1\n errors += 1\n continue\n\n defaults = {}\n # Map other fields dynamically\n for field_name, _ in VOTER_MAPPABLE_FIELDS:\n csv_column = mapping.get(field_name)\n if csv_column and csv_column in row:\n field_value = row[csv_column].strip()\n if field_name == \"birthdate\" or field_name == \"registration_date\":\n # Handle date conversions\n if field_value:\n try:\n # Attempt to parse common date formats\n if '/' in field_value:\n # Try MM/DD/YYYY or DD/MM/YYYY\n if len(field_value.split('/')[2]) == 2: # YY format\n dt = datetime.strptime(field_value, '%m/%d/%y').date() if len(field_value.split('/')[0]) < 3 else datetime.strptime(field_value, '%d/%m/%y').date() # noqa\n else:\n dt = datetime.strptime(field_value, '%m/%d/%Y').date() if len(field_value.split('/')[0]) < 3 else datetime.strptime(field_value, '%d/%m/%Y').date() # noqa\n elif '-' in field_value:\n # Try YYYY-MM-DD or DD-MM-YYYY or MM-DD-YYYY\n if len(field_value.split('-')[0]) == 4: # YYYY format\n dt = datetime.strptime(field_value, '%Y-%m-%d').date()\n elif len(field_value.split('-')[2]) == 4: # YYYY format\n dt = datetime.strptime(field_value, '%m-%d-%Y').date() if len(field_value.split('-')[0]) < 3 else datetime.strptime(field_value, '%d-%m-%Y').date() # noqa\n else:\n # Default to MM-DD-YY\n dt = datetime.strptime(field_value, '%m-%d-%y').date()\n else:\n dt = None\n\n if dt:\n defaults[field_name] = dt\n else:\n logger.warning(f\"Could not parse date '{field_value}' for field {field_name}. Skipping.\")\n except ValueError as ve:\n logger.warning(f\"Date parsing error for '{field_value}' in field {field_name}: {ve}\")\n except Exception as ex:\n logger.error(f\"Unexpected error parsing date '{field_value}' for field {field_name}: {ex}\")\n elif field_name == \"is_targeted\" or field_name == \"yard_sign\" or field_name == \"window_sticker\" or field_name == \"door_visit\":\n # Handle boolean fields\n if field_value.lower() == 'true' or field_value == '1':\n defaults[field_name] = True\n elif field_value.lower() == 'false' or field_value == '0':\n defaults[field_name] = False\n else:\n defaults[field_name] = None # Or sensible default/error\n elif field_name == \"phone\":\n defaults[field_name] = format_phone_number(field_value)\n elif field_name == \"email\":\n defaults[field_name] = field_value.lower() # Store emails as lowercase\n elif field_name == \"candidate_support\":\n if field_value in [choice[0] for choice in Voter.CANDIDATE_SUPPORT_CHOICES]:\n defaults[field_name] = field_value\n else:\n logger.warning(f\"Invalid candidate_support value: {field_value}. Skipping.\")\n elif field_name == \"phone_type\":\n if field_value in [choice[0] for choice in Voter.PHONE_TYPE_CHOICES]:\n defaults[field_name] = field_value\n else:\n logger.warning(f\"Invalid phone_type value: {field_value}. Skipping.\")\n elif field_name == \"secondary_phone_type\":\n if field_value in [choice[0] for choice in Voter.PHONE_TYPE_CHOICES]:\n defaults[field_name] = field_value\n else:\n logger.warning(f\"Invalid secondary_phone_type value: {field_value}. Skipping.\")\n elif field_name == \"state\" or field_name == \"prior_state\":\n # Ensure state is uppercase and valid length\n if field_value and len(field_value) <= 2:\n defaults[field_name] = field_value.upper()\n else:\n logger.warning(f\"Invalid state value: {field_value}. Skipping.\")\n else:\n defaults[field_name] = field_value\n\n # Try to get voter. If not found, create new. Update if found.\n voter, created = Voter.objects.update_or_create(\n tenant=tenant,\n voter_id=voter_id,\n defaults=defaults\n )\n if created:\n created_count += 1\n else:\n updated_count += 1\n\n # Special handling for interests - assuming a comma-separated list in CSV\n if 'interests' in mapping and row.get(mapping['interests']):\n interest_names = [name.strip() for name in row[mapping['interests']].split(',') if name.strip()]\n for interest_name in interest_names:\n interest, _ = Interest.objects.get_or_create(tenant=tenant, name=interest_name)\n voter.interests.add(interest)\n\n if (i + 1) % 100 == 0:\n print(f\"DEBUG: Voter import progress: {total_processed} processed. {created_count} created. {updated_count} updated.\")\n\n except Exception as e:\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n logger.error(f\"Error importing row: {row}. Error: {e}\")\n\n # Clean up the temporary file\n if os.path.exists(file_path):\n os.remove(file_path)\n\n if temp_error_file_path and os.path.exists(temp_error_file_path):\n os.remove(temp_error_file_path)\n\n self.message_user(request, f\"Import complete: {created_count + updated_count} voters created/updated. ({created_count} new, {updated_count} updated, {skipped_no_change} skipped with no changes, {skipped_no_id} skipped missing ID, {errors} errors)\")\n # Store failed rows in session for download, limit to avoid session overflow\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n\n if errors > 0:\n error_url = reverse(\"admin:voter-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = VoterImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n \"title\": \"Map Voter Fields\",\n \"headers\": headers,\n \"model_fields\": VOTER_MAPPABLE_FIELDS,\n \"tenant_id\": tenant.id,\n \"file_path\": file_path,\n \"action_url\": request.path,\n \"opts\": self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = VoterImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Voters\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)\n\n\n@admin.register(Event)\nclass EventAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('name', 'date', 'event_type', 'tenant', 'location_name', 'address', 'city', 'state', 'zip_code')\n list_filter = ('tenant', 'event_type')\n search_fields = ('name', 'location_name', 'address', 'city', 'state', 'zip_code')\n inlines = [VolunteerEventInline]\n change_list_template = \"admin/event_change_list.html\"\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='event-download-errors'),\n path('import-events/', self.admin_site.admin_view(self.import_events), name='import-events'),\n ]\n return my_urls + urls\n\n def import_events(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n mapping = {}\n for field_name, _ in EVENT_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n total_count = 0\n create_count = 0\n update_count = 0\n preview_data = []\n for row in reader:\n total_count += 1\n event_name = row.get(mapping.get('name'))\n event_date = row.get(mapping.get('date'))\n\n exists = False\n if event_name and event_date:\n try:\n # Assuming name and date uniquely identify an event\n # This might need refinement based on actual data uniqueness requirements\n if '/' in event_date:\n dt = datetime.strptime(event_date, '%m/%d/%Y').date()\n elif '-' in event_date:\n dt = datetime.strptime(event_date, '%Y-%m-%d').date()\n else:\n dt = None\n\n if dt:\n exists = Event.objects.filter(tenant=tenant, name=event_name, date=dt).exists()\n\n except ValueError:\n # Handle cases where date parsing fails\n pass\n\n if exists:\n update_count += 1\n action = 'update'\n else:\n create_count += 1\n action = 'create'\n\n if len(preview_data) < 10:\n preview_data.append({\n 'action': action,\n 'identifier': f\"Event: {event_name} (Date: {event_date})\",\n 'details': f\"Location: {row.get(mapping.get('location_name', '')) or ''}\"\n })\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Import Preview\",\n 'total_count': total_count,\n 'create_count': create_count,\n 'update_count': update_count,\n 'preview_data': preview_data,\n 'mapping': mapping,\n 'file_path': file_path,\n 'tenant_id': tenant_id,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n\n elif \"_import\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in EVENT_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n count = 0\n errors = 0\n failed_rows = []\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n for row in reader:\n try:\n event_name = row.get(mapping.get('name'))\n event_date = row.get(mapping.get('date'))\n event_type_name = row.get(mapping.get('event_type'))\n\n if not event_name or not event_date or not event_type_name:\n row[\"Import Error\"] = \"Missing event name, date, or type\"\n failed_rows.append(row)\n errors += 1\n continue\n\n # Date parsing for event_date\n try:\n if '/' in event_date:\n parsed_date = datetime.strptime(event_date, '%m/%d/%Y').date()\n elif '-' in event_date:\n parsed_date = datetime.strptime(event_date, '%Y-%m-%d').date()\n else:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n except ValueError:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n\n event_type_obj, _ = EventType.objects.get_or_create(tenant=tenant, name=event_type_name)\n\n defaults = {\n 'date': parsed_date,\n 'event_type': event_type_obj,\n 'description': row.get(mapping.get('description')) or '',\n 'location_name': row.get(mapping.get('location_name')) or '',\n 'address': row.get(mapping.get('address')) or '',\n 'city': row.get(mapping.get('city')) or '',\n 'state': row.get(mapping.get('state')) or '',\n 'zip_code': row.get(mapping.get('zip_code')) or '',\n 'latitude': row.get(mapping.get('latitude')) or None,\n 'longitude': row.get(mapping.get('longitude')) or None,\n }\n\n # Handle start_time and end_time\n start_time_str = row.get(mapping.get('start_time'))\n if start_time_str:\n try:\n defaults['start_time'] = datetime.strptime(start_time_str, '%H:%M').time()\n except ValueError:\n logger.warning(f\"Invalid start_time format: {start_time_str}. Skipping.\")\n end_time_str = row.get(mapping.get('end_time'))\n if end_time_str:\n try:\n defaults['end_time'] = datetime.strptime(end_time_str, '%H:%M').time()\n except ValueError:\n logger.warning(f\"Invalid end_time format: {end_time_str}. Skipping.\")\n\n Event.objects.update_or_create(\n tenant=tenant,\n name=event_name,\n defaults=defaults\n )\n count += 1\n except Exception as e:\n logger.error(f\"Error importing: {e}\")\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n\n if os.path.exists(file_path):\n os.remove(file_path)\n self.message_user(request, f\"Successfully imported {count} events.\")\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n if errors > 0:\n error_url = reverse(\"admin:event-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = EventImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Map Event Fields\",\n 'headers': headers,\n 'model_fields': EVENT_MAPPABLE_FIELDS,\n 'tenant_id': tenant.id,\n 'file_path': file_path,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = EventImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Events\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)\n\n\n@admin.register(Volunteer)\nclass VolunteerAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('first_name', 'last_name', 'email', 'phone', 'tenant')\n list_filter = ('tenant',)\n search_fields = ('first_name', 'last_name', 'email', 'phone')\n change_list_template = \"admin/volunteer_change_list.html\"\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='volunteer-download-errors'),\n path('import-volunteers/', self.admin_site.admin_view(self.import_volunteers), name='import-volunteers'),\n ]\n return my_urls + urls\n\n def import_volunteers(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n mapping = {}\n for field_name, _ in VOLUNTEER_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n total_count = 0\n create_count = 0\n update_count = 0\n preview_data = []\n for row in reader:\n total_count += 1\n email = row.get(mapping.get('email'))\n\n exists = False\n if email:\n exists = Volunteer.objects.filter(tenant=tenant, email=email).exists()\n\n if exists:\n update_count += 1\n action = 'update'\n else:\n create_count += 1\n action = 'create'\n\n if len(preview_data) < 10:\n preview_data.append({\n 'action': action,\n 'identifier': f\"Volunteer: {email}\",\n 'details': f\"Name: {row.get(mapping.get('first_name', '')) or ''} {row.get(mapping.get('last_name', '')) or ''}\"\n })\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Import Preview\",\n 'total_count': total_count,\n 'create_count': create_count,\n 'update_count': update_count,\n 'preview_data': preview_data,\n 'mapping': mapping,\n 'file_path': file_path,\n 'tenant_id': tenant_id,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n\n elif \"_import\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in VOLUNTEER_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n count = 0\n errors = 0\n failed_rows = []\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n for row in reader:\n try:\n email = row.get(mapping.get('email'))\n if not email:\n row[\"Import Error\"] = \"Missing email\"\n failed_rows.append(row)\n errors += 1\n continue\n\n defaults = {\n 'first_name': row.get(mapping.get('first_name')) or '',\n 'last_name': row.get(mapping.get('last_name')) or '',\n 'phone': format_phone_number(row.get(mapping.get('phone')) or ''),\n 'notes': row.get(mapping.get('notes')) or '',\n }\n\n Volunteer.objects.update_or_create(\n tenant=tenant,\n email=email,\n defaults=defaults\n )\n count += 1\n except Exception as e:\n logger.error(f\"Error importing volunteer: {e}\")\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n\n if os.path.exists(file_path):\n os.remove(file_path)\n self.message_user(request, f\"Successfully imported {count} volunteers.\")\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n if errors > 0:\n error_url = reverse(\"admin:volunteer-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = VolunteerImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Map Volunteer Fields\",\n 'headers': headers,\n 'model_fields': VOLUNTEER_MAPPABLE_FIELDS,\n 'tenant_id': tenant.id,\n 'file_path': file_path,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = VolunteerImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Volunteers\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)\n\n\n@admin.register(EventParticipation)\nclass EventParticipationAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('event', 'voter', 'participation_status')\n list_filter = ('event', 'participation_status', 'voter__tenant')\n search_fields = ('event__name', 'voter__first_name', 'voter__last_name', 'voter__voter_id')\n change_list_template = 'admin/eventparticipation_change_list.html'\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_list(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='eventparticipation-download-errors'),\n path('import-event-participations/', self.admin_site.admin_view(self.import_event_participations), name='import-event-participations'),\n ]\n return my_urls + urls\n\n def import_event_participations(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n mapping = {}\n for field_name, _ in EVENT_PARTICIPATION_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n total_count = 0\n create_count = 0\n update_count = 0\n preview_data = []\n for row in reader:\n total_count += 1\n voter_id = row.get(mapping.get('voter_id'))\n\n # Extract first_name and last_name from CSV based on mapping\n csv_first_name = row.get(mapping.get('first_name'), '')\n csv_last_name = row.get(mapping.get('last_name'), '')\n csv_full_name = f\"{csv_first_name} {csv_last_name}\".strip()\n\n exists = False\n voter_full_name = \"N/A\" # Initialize voter_full_name\n if voter_id:\n try:\n voter = Voter.objects.get(tenant=tenant, voter_id=voter_id)\n voter_full_name = f\"{voter.first_name} {voter.last_name}\" # Get voter's full name\n if event_name:\n exists = EventParticipation.objects.filter(voter=voter, event__name=event_name).exists()\n except Voter.DoesNotExist:\n pass\n\n if exists:\n update_count += 1\n action = 'update'\n else:\n create_count += 1\n action = 'create'\n\n if len(preview_data) < 10:\n preview_data.append({\n 'action': action,\n 'csv_full_name': csv_full_name, # Add CSV name\n 'identifier': f\"Voter: {voter_full_name} (ID: {voter_id})\" if voter_id else \"N/A\", # Include full name\n 'details': f\"Participation: {row.get(mapping.get('participation_status', '')) or ''}\"\n })\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Import Preview\",\n 'total_count': total_count,\n 'create_count': create_count,\n 'update_count': update_count,\n 'preview_data': preview_data,\n 'mapping': mapping,\n 'file_path': file_path,\n 'tenant_id': tenant_id,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n\n elif \"_import\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in EVENT_PARTICIPATION_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n count = 0\n errors = 0\n failed_rows = []\n for row in reader:\n try:\n voter_id = row.get(mapping.get('voter_id')) if mapping.get('voter_id') else None\n participation_status_val = row.get(mapping.get('participation_status')) if mapping.get('participation_status') else None\n\n if voter_id: # Only strip if voter_id is not None\n voter_id = voter_id.strip()\n\n if not voter_id:\n row[\"Import Error\"] = \"Missing voter ID\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n voter = Voter.objects.get(tenant=tenant, voter_id=voter_id)\n except Voter.DoesNotExist:\n error_msg = f\"Voter with ID {voter_id} not found\"\n logger.error(error_msg)\n row[\"Import Error\"] = error_msg\n failed_rows.append(row)\n errors += 1\n continue\n\n event = None\n event_name = row.get(mapping.get('event_name')) if mapping.get('event_name') else None\n if event_name:\n try:\n event = Event.objects.get(tenant=tenant, name=event_name)\n except Event.DoesNotExist:\n pass\n\n if not event:\n error_msg = \"Event not found (check Event Name)\"\n logger.error(error_msg)\n row[\"Import Error\"] = error_msg\n failed_rows.append(row)\n errors += 1\n continue\n\n defaults = {}\n if participation_status_val and participation_status_val.strip():\n status_obj, _ = ParticipationStatus.objects.get_or_create(tenant=tenant, name=participation_status_val.strip())\n defaults['participation_status'] = status_obj\n else:\n # Default to 'Invited' if not specified\n status_obj, _ = ParticipationStatus.objects.get_or_create(tenant=tenant, name='Invited')\n defaults['participation_status'] = status_obj\n EventParticipation.objects.update_or_create(\n event=event,\n voter=voter,\n defaults=defaults\n )\n count += 1\n except Exception as e:\n logger.error(f\"Error importing: {e}\")\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n\n if os.path.exists(file_path):\n os.remove(file_path)\n self.message_user(request, f\"Successfully imported {count} participations.\")\n # Optimization: Limit error log size in session to avoid overflow\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n logger.info(f\"Stored {len(failed_rows)} failed rows in session for {self.model._meta.model_name}\")\n if errors > 0:\n error_url = reverse(\"admin:eventparticipation-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = EventParticipationImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Map Participation Fields\",\n 'headers': headers,\n 'model_fields': EVENT_PARTICIPATION_MAPPABLE_FIELDS,\n 'tenant_id': tenant.id,\n 'file_path': file_path,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = EventParticipationImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Participations\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)\n\n@admin.register(Donation)\nclass DonationAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('voter', 'date', 'amount', 'method')\n list_filter = ('voter__tenant', 'method')\n search_fields = ('voter__first_name', 'voter__last_name', 'voter__voter_id', 'method__name')\n change_list_template = 'admin/donation_change_list.html'\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='donation-download-errors'),\n path('import-donations/', self.admin_site.admin_view(self.import_donations), name='import-donations'),\n ]\n return my_urls + urls\n\n def import_donations(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n mapping = {}\n for field_name, _ in DONATION_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n total_count = 0\n create_count = 0\n update_count = 0\n preview_data = []\n for row in reader:\n total_count += 1\n voter_id = row.get(mapping.get('voter_id'))\n\n exists = False\n if voter_id:\n exists = Voter.objects.filter(tenant=tenant, voter_id=voter_id).exists()\n\n if exists:\n update_count += 1\n action = 'update'\n else:\n create_count += 1\n action = 'create'\n\n if len(preview_data) < 10:\n preview_data.append({\n 'action': action,\n 'identifier': f\"Voter ID: {voter_id}\",\n 'details': f\"Amount: {row.get(mapping.get('amount', '')) or ''}, Method: {row.get(mapping.get('method', '')) or ''}\"\n })\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Import Preview\",\n 'total_count': total_count,\n 'create_count': create_count,\n 'update_count': update_count,\n 'preview_data': preview_data,\n 'mapping': mapping,\n 'file_path': file_path,\n 'tenant_id': tenant_id,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n\n elif \"_import\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in DONATION_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n count = 0\n errors = 0\n failed_rows = []\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n for row in reader:\n try:\n voter_id = row.get(mapping.get('voter_id'))\n date_str = row.get(mapping.get('date'))\n amount_str = row.get(mapping.get('amount'))\n method_name = row.get(mapping.get('method'))\n\n if voter_id: # Only strip if voter_id is not None\n voter_id = voter_id.strip()\n\n if not voter_id:\n row[\"Import Error\"] = \"Missing voter ID\"\n failed_rows.append(row)\n errors += 1\n continue\n\n if not date_str or not amount_str:\n row[\"Import Error\"] = \"Missing date or amount\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n voter = Voter.objects.get(tenant=tenant, voter_id=voter_id)\n except Voter.DoesNotExist:\n row[\"Import Error\"] = f\"Voter {voter_id} not found\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n if '/' in date_str:\n parsed_date = datetime.strptime(date_str, '%m/%d/%Y').date()\n elif '-' in date_str:\n parsed_date = datetime.strptime(date_str, '%Y-%m-%d').date()\n else:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n except ValueError:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n amount = Decimal(amount_str)\n except InvalidOperation:\n row[\"Import Error\"] = \"Invalid amount format\"\n failed_rows.append(row)\n errors += 1\n continue\n\n donation_method, _ = DonationMethod.objects.get_or_create(tenant=tenant, name=method_name)\n\n Donation.objects.create(\n voter=voter,\n date=parsed_date,\n amount=amount,\n method=donation_method\n )\n count += 1\n except Exception as e:\n logger.error(f\"Error importing: {e}\")\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n\n if os.path.exists(file_path):\n os.remove(file_path)\n self.message_user(request, f\"Successfully imported {count} donations.\")\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n if errors > 0:\n error_url = reverse(\"admin:donation-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = DonationImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Map Donation Fields\",\n 'headers': headers,\n 'model_fields': DONATION_MAPPABLE_FIELDS,\n 'tenant_id': tenant.id,\n 'file_path': file_path,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = DonationImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Donations\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)\n\n@admin.register(Interaction)\nclass InteractionAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('voter', 'date', 'type', 'description', 'volunteer')\n list_filter = ('voter__tenant', 'type', 'volunteer')\n search_fields = ('voter__first_name', 'voter__last_name', 'voter__voter_id', 'description', 'volunteer__first_name', 'volunteer__last_name')\n autocomplete_fields = ['voter', 'volunteer']\n change_list_template = 'admin/interaction_change_list.html'\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='interaction-download-errors'),\n path('import-interactions/', self.admin_site.admin_view(self.import_interactions), name='import-interactions'),\n ]\n return my_urls + urls\n\n def import_interactions(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n mapping = {}\n for field_name, _ in INTERACTION_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n total_count = 0\n create_count = 0\n update_count = 0\n preview_data = []\n for row in reader:\n total_count += 1\n voter_id = row.get(mapping.get('voter_id'))\n volunteer_email = row.get(mapping.get('volunteer_email'))\n\n exists = False\n if voter_id:\n exists = Voter.objects.filter(tenant=tenant, voter_id=voter_id).exists()\n\n if exists:\n update_count += 1\n action = 'update'\n else:\n create_count += 1\n action = 'create'\n\n if len(preview_data) < 10:\n preview_data.append({\n 'action': action,\n 'identifier': f\"Voter ID: {voter_id}\",\n 'details': f\"Type: {row.get(mapping.get('type', '')) or ''}, Volunteer: {volunteer_email or ''}\"\n })\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Import Preview\",\n 'total_count': total_count,\n 'create_count': create_count,\n 'update_count': update_count,\n 'preview_data': preview_data,\n 'mapping': mapping,\n 'file_path': file_path,\n 'tenant_id': tenant_id,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n\n elif \"_import\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in INTERACTION_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n count = 0\n errors = 0\n failed_rows = []\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n for row in reader:\n try:\n voter_id = row.get(mapping.get('voter_id'))\n volunteer_email = row.get(mapping.get('volunteer_email'))\n date_str = row.get(mapping.get('date'))\n type_name = row.get(mapping.get('type'))\n\n if voter_id: # Only strip if voter_id is not None\n voter_id = voter_id.strip()\n\n if not voter_id:\n row[\"Import Error\"] = \"Missing voter ID\"\n failed_rows.append(row)\n errors += 1\n continue\n\n if not date_str or not type_name:\n row[\"Import Error\"] = \"Missing date or description\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n voter = Voter.objects.get(tenant=tenant, voter_id=voter_id)\n except Voter.DoesNotExist:\n row[\"Import Error\"] = f\"Voter {voter_id} not found\"\n failed_rows.append(row)\n errors += 1\n continue\n\n volunteer = None\n if volunteer_email:\n try:\n volunteer = Volunteer.objects.get(tenant=tenant, email=volunteer_email)\n except Volunteer.DoesNotExist:\n pass # Volunteer is optional\n\n try:\n if '/' in date_str:\n parsed_date = datetime.strptime(date_str, '%m/%d/%Y').date()\n elif '-' in date_str:\n parsed_date = datetime.strptime(date_str, '%Y-%m-%d').date()\n else:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n except ValueError:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n\n interaction_type, _ = InteractionType.objects.get_or_create(tenant=tenant, name=type_name)\n\n Interaction.objects.create(\n voter=voter,\n volunteer=volunteer,\n date=parsed_date,\n type=interaction_type,\n description=row.get(mapping.get('description')) or '',\n notes=row.get(mapping.get('notes')) or ''\n )\n count += 1\n except Exception as e:\n logger.error(f\"Error importing: {e}\")\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n\n if os.path.exists(file_path):\n os.remove(file_path)\n self.message_user(request, f\"Successfully imported {count} interactions.\")\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n if errors > 0:\n error_url = reverse(\"admin:interaction-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = InteractionImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Map Interaction Fields\",\n 'headers': headers,\n 'model_fields': INTERACTION_MAPPABLE_FIELDS,\n 'tenant_id': tenant.id,\n 'file_path': file_path,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = InteractionImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Interactions\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)\n\n@admin.register(VoterLikelihood)\nclass VoterLikelihoodAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('voter', 'election_type', 'likelihood')\n list_filter = ('voter__tenant', 'election_type', 'likelihood')\n search_fields = ('voter__first_name', 'voter__last_name', 'voter__voter_id', 'election_type__name')\n change_list_template = 'admin/voterlikelihood_change_list.html'\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='voterlikelihood-download-errors'),\n path('import-likelihoods/', self.admin_site.admin_view(self.import_likelihoods), name='import-likelihoods'),\n ]\n return my_urls + urls\n\n def import_likelihoods(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n mapping = {}\n for field_name, _ in VOTER_LIKELIHOOD_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n total_count = 0\n create_count = 0\n update_count = 0\n preview_data = []\n for row in reader:\n total_count += 1\n voter_id = row.get(mapping.get('voter_id'))\n\n exists = False\n if voter_id:\n exists = Voter.objects.filter(tenant=tenant, voter_id=voter_id).exists()\n\n if exists:\n update_count += 1\n action = 'update'\n else:\n create_count += 1\n action = 'create'\n\n if len(preview_data) < 10:\n preview_data.append({\n 'action': action,\n 'identifier': f\"Voter ID: {voter_id}\",\n 'details': f\"Likelihood: {row.get(mapping.get('likelihood', '')) or ''}\"\n })\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Import Preview\",\n 'total_count': total_count,\n 'create_count': create_count,\n 'update_count': update_count,\n 'preview_data': preview_data,\n 'mapping': mapping,\n 'file_path': file_path,\n 'tenant_id': tenant_id,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n\n elif \"_import\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in VOTER_LIKELIHOOD_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n count = 0\n errors = 0\n failed_rows = []\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n for row in reader:\n try:\n voter_id = row.get(mapping.get('voter_id'))\n election_type_name = row.get(mapping.get('election_type'))\n likelihood_val = row.get(mapping.get('likelihood'))\n\n if voter_id: # Only strip if voter_id is not None\n voter_id = voter_id.strip()\n\n if not voter_id:\n row[\"Import Error\"] = \"Missing voter ID\"\n failed_rows.append(row)\n errors += 1\n continue\n\n if not election_type_name or not likelihood_val:\n row[\"Import Error\"] = \"Missing election type or likelihood\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n voter = Voter.objects.get(tenant=tenant, voter_id=voter_id)\n except Voter.DoesNotExist:\n row[\"Import Error\"] = f\"Voter {voter_id} not found\"\n failed_rows.append(row)\n errors += 1\n continue\n\n election_type, _ = ElectionType.objects.get_or_create(tenant=tenant, name=election_type_name)\n\n VoterLikelihood.objects.update_or_create(\n voter=voter,\n election_type=election_type,\n defaults={'likelihood': likelihood_val}\n )\n count += 1\n except Exception as e:\n print(f\"DEBUG: Likelihood import failed: {e}\")\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n\n if os.path.exists(file_path):\n os.remove(file_path)\n self.message_user(request, f\"Import complete: {count} likelihoods created/updated.\")\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n if errors > 0:\n error_url = reverse(\"admin:voterlikelihood-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = VoterLikelihoodImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Map Likelihood Fields\",\n 'headers': headers,\n 'model_fields': VOTER_LIKELIHOOD_MAPPABLE_FIELDS,\n 'tenant_id': tenant.id,\n 'file_path': file_path,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = VoterLikelihoodImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Likelihoods\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)\n\n\n@admin.register(VotingRecord)\nclass VotingRecordAdmin(BaseImportAdminMixin, admin.ModelAdmin):\n list_display = ('voter', 'election_date', 'election_description', 'primary_party')\n list_filter = ('voter__tenant', 'primary_party')\n search_fields = ('voter__first_name', 'voter__last_name', 'voter__voter_id', 'election_description')\n change_list_template = 'admin/votingrecord_change_list.html'\n\n def changelist_view(self, request, extra_context=None):\n extra_context = extra_context or {}\n from core.models import Tenant\n extra_context['tenants'] = Tenant.objects.all()\n return super().changelist_view(request, extra_context=extra_context)\n\n def get_urls(self):\n urls = super().get_urls()\n my_urls = [\n path('download-errors/', self.admin_site.admin_view(self.download_errors), name='votingrecord-download-errors'),\n path('import-voting-records/', self.admin_site.admin_view(self.import_voting_records), name='import-voting-records'),\n ]\n return my_urls + urls\n\n def import_voting_records(self, request):\n if request.method == \"POST\":\n if \"_preview\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n mapping = {}\n for field_name, _ in VOTING_RECORD_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n try:\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n total_count = 0\n create_count = 0\n update_count = 0\n preview_data = []\n for row in reader:\n total_count += 1\n voter_id = row.get(mapping.get('voter_id'))\n election_date = row.get(mapping.get('election_date'))\n\n exists = False\n if voter_id and election_date:\n try:\n # Assuming voter_id and election_date uniquely identify a voting record\n # This might need refinement based on actual data uniqueness requirements\n if '/' in election_date:\n dt = datetime.strptime(election_date, '%m/%d/%Y').date()\n elif '-' in election_date:\n dt = datetime.strptime(election_date, '%Y-%m-%d').date()\n else:\n dt = None\n\n if dt:\n exists = VotingRecord.objects.filter(voter__tenant=tenant, voter__voter_id=voter_id, election_date=dt).exists()\n\n except ValueError:\n # Handle cases where date parsing fails\n pass\n\n if exists:\n update_count += 1\n action = 'update'\n else:\n create_count += 1\n action = 'create'\n\n if len(preview_data) < 10:\n preview_data.append({\n 'action': action,\n 'identifier': f\"Voter ID: {voter_id} (Election: {election_date})\",\n 'details': f\"Party: {row.get(mapping.get('primary_party', '')) or ''}\"\n })\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Import Preview\",\n 'total_count': total_count,\n 'create_count': create_count,\n 'update_count': update_count,\n 'preview_data': preview_data,\n 'mapping': mapping,\n 'file_path': file_path,\n 'tenant_id': tenant_id,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_preview.html\", context)\n except Exception as e:\n self.message_user(request, f\"Error processing preview: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n\n elif \"_import\" in request.POST:\n file_path = request.POST.get('file_path')\n tenant_id = request.POST.get('tenant')\n tenant = Tenant.objects.get(id=tenant_id)\n\n mapping = {}\n for field_name, _ in VOTING_RECORD_MAPPABLE_FIELDS:\n mapping[field_name] = request.POST.get(f'map_{field_name}')\n\n try:\n count = 0\n errors = 0\n failed_rows = []\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.DictReader(f)\n for row in reader:\n try:\n voter_id = row.get(mapping.get('voter_id'))\n election_date_str = row.get(mapping.get('election_date'))\n election_description = row.get(mapping.get('election_description'))\n primary_party = row.get(mapping.get('primary_party'))\n\n if voter_id: # Only strip if voter_id is not None\n voter_id = voter_id.strip()\n\n if not voter_id:\n row[\"Import Error\"] = \"Missing voter ID\"\n failed_rows.append(row)\n errors += 1\n continue\n\n if not election_date_str or not election_description:\n row[\"Import Error\"] = \"Missing election date or description\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n voter = Voter.objects.get(tenant=tenant, voter_id=voter_id)\n except Voter.DoesNotExist:\n row[\"Import Error\"] = f\"Voter {voter_id} not found\"\n failed_rows.append(row)\n errors += 1\n continue\n\n try:\n if '/' in election_date_str:\n parsed_election_date = datetime.strptime(election_date_str, '%m/%d/%Y').date()\n elif '-' in election_date_str:\n parsed_election_date = datetime.strptime(election_date_str, '%Y-%m-%d').date()\n else:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n except ValueError:\n row[\"Import Error\"] = \"Invalid date format\"\n failed_rows.append(row)\n errors += 1\n continue\n\n VotingRecord.objects.update_or_create(\n voter=voter,\n election_date=parsed_election_date,\n defaults={\n 'election_description': election_description,\n 'primary_party': primary_party or ''\n }\n )\n count += 1\n except Exception as e:\n logger.error(f\"Error importing: {e}\")\n row[\"Import Error\"] = str(e)\n failed_rows.append(row)\n errors += 1\n\n if os.path.exists(file_path):\n os.remove(file_path)\n self.message_user(request, f\"Successfully imported {count} voting records.\")\n request.session[f\"{self.model._meta.model_name}_import_errors\"] = failed_rows[:1000]\n request.session.modified = True\n if errors > 0:\n error_url = reverse(\"admin:votingrecord-download-errors\")\n self.message_user(request, mark_safe(f\"Failed to import {errors} rows. <a href='{error_url}' download>Download failed records</a>\"), level=messages.WARNING)\n return redirect(\"../\")\n except Exception as e:\n self.message_user(request, f\"Error processing file: {e}\", level=messages.ERROR)\n return redirect(\"../\")\n else:\n form = VotingRecordImportForm(request.POST, request.FILES)\n if form.is_valid():\n csv_file = request.FILES['file']\n tenant = form.cleaned_data['tenant']\n\n if not csv_file.name.endswith('.csv'):\n self.message_user(request, \"Please upload a CSV file.\", level=messages.ERROR)\n return redirect(\"../\")\n\n with tempfile.NamedTemporaryFile(delete=False, suffix='.csv') as tmp:\n for chunk in csv_file.chunks():\n tmp.write(chunk)\n file_path = tmp.name\n\n with open(file_path, 'r', encoding='utf-8-sig') as f:\n reader = csv.reader(f)\n headers = next(reader)\n\n context = self.admin_site.each_context(request)\n context.update({\n 'title': \"Map Voting Record Fields\",\n 'headers': headers,\n 'model_fields': VOTING_RECORD_MAPPABLE_FIELDS,\n 'tenant_id': tenant.id,\n 'file_path': file_path,\n 'action_url': request.path,\n 'opts': self.model._meta,\n })\n return render(request, \"admin/import_mapping.html\", context)\n else:\n form = VotingRecordImportForm()\n\n context = self.admin_site.each_context(request)\n context['form'] = form\n context['title'] = \"Import Voting Records\"\n context['opts'] = self.model._meta\n return render(request, \"admin/import_csv.html\", context)"